AAT Reuse Data Collection

JavaScript isn't enabled in your browser, so this file can't be opened. Enable and reload.

Untitled form

Send

Untitled form

AAT Reuse Data Collection

Questions

Responses

Section 1 of 1

Name

Question

Question Type

Required

Loading…

Company's Name

Question

Question Type

Company's Name

Required

Loading…

Role (e.g., Developer, Tester, Product Manager)

Question

Question Type

Role (e.g., Developer, Tester, Product Manager)

Required

Loading…

Objective

Title

Objective

Loading…

Background

Title

Recently acceptance test-driven development has gained popularity, and several variants of acceptance test-driven development have been introduced, e.g., Behavior Driven Development (BDD), Story-Driven Development (SDD), Key-word Driven Development, FIT Tables, and specification by example. These test-driven development methodologies specify the AATs in natural language format, e.g., feature files or text files in BDD, and stories in SDD. However, FIT tables are not specified in a natural language format. A recent literature review found that the majority of the studies present in literature are based on the FIT framework (21 out of 26 identified studies). Software practitioners use various tools and technologies to develop these test cases such as Cucumber, robot framework, and FitNesse.

Angmo and Sharma evaluated AAT tools and proposed that cost-benefits analysis should be done when considering a tool for automated acceptance tests. Haugset and Stalhane claim that AAT can provide cost-related benefits in two ways (i) correctness of requirement, and, (ii) automated test-ability. However, they suggest that the software community believes that the costs of AATs are greater than the benefits of AATs. Borg and Kropp described a tool that helps in the maintenance, and refactoring automated acceptance tests, reducing their maintenance costs.

In short, research on AATs is still new, and work is further required to evaluate aspects such as tool support, reuse across test cases, and cost-benefit analysis. To summarise, the existing studies on the reuse of AAT have shown that:

- each AAT framework uses different types of text-based AAT types (e.g., BDD, Story-driven, etc.).

- a reuse process for AATs is needed that covers identification, relevance, and adaptation of AATs for reuse. 
- AATs are costly to write and reuse, and reuse cost of AATs should be considered when reusing AATs.

In this study, we will attempt to address the above issues by providing an integrated solution that can enable practitioners to reuse AATs cost-efficiently.

Optional Reading - You can know more about the keep concepts associated with our research by accessing the following sources:
- Acceptance Testing: https://www.agilealliance.org/glossary/acceptance/
- Acceptance Testing Example:  https://semaphoreci.com/community/tutorials/introduction-to-writing-acceptance-tests-with-cucumber

Background

Loading…

The Proposed Process (*Zoom if not readable)

Image title

The Proposed Process (*Zoom if not readable)

Loading…

Description of Activities in the Process (* Zoom if the text is not readable)

Image title

Description of Activities in the Process (* Zoom if the text is not readable)

Loading…

Tables to be used for Example Below

Image title

Tables to be used for Example Below

Loading…

Example - Development of ATT for login system using our proposed process

Image title

Example - Development of ATT for login system using our proposed process

Loading…

Survey Questions

Title

Survey Questions

Loading…

In your opinion, what are the benefits of using the proposed process?

Question

Question Type

In your opinion, what are the benefits of using the proposed process?

Required

Loading…

What are the drawbacks/limitations of the process? How can we improve/revise the process?

Question

Question Type

What are the drawbacks/limitations of the process? How can we improve/revise the process?

Required

Loading…

In your opinion, how easy is it to use this process?

Question

Question Type

In your opinion, how easy is it to use this process?

Very Easy

Very Difficult

Label (optional)

Required

Loading…

Do you have any recommendations to improve the ease of use?

Question

Question Type

Do you have any recommendations to improve the ease of use?

Required

Loading…

Are there steps that should be removed from the process? If yes, then kindly list those items here and also state why do you recommend removing them?

Question

Question Type

Are there steps that should be removed from the process? If yes, then kindly list those items here and also state why do you recommend removing them?

Required

Loading…

Are there any other steps that should be added to the process?

Question

Question Type

Are there any other steps that should be added to the process?

Required

Loading…

4 responses

Not accepting responses

Accepting responses

Message for respondents

This form is no longer accepting responses

Summary

Question

Individual

Insights

Total points distribution

Name4 responses

Muhammad Waqas

Nayla Nasir

Hassan Azhar

Nasir Mehmood Minhas

Company's Name4 responses

Value	Count
BTH	2
Soliton Technologies	1
Telenor Sweden	1

Role (e.g., Developer, Tester, Product Manager)4 responses

Tester

Testing researcher

Developer

Software Testing Researcher

Objective

Background

Survey Questions

In your opinion, what are the benefits of using the proposed process?4 responses

In my point of view this process will really help the software practitioner to select the test cases and use it for automated acceptance tests. The use of this process can be cost effective in sense of saving time by selecting the test cases with respect to the score of each test.

The possible benefit of the process is the reduction in the effort to write the new test case.

In case of low assess relevance and less cost of reuse, it will improve the quality of the tests by having already reliable tests. Also, it'll reduce the time to test a functionality that is closer to already existing and selected TC.

It guides a systematic approach, and it will help in optimizing the AAT process. Provided if it does not involve additional execution cost and the process is automated.

What are the drawbacks/limitations of the process? How can we improve/revise the process? 4 responses

The only thing I have found its hard to implement this process in the existing project because it takes time to change the process and this process has involved many steps but this one time cost of implementation can save a lot in future.

1) If the reuse practices are not defined, the test developer has to wait for the actions A2 and A3 to be performed at organizational level. It will cause a delay in test development. 2) If estimitating the relevance and the costs is not automated, then the process it self will contribute to the test development costs. 3) It is not clear enough if the process caters for synonyms while evaluating relevance.i.e. If a similar requirement is written using a different set of words, how the relevance is measured.

In case of not having good assess relevance and cost of reuse, this process will take some extra time to implement new TCs as step to write new TCs will start after several assessments steps.

Some parts of the process are subject to assessment, for instance, relevance and cost. It is difficult to judge how accurate and robust these parts are. Furthermore, while comparing the cost of reuse vs writing of new test cases, the authors missed the segment of evaluating the cost of writing new test cases. It is also not clear if the synonyms are considered in the keyword-based relevance assessment. Otherwise, there is a chance that the process measure low or zero relevance between similar test cases. For instance, an existing test case is, "As a user, I should be able to delete the product". Suppose that the new test case is that "As a user, I should be able to remove any item from the product table. Now the question is that, how relevant are these two for the relevance assessment module?

In your opinion, how easy is it to use this process? 4 responses

Value	Count
1	0
2	1
3	2
4	1
5	0

Do you have any recommendations to improve the ease of use?4 responses

not i don't have any better idea to make this process more better.

The relevance and cost estimation should be automated. The increasing number of reusable artifacts will affect the efficiency of search. So some mechanisms should be introduced to speed up the search, e.g indexing etc.

Overall, I don't find any drawback in this process, except with a question, what will happen if a TC has good assess relevance but high cost to reuse as compared to not so good assess relevance but has lower cost to reuse. e.g. To delete a product you have to be a registered user and you have to view the product. but a closely related TC says that a person can view product without registration. (Just a suggestion, overall I don't find any lacking step.)

The process seems to be simple and easy to use, provided if process tasks are automated.

Are there steps that should be removed from the process? If yes, then kindly list those items here and also state why do you recommend removing them?4 responses

No according to the process diagram there is no extra step that we can remove it.

Steps A2 and A3 can cause possible delay, so they should be a pre-requisite to the process, and not a part of the process.

Organizational level activities should not be the part of process, rather these should be the prerequisites of the process.

Are there any other steps that should be added to the process?4 responses

If there are more than one reusable artifacts with high relevance, then there should be some selection criteria to use one of them. For instance in the given examples, if NCD value is same for two test cases, then there should be a mechanism to select one of them.

Reassessment after cost evaluation with other closely related test cases.

The module that evaluates the cost of writing new test cases should be part of the process. It should be clear how the test repository would be managed, and what if when it grows larger? Because selecting relevant test cases could become a bottleneck with a poorly managed repository.

Responses cannot be edited

The Reusability of Automated Acceptance Tests

Author 1: Mohsin Irshad (mir@bth.se,* contact for any queries)

Name *

Nayla Nasir

Company's Name *

BTH

Role (e.g., Developer, Tester, Product Manager) *

Testing researcher

Objective

The objective of this study is to propose and evaluate an integrated solution to assess the reusability of automated acceptance tests.

Background

An acceptance test is used to verify that the requirements are developed according to the contract/agreed specification. The acceptance tests are derived from the customer requirements, and their focus is to verify the completeness of the implementation of requirements. It was proposed that customers specify these acceptance tests, and these tests can be used as a form of requirements. However, researchers believe that the development of an acceptance test should be left to the software practitioners as it is difficult to involve customers in the development process. The acceptance tests are often automated in agile methodologies like XP, as part of the build cycle. These are executed at different levels of software development life-cycle, e.g., functional testing, user acceptance testing, etc., .

Recently acceptance test-driven development has gained popularity, and several variants of acceptance test-driven development have been introduced, e.g., Behavior Driven Development (BDD), Story-Driven Development (SDD), Key-word Driven Development, FIT Tables, and specification by example. These test-driven development methodologies specify the AATs in natural language format, e.g., feature files or text files in BDD, and stories in SDD. However, FIT tables are not specified in a natural language format. A recent literature review found that the majority of the studies present in literature are based on the FIT framework (21 out of 26 identified studies). Software practitioners use various tools and technologies to develop these test cases such as Cucumber, robot framework, and FitNesse.

Angmo and Sharma evaluated AAT tools and proposed that cost-benefits analysis should be done when considering a tool for automated acceptance tests. Haugset and Stalhane claim that AAT can provide cost-related benefits in two ways (i) correctness of requirement, and, (ii) automated test-ability. However, they suggest that the software community believes that the costs of AATs are greater than the benefits of AATs. Borg and Kropp described a tool that helps in the maintenance, and refactoring automated acceptance tests, reducing their maintenance costs.

In short, research on AATs is still new, and work is further required to evaluate aspects such as tool support, reuse across test cases, and cost-benefit analysis. To summarise, the existing studies on the reuse of AAT have shown that:

- each AAT framework uses different types of text-based AAT types (e.g., BDD, Story-driven, etc.).

- a reuse process for AATs is needed that covers identification, relevance, and adaptation of AATs for reuse.
- AATs are costly to write and reuse, and reuse cost of AATs should be considered when reusing AATs.

In this study, we will attempt to address the above issues by providing an integrated solution that can enable practitioners to reuse AATs cost-efficiently.

Optional Reading - You can know more about the keep concepts associated with our research by accessing the following sources:
- Acceptance Testing: https://www.agilealliance.org/glossary/acceptance/
- Acceptance Testing Example: https://semaphoreci.com/community/tutorials/introduction-to-writing-acceptance-tests-with-cucumber

The Proposed Process (*Zoom if not readable)

Description of Activities in the Process (* Zoom if the text is not readable)

Tables to be used for Example Below

Example - Development of ATT for login system using our proposed process

Survey Questions

This part is very important, as your suggestions will guide us in improving the proposed process. We have tried to identify performance expectancy, effort expectancy and facilitating conditions of this process.

In your opinion, what are the benefits of using the proposed process? *

The possible benefit of the process is the reduction in the effort to write the new test case.

What are the drawbacks/limitations of the process? How can we improve/revise the process? *

In your opinion, how easy is it to use this process? *

Very Easy

Very Difficult

Do you have any recommendations to improve the ease of use? *

Are there steps that should be removed from the process? If yes, then kindly list those items here and also state why do you recommend removing them? *

Steps A2 and A3 can cause possible delay, so they should be a pre-requisite to the process, and not a part of the process.

Are there any other steps that should be added to the process? *

Submitted 7/16/20, 7:31 PM