Data and Code for the paper "An Empirical Study on Exploratory Crowdtesting of Android Applications"

This package contains data and code to replicate the findings presented in our paper titled " An Empirical Study on Exploratory Crowdtesting of Android Applications".

Abstract

Crowdtesting is an emerging paradigm in which a ``crowd'' of people independently carry out testing tasks, and proved to be especially promising in the mobile apps domain and in combination with exploratory testing strategies, in which individual testers pursue a creative, experience-based approach to design tests.

Managing the crowdtesting process, however, is still a challenging task, that can easily result either in wasteful spending or in inadequate software quality, due to the unpredictability of remote testing activities. A number of works in the literature investigated the application of crowdtesting in the mobile apps domain. These works, however, consider crowdtesting tasks in which the goal is to find bugs, and not to generate a re-executable test suite. Moreover, existing works do not consider the impact of the application of  different exploratory testing strategies.

As a first step towards filling this gap in the literature, in this work, we conduct an empirical evaluation involving four open source Android apps and twenty masters students, that we believe can be representative of practitioners partaking in crowdtesting activities. The students were asked to generate test suites for the apps using a Capture and Replay tool and different exploratory testing strategies. We then compare the effectiveness, in terms of aggregate code coverage, that different-sized crowds of students achieve using different exploratory testing strategies. Results provide useful insights to project managers interested in using crowdtesting to produce GUI test suites for mobile apps, on which they can make more informed decisions.

Contents and Instructions

This package contains:

• apps-under-test.zip A zip archive containing the source code of the four Android applications we considered in our study, namely MunchLife, TippyTipper, Trolly, and SimplyDo.
• students-test-suites.zip A zip archive containing the test suites developed by the students using Uninformed Exploratory Testing (referred to as "Black Box" in the subdirectories) and Informed Exploratory Testing (referred to as "White Box" in the subdirectories). This also includes coverage reports.
• compute-coverage-unions.zip A zip archive containing Python scripts we developed to compute the aggregate coverage of all possible subsets of students. The scripts have been tested on MS Windows. To compute the code coverage achieved by any possible subsets of testers using IET and UET strategies, run the analysisAndReport.py script. To compute the code coverage achieved by mixed crowds in which some testers use a IET approach and other use a UET approach, run the analysisAndReport_UET_IET_combinations_emma.py script.
• data-analysis-scripts.zip A zip archive containing a RStudio project and all the R scripts we developed to carry out statistical analysis and draw plots. All data concerning RQ1 and RQ2 is available as a ready-to-use R dataframe in the ./data/ce/data_augmented.rds file. All data concerning RQ3 is available in the ./data/all_strategies.csv file. Moreover, the hypotheses_testing.R script performs statistical tests and measures effect size for RQ1. The script hypotheses_testing_across_strategy.R performs statistical tests and measures effect size for RQ2.