Adequate vs. Inadequate Test Suite Reduction Approaches. Raw Data

Context: Regression testing is an important activity that allows ensuring the correct behavior of a system after a change. As the system grows, the time and resources to perform regression testing increase. Test Suite Reduction (TSR) approaches aim to speed up regression testing by removing obsolete or redundant test cases. These approaches can be classified as adequate or inadequate. Adequate TSR approaches reduce test suites and completely preserve test requirements (e.g., covered statements) of the original test suites. Inadequate TSR approaches do not preserve test requirements. The percentage of satisfed test requirements indicates the inadequacy level.

Objective: We compare some state-of-the art adequate and inadequate TSR approaches with respect to the size of the reduced test suites and their fault-detection capability. Specifcally, we aim to increase our body of knowledge on TSR approaches by performing the following comparisons: (i) well-known adequate TSR approaches; (ii) their inadequate variants; and (iii) several variants of a novel Clustering-Based (CB) approach for (adequate and inadequate) TSR.

Method: We conducted an experiment to compare adequate and inadequate TSR approaches and this comparison is founded on a public dataset containing information on real faults.

Results: The most important findings from our experiment can be summarized as follows: (i) there is not an inadequate TSR approach that performs better than others; (ii) some variants of the CB approach, and a few well-known inadequate approaches, outperform the adequate ones in terms of the reductions in test suite size with a negligible, or no, eect on fault-detection capability; and (iii) the CB approach is less sensitive than the other inadequate approaches, that is, variations in the inadequacy level have a small effect on the reduction in test suite size and on the loss in fault-detection capability.

Conclusions: These findings imply that inadequate TSR approaches and especially the CB approach might be appealing because they lead to a greater reduction in test suite size (with respect to the adequate ones) at the expense of a small loss in fault-detection capability.