Boolean condition coverage with evaluation trees

This is an instance of the LabPal experimental environment. It contains experiments aimed at comparing a new technique for Boolean test input generation to existing tools and algorithms. The artifact has been accepted as complementary material for the paper entitled Test Suite Generation for Boolean Conditions with Equivalence Class Partitioning, published at the FormaliSE 2022 conference.

Context

Consider a Boolean expression φ formed of the traditional connectives ∧ ("and"), ∨ ("or") and ¬ ("not"). In this context, a test case is a particular way of assigning the values true and false to the variables contained in the formula. A test suite is a set of test cases. The goal of Boolean test input generation is to generate a test suite that satisfies a specific coverage criterion. For example, a criterion could be that for each variable x, there exist test cases where x takes both the value true and false.

Many coverage criteria have been suggested in the past: MC/DC coverage, MUMCUT coverage, combinatorial or "t-way" coverage, clause coverage, etc. Algorithms and tools have been developed to generate test suites satisfying each of these criteria, ideally using the fewest possible test cases.

In this benchmark, we compare a new technique for generating test suites according to a coverage criterion and a Boolean formula, using the concept of tree transformation and a reduction to the problem of hypergraph vertex covering. The benchmark runs this algorithm on multiple formulas and multiple criteria and measures the size of the generated test suite as well as the time taken to produce it. It also does the same thing for other tools:

• ACTS can be used to generate test suites following combinatorial coverage
• mcdc can be used to generate test suites following MC/DC coverage
• in addition, the lab contains a test suite generator that randomly picks test cases, to be used as a form of baseline

The benchmark also contains results from this publication for MUMCUT coverage:

• T. Chen, M. Lau, Y. Yu. (1999). MUMCUT: a fault-based strategy for testing Boolean specifications. In Proc. APSEC, pp. 606–613. IEEE Comp. Soc.
• Y. Yu, M.F. Lau, T.Y. Chen. (2006). Automatic Generation of Test Cases from Boolean Specifications using the MUMCUT Strategy. J. of Systems and Software, 79 (5), pp. 820-840.

And for MC/DC coverage:

• T. Kitamura, Q. Maisonneuve, E. Choi, C. Artho, A. Gargantini. (2018). Optimal Test Suite Generation for Modified Condition Decision Coverage Using SAT Solving. In Proc. SAFECOMP, Springer LNCS 11093, pp. 123-138.

For all input problems where a comparison is possible, the benchmark dynamically produces tables and plots showing the respective test suite size/generation time of the hypergraph technique vs. its "competitors", among other things.

Contents of this artifact

This artifact contains a file called Formalise22.ova, which a virtual machine image with a small Linux installation called SliTaz. You can import this virtual machine and run it using either VirtualBox or VMWare. You need a 64-bit host operating system; the machine is configured to use a maximum of 2 GB of RAM.

Boot using the default options. Login with user "tux" and password "tux". If you ever need root access (which you shouldn't), the root password is "root". On the desktop, head for the "Documents" folder, which contains a file called Readme.txt for instructions on what to do inside the VM.