Abstraction-Aware Inference of Metamorphic Relations (Replication Package)

Replication package for the paper: Abstraction-Aware Inference of Metamorphic Relations. Agustin Nolasco, Facundo Molina, Renzo Degiovanni, Alessandra Gorla, Diego Garbervetsky, Mike Papadakis, Sebastian Uchitel, Nazareno Aguirre, and Marcelo F. Frias. Accepted for presentation at the ACM International Conference
on the Foundations of Software Engineering (FSE 2024)

Getting started

Uncompress the downloaded file in your working directory. You will see four folders:

• epa-benchmark-subjects: the set of subjects used in the evaluation of the technique.

• metamorphic-relations-inference: implementation of the 1st phase of the technique (candidate generation of MRs using fuzzing and EPAs). 

• randoop-mr-inference: implementation of the 2nd phase of the technique (run-time checking of the candidate MRs. This also includes the code to perform mutation analysis in our technique and SBES).

• alloy-mr-reducer: implementation of the last phase of the technique (using SAT to reduce the set of "valid" MRs).

Requirements

• Java 1.8

• Gradle

• Python3 (make sure you have installed pandas)

Installation

Install the requirements and run:

$ ./setup.sh

this script will automatically generate the init_env.sh file (needed to run the experiments), and also will compile all the subjects and the three-steps tool (EPA-Aware fuzzing, dynamic analysis, SAT-reduction).

A Running Example

Once installed, you can infer metamorphic relations for MyBoundedStack class (motivating example of our paper) running the following commands:

1. From the metamorphic-relations-inference folder run:

$ ./fuzz.sh EPA MyBoundedStack 1000

This will produce EPA-Aware metamorphic relation candidates in the output folder metamorphic-relations-inference/output.

2. The following step uses Randoop in order to check whether de candidates are valid properties. To do so, inside the randoop-mr-inference folder run:

$ ./validate.sh EPA MyBoundedStack 1000

After run this command you will find the detailed output in randoop-mr-inference/output/MyBoundedStack/ allow_epa_loops_true/EPA_AWARE/1000/log.txt. There you will see what happened with each candidate and, at the end of the file, you will find the metamorphic relations that could not be invalidated by the process.

3. The last step is the SAT-based reduction. To execute it , inside alloy-mr-reducer run:

$ ./reduce.sh MyBoundedStack 1000

In alloy-mr-reducer/output/MyBoundedStack/allow_epa_loops_true/EPA_AWARE/1000/mrs.txt you will find the reduced set of metamorphic relations.

Experiments

To reproduce the experiments performed in our paper, follow the instructions below:

This RQ analyses the effectiveness of the EPAs for discarding invalid MRs. To reproduce the experiment, in the root directory, run:

$ ./experiment-rq1.sh

The output can be found in metamorphic-relations-inference/output/epa_filter_analysis_result.csv 

For the assessment of the SAT-based reduction and the comparison with the state of the art (SBES), we use mutation analysis in order to quantify the fault detection capabilities of our technique without reduction and using it. Finally we compare in the same way with SBES.

To reproduce this experiments, in the root directory, run:

$ ./experiment-rq2-3.sh EPA MyBoundedStack

This will only compute the results for the row of MyBoundedStack of Table 3. The other subjects can be found in epa-benchmark-subjects/subjects/.
Note that the subjects of SBES must be run as follow:

$ ./experiment-rq2-3.sh SBES Stack