Running the experiments

Prerequisites for running the experiments:

• Docker
• Poetry

Steps to run the experiments:

1. Download the experiment zip file you wish to run (single_parameter_experiment.zip or multi_parameter_experiment.zip).
2. Un-zip the file.
3. Open a terminal and navigate to the unzipped folder (e.g. cd single_parameter_experiment).
4. Run poetry install --only main to install all dependencies.
5. To run the experiment, run poetry run python run_experiment.py.
6. All results can be found in the folder data/.

The modules used for the experiment are defined in the file experiment_modules.py and to see the experiment configuration, look in run_experiment.py.

Warning: the experiments take several weeks to run on a single machine, therefore it is advisable to split the experiments based on modules and run them in parallel.

Running the analysis

Prerequisites for running the analysis:

• Conda

Steps to run the analysis:

1. Download the analysis zip file (analysis-adaptive-parameter-control.zip).
2. Un-zip the file.
3. Open a terminal and navigate to the unzipped folder (e.g. cd analysis-adaptive-parameter-control).
4. Run the following command to install the conda environment and all dependencies: conda env create -f environment.yml
5. If you want to re-run the Bayesian models locally on your machine, follow the optional step below, otherwise download and unzip the trace data from the replication package, i.e., Trace data single.zip and Trace data multi.zip. 
6. Place the .nc files in the corresponding folder: analysis-adaptive-parameter-control/single_parameter/ or analysis-adaptive-parameter-control/multi_parameter/.
   1. E.g. the coverage_rate_model_single_parameter.nc goes in the single_parameter folder, while the  coverage_rate_model_multi_parameter.nc goes in the multi_parameter folder.
7. Navigate to the notebooks folder (Notebooks/).
8. Open a notebook of choice (coverage_rate_multi_parameter.ipynb, coverage_rate_single_parameter.ipynb, final_coverage_multi_parameter.ipynb, final_coverage_single_parameter.ipynb, overhead_model_multi_parameter.ipynb, or overhead_model_single_parameter.ipynb).
9. Navigate to the section called "Data analysis" and run all cells in order.

(Optional) Running the Bayesian models locally before the analysis.

1. Navigate to the notebooks folder (Notebooks/).
2. Open a notebook of choice (coverage_rate_multi_parameter.ipynb, coverage_rate_single_parameter.ipynb, final_coverage_multi_parameter.ipynb, final_coverage_single_parameter.ipynb, overhead_model_multi_parameter.ipynb, or overhead_model_single_parameter.ipynb).
3. Navigate to the section called "Model specification" and run the three notebook cells.

Warning: this will take a long time, if you don't have the time, use the following alternative instead

Data

The data from when we ran the experiments is available in the Single data.zip and Multi data.zip files.

The structure of these are the following:

• There are folders for each module the experiment was run on, further divided into each unique run. All these folders include: 
  • Coverage reports.
  • Complete logs for the unique run.
  • A timeline over controlled parameter values during the test generation process.
  • The complete Pynguin configuration for the run.
  • The generated test suite.
• There is one statistics.csv file containing some information about each run and their branch coverage timelines.

Running the parameter assignment analysis

Prerequisites for running the parameter assignment analysis:

• Conda

Steps to run the parameter assignment analysis:

1. Download the analysis zip file (parameter-assignment.zip).
2. Un-zip the file.
3. Open a terminal and navigate to the unzipped folder (e.g. cd parameter-assignment).
4. Run the following command to install the conda environment and all dependencies: conda env create -f environment.yml
5. Navigate to the notebooks folder (Notebooks/).
6. Open the notebook parameter_assignment_analysis.ipynb.
7. Run all cells in order.