Published September 24, 2021 | Version v1
Other Open

TestAxis Replication Package



TestAxis Replication Package

Replication package for the evaluation experiment of TestAxis: a CI build test result inspection tool.

TestAxis brings CI build results to the IDE to avoid scrolling through hundreds to thousands of lines of log output.

In addition to the build results, it offers inspection tools, such as the display of the changed code under test, based on data that is too expensive to collect in a local development environment.

We conducted a user study with 16 participants to evaluate the usefulness and performance improvements of TestAxis.


The package contains all resources used to set up the experiment, the results of the questionnaires, and the timing results of the programming assignments.

Moreover, the package includes the scripts to generate plots for this data.


Assignment Ordering

The order of the assignments is pre-generated per participant using a script that verifies a fair and balanced distribution, see `assignments/`.

Based on the order, we generate PDFs with all assignment descriptions in the right order with a mark of the right variant for each participant using `assignments/`.

`assignments/` generates a separate assignment description PDF for each assignment variant.

The output of these scripts that we used in the experiment can be found in `assignments/output.pdf`.



Raw Results

The raw results are available in `results/input`.

Analysis Notebooks

The notebooks we used to explore the raw data are available in `results/analysis`.

Thesis Report Plots

To generate all plots that can be found in the thesis report, run `results/report/`.

This script will generate all plots defined in `results/report/plots` and output them to `results/report/output`.

Experiment Notes

The notes of the experiments can be found in `notes/`.

In the notes, we denote the participant by _P_ and the researcher by _C_.

Experiment Support Tool

To support smooth transitions between the different elements of the script and between assignments, we created an application that could run actions on the experiment machine remotely.

This way we could show the explanation videos or one of the questionnaires with a single button press from another computer.

The application also supported assignment switching which would reset the state of the IDE by closing any windows or tabs, undoing code changes, and checking out the right branch.

It also shows the assignment description PDFs to the participant.

Furthermore, it offers the ability to fire TestAxis CI build notifications for predefined builds corresponding to the assignments.

The application also showed the pre-generated assignment order for the participants.

The server controlled the experiment machine through AppleScripts, git interaction, and TestAxis API communication.


The client application is a Vue application built using Vite and can be found in `experiment-support-tool/dashboard`.

The server-side application for the experiment machine is a node express application that enables a form of remote method invocation and can be found in `experiment-support-tool/experiment-machine`.


Files (31.4 MB)

Name Size Download all
31.4 MB Preview Download