Artifact of Mastery paper

Evaluation of mergers

This guide describes how to repoduce the evaluation described in our paper. We evaluate five tools: Mastery, IntelliMerge, JDime, jFSTMerge and GitMerge.

Set up docker

Firstly, you need to download the provided tar archive (mastery-artifact-image.tar), which is a Docker image containing all of the tools, environments and merge scenarios to reproduce the evaluation. You could download Docker from here. Once you have Docker installed, the next step is to load the image from this tar archive and run the image as a container.

# load image
docker load -i mastery-artifact-image.tar

# run image
docker run --name mastery-artifact-container -it mastery-artifact-image

This will drop you into a shell inside the container. The rest of this guide will assume that you run commands from the ~ directory -- things may not work as expected otherwise.

We have installed vim in the image for convenience to edit and view files. You can use Docker command cp to copy files into and out of the container.

File structure

• bin/: jar files of semistructured and structured merging tools
• mastery/: source code of mastery
• merge-scenarios/: the dataset (i.e. all considered merge scenarios) of our evaluation
• scripts/: scripts to run the experiments and show the results
• README.md: this guidance

Reproduce evaluation

To run all mergers on all of our merge scenarios, you can use the following command:

python3 scripts/run.py

As we consider a large dataset, this execution may cost hours or days, please be patient. For speeding up, this script will run on about 3/43 / 43/4 CPUs in parallel. Some error messages may be shown when a merger fails on a merge scenario, which doesn't disturb the functionality of the script.

The raw results will be located in ~/merge-scenarios-results/. We will analyze the results in metrics mentioned in our paper by ~/scripts/summarize.py.

For a more customized reproducing, see also python3 scripts/run.py -h and python3 scripts/summarize.py -h.

To summarize and show the results, you can use the following command:

python3 scripts/summarize.py

Two tables similar to the followings will be print. The first table replicates Table 2 and Figure 6. The second one replicates Figure 8 and Figure 9.

config: Mastery found
config: JDime found
config: jFSTMerge found
config: IntelliMerge found
config: GitMerge found
40533 benchmarks found

    Tool         Expected       Unexpected     Conflicting      Failed
------------  --------------  --------------  -------------  ------------
  Mastery     33602 (82.90%)   3148 (7.77%)   3782 (9.33%)    1 (0.00%)
   JDime      32200 (79.44%)   2406 (5.94%)   4538 (11.20%)  1389 (3.43%)
 jFSTMerge    30062 (74.17%)   3837 (9.47%)   6626 (16.35%)   8 (0.02%)
IntelliMerge  9777 (24.12%)   24553 (60.58%)  3442 (8.49%)   2761 (6.81%)
  Gge    30643 (75.60%)   791 (1.95%)    9099 (22.45%)   0 (0.00%)

    Tool       Conflict Block    Time (s)
------------  ----------------  ----------
  Mastery           7057          10.16
   JDime           11319          25.11
 jFSTMerge         12744          13.73
IntelliMerge        5655           4.3
  GitMerge         17314           0.01