# Artifact for the paper "Not In My Git Yard: Catching Backdoors at Commit and Release Time" (ASE'26) This artifact enables evaluating Lily, a new approach and tool for preventing backdoor injections in the CI and release vetting pipelines. More precisely, this artifact makes it possible to reproduce the evaluation results reported in our ASE'26 paper about Lily, as they are detailed in tables 3–6 of this paper. ## Table of contents 1. [Getting started](#getting-started) 1. [Installing the Docker image](#installing-the-docker-image) 2. [Uninstalling the Docker image](#uninstalling-the-docker-image) 3. [Smoke test](#smoke-test) 2. [Step-by-step reproduction](#step-by-step-reproduction) 1. [Guided partial reproduction](#guided-partial-reproduction) 2. [Unguided partial reproduction (optional)](#unguided-partial-reproduction-optional) 3. [Complete reproduction (optional)](#complete-reproduction-optional) 4. [Discovering how Lily can be reused or repurposed](#discovering-how-lily-can-be-reused-or-repurposed) 3. [Documentation on the artifact code and layout](#documentation-on-the-artifact-code-and-layout) ## Getting started ### Installing the Docker image The image can be loaded transparently via [Docker Hub](https://hub.docker.com): ```console docker pull plumtrie/lily-ase26-artifact:0.2.5 ``` **Note for Apple Silicon users: the base image is built for x86_64 and has known issues on Apple Silicon. Run the following command instead:** ```console docker pull plumtrie/lily-ase26-artifact_apple-silicon:0.2.5 ``` As a more durable alternative, it can also be downloaded from the Zenodo archive and then loaded locally. For example, using [curl](https://curl.se/): ```console curl https://zenodo.org/records/21390730/files/lily-ase26-artifact_docker-image_0-2-5.tar | docker load ``` **Note for Apple Silicon users: the base image is built for x86_64 and has known issues on Apple Silicon. Run the following command instead:** ```console curl https://zenodo.org/records/21390730/files/lily-ase26-artifact_apple-silicon_docker-image_0-2-5.tar | docker load ``` Be aware that loading the image takes several minutes and might use considerable resources. Once the image has been loaded, a container can be started using the following command, **assuming you have 16 GiB of free RAM to dedicate to the evaluation of the artifact**: ```console docker run -ti --rm --name lily-ase26-artifact --publish 4000:4000 --tmpfs /root/scratch:size=16g plumtrie/lily-ase26-artifact:0.2.5 ``` **Note for Apple Silicon users: the base image is built for x86_64 and has known issues on Apple Silicon. Run the following command instead:** ```console docker run -ti --rm --name lily-ase26-artifact --publish 4000:4000 --tmpfs /root/scratch:size=16g plumtrie/lily-ase26-artifact_apple-silicon:0.2.5 ``` If you do _not_ have 16 GiB of free RAM, you can start the container using the following command, but **be aware that it might impact the evaluation results**: ```console docker run -ti --rm --name lily-ase26-artifact --publish 4000:4000 plumtrie/lily-ase26-artifact:0.2.5 ``` **Note for Apple Silicon users: the base image is built for x86_64 and has known issues on Apple Silicon. Run the following command instead:** ```console docker run -ti --rm --name lily-ase26-artifact --publish 4000:4000 plumtrie/lily-ase26-artifact_apple-silicon:0.2.5 ``` This will immediately start an interactive session within the container, from which you can start using the artifact. Note that you should also see the message `Go to http://localhost:4000/toolchain/rosa_filter_diff.html to see the documentation for Lily.`. Indeed, since the previous command binds the host port 4000 (i.e., on your host machine) to the guest port 4000 (i.e., inside the container), you can consult the documentation of Lily (and the entire ROSA toolchain, as Lily reuses some of its components) with a web browser on your host machine. This is entirely optional, but it might help you understand how Lily works. ### Uninstalling the Docker image Once you are done using the artifact, you can dispose of it by simply removing the Docker image. Make sure to exit any running container(s) associated with the artifact's Docker image, and then run: ```console docker rmi plumtrie/lily-ase26-artifact:0.2.5 ``` For Apple Silicon users, run: ```console docker rmi plumtrie/lily-ase26-artifact_apple-silicon:0.2.5 ``` ### Smoke test To test that the container works as intended, you can run the following command **in the container**: ```console RUNS=1 SECONDS_PER_RUN=10 /root/artifact/run-experiment.sh /root/experiments/backdoored-releases/vsftpd/vsftpd__backdoored-release-3.0.3.sh ``` It should take less than 5 minutes to complete. This will run one of the experiments from the Lily paper (specifically, one of the _backdoored release_ simulations for the vsFTPd program), with a single trial, and ten seconds of fuzzing. After the experiment is done, the script should compile the results to a PDF file and print `Done! PDF available at: /root/evaluation/vsftpd__backdoored-release-3.0.3.pdf`. You should run the following command **on the host machine** to retrieve the generated PDF: ```console docker cp lily-ase26-artifact:/root/evaluation/vsftpd__backdoored-release-3.0.3.pdf . ``` **If the smoke test fails, and you encounter the message `ERROR: could not find ROSA's trace marker` in the generated logs, you are most likely running the Docker image built for x86_64 on an Apple Silicon device. Please follow the instructions for Apple Silicon in [_Installing the Docker image_](#installing-the-docker-image).** You should open and examine the PDF; it should contain the four relevant tables from the paper (tables 3–6), with one table per page. Tables 4–6 will contain "/0" in their data, as the experiment which runs during the smoke test does not cover them. It is fully okay if the numbers in the tables are not the same as those in the paper; in fact, they most likely will not be, as 10 seconds is usually not enough for Lily to detect this specific backdoor. If you have made it thus far without errors, then you have configured the artifact correctly. If there were any errors along the way, please contact the authors. ## Step-by-step reproduction ### Guided partial reproduction In our ASE'26 paper, we perform 20 trials of 10-minute backdoor detection campaigns for 545 version pairs across 13 different software projects. A complete reproduction of this process (see [_Complete reproduction_](#complete-reproduction)) takes several months without massive parallelization of the experiments. **The goal of the proposed _Guided partial reproduction_ is to enable a quick (a few hours) reproduction of the trends observed in the paper for each of the four different use cases studied in the paper** (_backdoored_ or _safe_ commit, _backdoored_ or _safe_ release). More precisely, the _Guided partial reproduction_ only performs two trials of 10-minute backdoor detection campaigns for the following version pairs: - Sudo: commit `ce0ec8ddca`, commit `ce0ec8ddca` **with an added synthetic backdoor** - vsFTPd: release `3.0.3`, release `3.0.3` **infected with authentic backdoor (real-world attack)** - ProFTPD: commit `ce3a4d3dc2`, commit `d17903c879` (no backdoors) - libpng: release `v1.6.37`, release `v1.6.39` (no backdoors) Once an interactive session has been started within the container (see [_Installing the Docker image_](#installing-the-docker-image)), you can start the _Guided partial reproduction_ by running the following command: ```console /root/artifact/run-reduced-evaluation.sh ``` This should take a few hours to complete. When finished, the last printed message should read `Done! PDF available at: /root/evaluation/reduced-evaluation.pdf`. If any warning messages appear, it is possible that some of the experiments failed, most likely due to insufficient disk size or platform incompatibilities (see [REQUIREMENTS.md](./REQUIREMENTS.md)). In that case, the resulting PDF will contain _partial_ results (i.e., including only the experiments which succeeded). **From your host machine**, you should run the following command to retrieve the PDF from the container: ```console docker cp lily-ase26-artifact:/root/evaluation/reduced-evaluation.pdf . ``` You can then compare the results in the generated PDF to tables 3–6 in the paper. In particular, verify the following elements: - Table 3: - Only two targets (Sudo and vsFTPd) should be present, as they are the only targets in the _backdoored_ commit/release use case examined in the _Guided partial reproduction_. - For Sudo (_backdoored commit_): the first column should show "2/2", indicating that Lily detected the backdoor during both trials. The rest of the row should not show any data (—), as the results contain no _backdoored release_ experiments for Sudo. - For vsFTPd (_backdoored release_): the first column should not show any data (—), as the results contain no _backdoored release_ experiments for vsFTPd. The second and third column should both show "1/1", indicating that the backdoored release was successfully detected by Lily. The fourth column should show "2/2", indicating that Lily detected the backdoor during both trials. - For the last row: the total of each metric should follow the same trend as observed in the paper. - Table 4: - Only two targets (ProFTPD and libpng) should be present, as they are the only targets in the _safe_ commit/release use case examined in the _Guided partial reproduction_. - For ProFTPD (_safe commit_): the first and second column should show "0/1" for Lily (and potentially "1/1" for Atypical Lily and/or Novel Lily), indicating that Lily did not emit any false alarms for the safe commit. The third column should show "0/2" for Lily (and potentially "1/2" or "2/2" for Atypical Lily and/or Novel Lily), indicating that Lily did not emit any false alarms during either of the two trials. The rest of the row should not show any data (—), as the results contain no _safe release_ experiments for ProFTPD. - For libpng (_safe release_): the first three columns should not show any data (—), as the results contain no _safe commit_ experiments for libpng. The fourth and fifth columns should show "0/1" for Lily (and potentially "1/1" for Atypical Lily and/or Novel Lily), indicating that Lily did not emit any false alarms for the safe release. The sixth column should show "0/2" for Lily (and potentially "1/2" or "2/2" for Atypical Lily and/or Novel Lily), indicating that Lily did not emit any false alarms during either of the two trials. - For the last row: the total of each metric should follow the same trend as observed in the paper. - Table 5: - Only one target (vsFTPd) should be present, as it is the only target in the _backdoored release_ use case examined in the _Guided partial reproduction_. - For vsFTPd (and the total in the last row): the difference between the first (average number of lines in the release diff) and second (average number of lines in the report generated by Lily) column should be similar to the one in the paper. It might not be _exactly_ the same, as the exact number of lines in the report generated by Lily depends on the specific inputs of each detection, and as the paper contains an average of multiple trials across multiple releases (whereas the _Guided partial reproduction_ only examines two trials of a single release). - Table 6: - Top part (_backdoor detection rate_): comparing Lily and LilySelective should yield conclusions similar to those in the paper. Namely, LilySelective should detect backdoors in as many or more cases than Lily (i.e., more commits/releases _and_ more individual runs should be correctly blocked), thanks to the pruning mechanism described in the paper. - Bottom part (_false alarm rate_): comparing Lily and LilySelective should yield conclusions similar to those in the paper. Namely, LilySelective should produce as many or more false alarms than Lily (i.e., more commits/releases _and_ more individual runs should be incorrectly blocked), due to the side-effect of having a less representative standard behavior corpus, as described in the paper. **As Lily is based on fuzzing, which is stochastic, you might not be able to reproduce exactly the same numbers in this reproduction as those reported in the paper. Additionally, due to the short duration of the fuzzing campaigns, the specifications of your machine might affect the results. What should be observed, though, is that, with a sufficiently large number and duration of campaigns, the obtained results converge towards the conclusions made in the paper about the performance of Lily.** ### Unguided partial reproduction (optional) **Unguided partial reproduction enables launching as many detection campaigns as you want, of any custom length, over any of the experiments from our paper**, by running the following command in the container with (1) the number of trials/runs, (2) the duration of each trial/run, and (3) the experiment to run: ```console RUNS= SECONDS_PER_RUN= /root/artifact/run-target.sh ``` Files for all experiments evaluated in the paper can be found under `/root/experiments/` in the container. Specifically, the following categories are available (in the corresponding subdirectories): - `backdoored-commits/`: a recent commit, compared to the same commit with an (authentic or synthetic) backdoor injected on top; - `backdoored-releases/`: a pair of releases, based on the three latest Ubuntu and Debian releases, with the most recent release having an (authentic or synthetic) backdoor injected on top; - `safe-commits/`: a pair of _safe_ (i.e., introducing no backdoors) commits; - `safe-code-commits/`: a pair of _safe_ (i.e., introducing no backdoors) commits, where the change between the two commits affects at least a single source code file; - `safe-releases/`: a pair of _safe_ (i.e., introducing no backdoors) releases, based on the three latest Ubuntu and Debian releases. When finished, the last printed message should read `Done! PDF available at: /root/evaluation/.pdf`. **From your host machine**, you should run the following command to retrieve the PDF from the container: ```console docker cp lily-ase26-artifact:/root/evaluation/.pdf . ``` You can then compare the results in the generated PDF to the corresponding tables in the paper. **As Lily is based on fuzzing, which is stochastic, you will never be able to reproduce exactly the same numbers as reported in the paper. Additionally, due to the short duration of the fuzzing campaigns, the specifications of your machine might affect the results. What should be observed, though, is that, with a sufficiently large number and duration of campaigns, the obtained results converge towards the conclusions made in the paper about the performance of Lily.** ### Complete reproduction (optional) **Complete reproduction means running 20 10-minute trials, for each of the 545 version pairs across the 13 projects evaluated in the paper**, thus obtaining the complete tables 3–6 from the paper. **It is not recommended to run _Complete reproduction_ on a personal desktop or laptop machine** (where it will take at least five months to complete) but only on a sufficiently powerful server, if you have one. In order to perform the complete reproduction, **in the container**, run: ```console /root/artifact/run-full-evaluation.sh ``` When finished, the last printed message should read `Done! PDF available at: /root/evaluation/full-evaluation.pdf`. If any warning messages appear, it is possible that some of the experiments failed, most likely due to insufficient disk size or platform incompatibilities (see [REQUIREMENTS.md](./REQUIREMENTS.md)). In that case, the resulting PDF will contain _partial_ results (i.e., including only the experiments which succeeded). **From your host machine**, you should run the following command to retrieve the PDF from the container: ```console docker cp lily-ase26-artifact:/root/evaluation/full-evaluation.pdf . ``` **As Lily is based on fuzzing, which is stochastic, you will never be able to reproduce exactly the same numbers as reported in the paper. Additionally, due to the short duration of the fuzzing campaigns, the specifications of your machine might affect the results. What should be observed, though, is that, with a sufficiently large number and duration of campaigns, the obtained results converge towards the conclusions made in the paper about the performance of Lily.** ### Discovering how Lily can be reused or repurposed Lily is integrated into the [ROSA backdoor detection toolchain](https://github.com/binsec/rosa/tree/lily), and welcomes new pull requests on GitHub! Furthermore, ROSA's documentation has been updated to include [detailed instructions](https://github.com/binsec/rosa/blob/lily/doc/src/toolchain/rosa_filter_diff.md) on how to set up Lily in a production environment. The experiment infrastructure packaged in this artifact is designed to be modular and easily extendable. Namely, you can add the following things: - **New version pairs for existing projects**: by following the format laid out in the experiment files under `/root/experiments`, you can add "recipes" for new experiments to evaluate with Lily. - **New software projects**: by following the format laid out in `/root/artifact/targets`, you can add new software projects to evaluate with Lily. Specifically, you need to add the new project as a submodule under `/root/artifact/targets//original`, provide any patches and a `build.sh`, and fill in the project's information in `/root/artifact/tools/targets.toml`. To go even further, several analysis tools are present under `/root/artifact/tools`. See [_Artifact structure_](#artifact-structure) for a description of each tool (as well as a description of the general structure of the artifact). Lastly, **significant engineering effort was put into providing robust support for AFL++'s source-level instrumentation**, which in turn resulted in a major refactoring of the ROSA backdoor detection toolchain, including the addition of new unit tests to further support refactoring efforts in the future. #### Long-term archive For archiving purposes, the code and documentation of Lily's implementation in ROSA has been bundled into this artifact, but a separate archive is also available on [Software Heritage](https://archive.softwareheritage.org/browse/origin/directory/?branch=refs/heads/lily&origin_url=https://github.com/binsec/rosa). ## Documentation on the artifact code and layout The documentation of Lily can be found: - On GitHub: - In the archived repo, on Software Heritage: - In your browser, while the Docker container is running (see [_Installing the Docker image_](#installing-the-docker-image)): The various Bash and Python scripts forming the experiment infrastructure of the artifact all contain a documentation header (typically a comment header in the first few lines, explaining the purpose of the script). The structure of the artifact is the following: - `AUTHORS`: a list of the authors of the artifact. - `LICENSE`, `LICENSE.gpl`: license files. - `README.md`: this file. - `REQUIREMENTS.md`: a description of the (hardware and software) requirements needed to use this artifact. - `STATUS.md`: a description and justification of the artifact badges we (the authors) are applying for. - `VERSION`: the current version of the artifact. - `compile-pdf.py`: a script to compile experiment results to a PDF closely resembling part of the paper. - `run-experiment.sh`: a script to run a single experiment with Lily (covering the evaluation shown in tables 3–6 in the paper). - `run-full-evaluation.sh`: a script running the full evaluation shown in the paper. - `run-reduced-evaluation.sh`: a script running part of the evaluation shown in the paper (see [_Guided partial reproduction_](#guided-partial-reproduction)). - `start.sh`: a startup script, launching the documentation server. - `seeds/`: seed corpora used during fuzzing (adapted from AFL++, see [LICENSE](./LICENSE)). - `dictionaries/`: dictionaries used during fuzzing (adapted from AFL++, see [LICENSE](./LICENSE)). - `targets/`: source code of the software projects analyzed in the evaluation of the paper and related utilities. For each _target_: - `backdoor-triggers/`: upwards of 100 backdoor-triggering inputs, used in the poisoning attack evaluated in Table 6 in the paper. - `build.sh`: script which builds the software project. - `config-binary.toml.template`, `config.toml.template`: templates for the configuration of (binary-only) ROSA and Lily respectively. They are instantiated before an experiment begins. - `original/`: source code of the software project (usually a submodule). - `patches/`: patches for the software project, either to apply an (authentic or synthetic) backdoor (with or without a _ground truth_ marker), or to patch in a fuzzing harness. - `safe-code-commits.toml`: a description of the _safe code commit_ version pairs considered for this software project, with instructions on how to build them (e.g., what patch to apply). - `safe-commits.toml`: similar to `safe-code-commits.toml`, but for the _safe commit_ version pairs. - `safe-releases.toml`: similar to `safe-commits.toml`, but for the three latest Debian and Ubuntu releases of the software project. - `tools/`: experiment infrastructure and tooling. - `analysis/`: tools for analyzing software projects or experiment results. - `extract-results.py`: a script which extracts and summarizes results from a zipped experiment archive. - `get-lily-report-data.sh`: a script which extracts and summarizes the report data (number of suspicious lines, number of suspicious system call callsites and so on) given the suspicious code change reports generated by Lily. - `get-representative-commits.py`: a script which extracts a semi-randomly sampled set of _representative commits_ from the Git repository of a software project. This was used to select the commits in the experiments found under `/root/experiments/` in the Docker container of the artifact. - `common.sh`: functions and variables commonly used by various scripts in the artifact. - `evaluation`: tools for evaluating experiment artifacts. - `build-target.sh`: a script which builds a software project, boiling down to a single binary program to fuzz with AFL++. It uses the build recipe and instantiates the ROSA/Lily configuration templates provided in `targets/`. - `collect-coverage.sh`: a script which collects LLVM coverage for a set of test inputs. - `coverage.py`: a small library containing utility function to deal with coverage (mostly translating back and forth from LLVM binary coverage formats). - `disable-exec.so.c`, `disable-exec.so`: the C source code and resulting shared library, used to disable `exec*()` calls in running targets, as those are known to interfere with LLVM coverage measurements. This is only used when measuring coverage. - `evaluate-benchmark.py`: a script which performs full, end-to-end evaluation of a set of experiment artifacts. This is the main script used when Lily has finished running, to build a detailed log of all results. - `generate-lily-reports.py`: a script which generates Lily's suspicious code change reports. - `run-benchmark.py`: a script which performs full, end-to-end execution of an experiment (without evaluating the results). Typically, its output is fed to `evaluate-benchmark.py`. - `run-naive-diff.sh`: a script which runs a naive differential oracle (used in the ablation study of Lily's components, see Table 4). - `experiment-helpers`: tools for running experiments, and experiment definitions. - `backdoored-commits/`, `backdoored-releases/`, `safe-commits`, `safe-code-commits/`, `safe-releases/`: contain per-target experiment files for each category evaluated in the paper (see [_Unguided partial reproduction_](#unguided-partial-reproduction-optional)) - `run.sh`: a script which handles setup and teardown of an experiment, including packing the results in a zip file. - `startup`: tools for generation of various parts of the infrastructure during startup. - `generate-configs.py`: a script which generates ROSA/Lily configuration templates, ensuring that the same fuzzer configurations are used across all experiments. - `generate-experiment-files.py`: a script which generates the experiment files/recipes used to run all of the experiments in the evaluation of the paper. The generated files can be found under `/root/experiments/` in the Docker container of the artifact. - `startup.sh`: a script which is responsible for running all startup-time tasks. - `targets.toml`: high-level configuration for each of the supported software projects, used by various tools in the artifact.