Research Artifact for Artemis

Cong Li, Yanyan Jiang, Chang Xu, Zhendong Su

This research artifact is made for our SOSP 2023 paper: Validating JIT Compilers via Compilation Space Exploration.

Overview

This artifact contains a minimal example to test HotSpot's JIT compilers (revision: bd9c2fb38532) using Artemis. The authors prepared the artifact to demonstrate the main contributions of Artemis with the following two purposes:

  1. Artifacts Available: The authors have made their tool Artemis along with all the reported bugs, publicly available to facilitate future research, as claimed in their paper.
  2. Artifacts Evaluated - Functional: The provided tool Artemis, is well documented, exercisable, and functional. Specifically, (1) Artemis can mutate existing Java programs and the generated mutants can indeed cause HotSpot to crash and (2) Artemis also provides a fuzzing framework to test HotSpot's JIT compilers continuously.

To make it as straightforward and easy as possible, the authors (1) packaged the Artemis tool into a Docker image (with all dependencies installed) and (2) publicized all reported JIT compiler bugs directly in this README.

Exercising this artifact involves the following steps:

Step 1: Pull and Run a Docker Container

Step 1.1. Run the following command to pull the image:

docker pull leetsong/artemis-sosp23ae:latest

If succeeded, the following (likely) message will be printed:

latest: Pulling from leetsong/artemis-sosp23ae
9d19ee268e0d: Pull complete
5e8b0651f93e: Pull complete
d1b09886c784: Pull complete
Digest: sha256:0bc0135f53a96f8805372d079b094c42524cfb52dd2e99b9a9699ceb6ed389e3
Status: Downloaded newer image for leetsong/artemis-sosp23ae:latest
docker.io/leetsong/artemis-sosp23ae:latest

One can also exercise the following command to double-check that the image was pulled successfully (Note, the CREATED field might be different; this is expected):

$ docker image ls
REPOSITORY                   TAG       IMAGE ID       CREATED          SIZE
leetsong/artemis-sosp23ae    latest    e54c391b6a28   47 minutes ago   2.03GB

Otherwise, it fails. Please

Step 1.2. Run the following command to start a container:

docker run --rm -it leetsong/artemis-sosp23ae:latest

After the container started, a command line prompt is printed. In this artifact, all Artemis-related files (including Artemis itself, the HotSpot build, and example configuration files) are placed under /artemis, where:

Step 2: Mutate an Example Seed Prgram by Artemis

The authors have put an example seed program Test.java (generated by JavaFuzzer) under /artemis/example/crash/. One can check its content via the following command:

cat /artemis/example/crash/Test.java

Run the following command to mutate the seed program by Artemis and save it as /artemis/example/crash/mutant/Test.java:

java -jar /artemis/artemis/artemis.jar -v -s 3980266633 -p artemis -m 1000 -M 20000 -b /artemis/code_bricks -o /artemis/example/crash/mutant -i /artemis/example/crash/Test.java

Note:

Once the command completes, one can check the mutant via the following command:

cat /artemis/example/crash/mutant/Test.java

Since all Artemis-generated variables are prefixed by ax$, one can also inspect the Artemis-generated code via the following command:

cat /artemis/example/crash/mutant/Test.java | grep ax

Or search for ax in the mutant.

Step 3: Crack HotSpot's JIT Compiler C2 by Artemis-Generated Mutant

Compile the mutant via the following command (Note, some compilation warnings will be printed; it is safe to ignore them directly):

/artemis/example/hotspots/jdk11u.bd9c2fb38532/bin/javac -cp /artemis/example/crash/mutant/ /artemis/example/crash/mutant/Test.java

Enter the following command to run the mutant via the HotSpot build:

/artemis/example/hotspots/jdk11u.bd9c2fb38532/bin/java -cp /artemis/example/crash/mutant Test

If everything works as planned, the HotSpot build will crash with the assertion "assert(false) failed: infinite loop in PhaseIterGVN::optimize" being printed, like:

# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/phaseX.cpp:1193
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/home/simon/JVMs/jdk11u-dev/src/hotspot/share/opto/phaseX.cpp:1193), pid=91, tid=103
#  assert(false) failed: infinite loop in PhaseIterGVN::optimize
#
# JRE version: OpenJDK Runtime Environment (11.0.19) (fastdebug build 11.0.19-internal+0-adhoc.simon.jdk11u-dev)
# Java VM: OpenJDK 64-Bit Server VM (fastdebug 11.0.19-internal+0-adhoc.simon.jdk11u-dev, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# //hs_err_pid91.log
#
# Compiler replay data is saved as:
# //replay_pid91.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp

This means HotSpot's JIT Compiler C2 (opto) crashed when compiling a loop introduced by Artemis.

To confirm that it is indeed a JIT compiler bug, one may run the mutant by the -Xint option; this disables JIT compilation and HotSpot will execute the mutant with its interpreter only:

/artemis/example/hotspots/jdk11u.bd9c2fb38532/bin/java -cp /artemis/example/crash/mutant -Xint Test

One may notice that HotSpot runs correctly. However, since Artemis generated many large loops, executing the mutant without JIT compilations is pretty slow. One can kill the process by keyboard shortcut Ctrl+C directly.

Step 4: Test HotSpot's JIT Compilers by Artemis

Run the following commands to start testing:

cd /artemis/artemis
python3 artemi.py ../example/artemi.yaml

After running the command, the test procedure is started and the following logs will be printed unless the process is killed:

< Submit: new item (index: 1)
+ Execute: starts to execute item (index: 1)
+ Run test: received reference /artemis/example/jaf/0/Test.java, num-mutation: 4, run-timeout: 60s
< Submit: new item (index: 2)
+ Execute: starts to execute item (index: 2)
+ Run test: received reference /artemis/example/jaf/1/Test.java, num-mutation: 4, run-timeout: 60s
< Submit: new item (index: 3)
< Submit: new item (index: 4)
< Submit: new item (index: 5)
- JVM exec: running reference under JVM, /artemis/example/jaf/0/Test.java
- JVM exec: running reference under JVM, /artemis/example/jaf/1/Test.java
- Mutate 1: 0/4, /artemis/example/jaf/1/mutants/0
- JVM exec: running mutant under JVM, /artemis/example/jaf/1/mutants/0/Test.java
- Mutate 1: 1/4, /artemis/example/jaf/1/mutants/1
- JVM exec: running mutant under JVM, /artemis/example/jaf/1/mutants/1/Test.java
- Mutate 1: 2/4, /artemis/example/jaf/1/mutants/2
- JVM exec: running mutant under JVM, /artemis/example/jaf/1/mutants/2/Test.java
- Mutate 1: 3/4, /artemis/example/jaf/1/mutants/3
- JVM exec: running mutant under JVM, /artemis/example/jaf/1/mutants/3/Test.java
> Finished: item (index: 2) is executed
+ Execute: starts to execute item (index: 3)
+ Run test: received reference /artemis/example/jaf/2/Test.java, num-mutation: 4, run-timeout: 60s
< Submit: new item (index: 6)
> Great: 0, discard /artemis/example/jaf/1/mutants/0
> Great: 1, discard /artemis/example/jaf/1/mutants/1
> Great: 2, discard /artemis/example/jaf/1/mutants/2
> Great: 3, discard /artemis/example/jaf/1/mutants/3
- JVM exec: running reference under JVM, /artemis/example/jaf/2/Test.java

Please keep the process running if one enjoys testing HotSpot's JIT compilers and check if anything interesting could be found in this time. Otherwise, one can stop the process at any time by keyboard shortcut Ctrl+C.

Note, if one enjoys the testing HotSpot's JIT compilers, one can change the num_proc and num_mutation fields in /artemis/example/artemi.yaml to enable much concurrency and more mutations per seed, respectively, according to the evaluating machines.

It should also be noted that, the test procedure is likely to generate some random files; this is expected and they are safe to be deleted after testing.

Step 5: Check the Testing Results (If Any)

In this artifact, all bugs detected by Artemis will be listed in /artemis/example/out/differences/diffs.csv, where each subdirectory /artemis/example/out/differences/<diff_id> saves the information of one detected bug, including the used seed program (<diff_id>/Test.java), the Artemis-generated mutant (<diff_id>/mutant/Test.java), the output of the seed (<diff_id>/reference.txt), the output of the mutant (<diff_id>/mutant.txt), and Artemis' mutation log (<diff_id>/mutant/mutations.txt).

It should be noted that, since Artemis is a bug-finding tool, /artemis/example/out/differences/diffs.csv will list noting if one has tested the HotSpot build only for a short time.

Step 6: Stop and Cleanup Artemis

To stop Artemis' test procedure, type the keyboard shortcut Ctrl+C.

To exit the container, directly enter exit from the command line.

To cleanup the image from the evaluating machine, exercise the following command after exiting the container:

docker rmi leetsong/artemis-sosp23ae:latest

Step 7: Check All Reported JIT Compiler Bugs

To facilitate future research, the authors list all the reported JIT compiler bugs in this section. This includes their URLs, severity (if there are), type of the bug, and the affected JVM component(s).

It should be noted that, to avoid flooding their issue trackers, the authors discussed with the corresponding JVM developers and reported some difficult-to-reproduce, flaky tests into a single issue. This kept the number of bug reports under a small limit.

In addition, some bugs listed below were made internal by the respective JVM developers and are no longer publicly accessible for security reasons.

HotSpot

OpenJ9

ART

Step 8: Beyond the Artifact

Beyond this artifact, the authors also publicized the Artemis tool in the following Github link:

https://github.com/test-jitcomp/Artemis