Research Artifact for "Generating Semantic-Aware Mutators to Boost Compiler Fuzzing"

# Research Artifact for MetaMut

Xianfei Ou, Cong Li, Yanyan Jiang, Chang Xu

This research artifact is made for ASPLOS 2024 paper: Generating Semantic-Aware Mutators to Boost Compiler Fuzzing

The authors prepared this artifact to demonstrate the main contributions of MetaMut.

## Overview

A Docker image is provided to reproduce all of the evaluation results. Specifically, it includes the following:

1. **MetaMut Source Code**:
   - Located at `/root/MetaMut`.

2. **Previously Generated Mutators**:
   - Located at `/root/fuzzers/Muss`.
   - The supervised version can be found in the `YSMut` subdirectory.
   - The unsupervised version is in the `LLMut` subdirectory.

3. **Fuzzing Tools Evaluated**:
   - Source code and compiled objects are available for:
     - **AFL++**: Located at `/root/fuzzers/AFLplusplus`.
     - **GrayC**: Located at `/root/fuzzers/GrayC`.
     - **Csmith**: Located at `/root/fuzzers/csmith`.
     - **Yarpgen**: Located at `/root/fuzzers/yarpgen`.

4. **Seed Programs for Evaluating Fuzzing Tools**:
   - Found in `/root/seeds`.
   - This directory contains 1839 C programs, named from `0001.c` to `1839.c`.

5. **Compilers**:
   - Used for evaluating branch discovery and bug hunting capabilities.
   - Located in `/root/compilers`.
   - Includes:
     - **GCC-14** (revision 08b887).
     - **Clang-18** (revision d437e6).
   - Compiled objects can be found in `/root/objects`.

6. **Evaluation Scripts**:
   - Scripts for running evaluations of each tool's branch discovery and bug hunting capabilities.
   - Includes scripts for summarizing the results.
   - Located in `/root/driver`.

## Step 1: Import and Start the Docker Image

This step requires 40GB of disk space and may take several hours to complete. After this, you will enter a shell with root user privileges.
1. Extract the Docker image:
    ```sh
    tar -xf metamut-asplos24-ae.tar.gz
    ```
2. Import the Docker image:
    ```sh
    cat metamut-asplos24-ae.tar | sudo docker import - metamut
    ```
3. Run the Docker container:
    ```sh
    sudo docker run -it metamut /bin/zsh
    ```

## Step 2: Generate Semantic-Aware Mutators via MetaMut

### Step 2.1: Configure Your OpenAI API Key

1. Change the working directory to `/root/MetaMut`:
    ```sh
    cd /root/MetaMut
    ```
2. Edit the `configs.py` file located in `MetaMut/scripts` and fill in the `openai.api_key` field.
    - For non-official sources of the OpenAI API, you may also need to modify the `openai.api_base` field accordingly.
3. Verify that your API key is working:
    ```sh
    python3 scripts/main.py --print-model-list
    ```
    On success, this will print out a list of available models, including GPT-4.

### Step 2.2: Run MetaMut

To generate 100 mutators:

```sh
cd /root/MetaMut
python3 scripts/main.py --num-mutators=100
```

**Quick Verification**:
You may use a smaller value of `--num-mutators`, such as 3.

This step may take up to 24 hours to complete. The generated mutators can be found in the `MetaMut/lib/mutators` directory, and the generation logs in the `MetaMut/logs` directory.

Note: The `--num-mutators` option specifies the maximum number of generation attempts. Because mutator generation can fail for various reasons, the final number of generated mutators may be less than the specified amount.

### Step 2.3: Statistics for Fixed Mutator Bugs (Table 1)

Assuming the `MetaMut/logs` directory exists (generated in Step 2.2):

```sh
cd /root/MetaMut
python3 scripts/analyzeGoalValidation.py --idir logs
```

If successful, a message similar to the following will be printed, corresponding to Table 1. The numbers are examples; your output may vary:

```
m not compile: 55
m hangs: 0
m crashes: 4
m outputs nothing: 11
m does not rewrite: 1
m creates compile-error mutant: 36
```

## Step 2.4: Analyze Generation Cost (Table 2 and Table 3)

Assume the `MetaMut/logs` directory is present (generated in Step 2.2).

To analyze the generation cost of all generated mutators:

```sh
cd /root/MetaMut
python3 scripts/analyzeGenerationCost.py --idir logs
```

If successful, a message similar to the following will be printed, corresponding to Table 2 and Table 3:

```
tokens:
Invention: min: 359 max: 2240 median: 1130.5 mean: 1158.75
Implementation: min: 372 max: 3870 median: 2488.0 mean: 2501.815789473684
Bug-Fixing: min: 335 max: 30923 median: 2077.5 mean: 4935.328947368421
Total: min: 3214 max: 35312 median: 6054.0 mean: 8595.894736842105

QA rounds:
BugFixing: min: 1 max: 23 median: 2.0 mean: 4.0394736842105265
Total: min: 3 max: 25 median: 4.0 mean: 6.0394736842105265

time:
min:11 max:21 median:15 mean:15
min:14 max:101 median:49 mean:49
min:29 max:1876 median:130 mean:281
min:83 max:1949 median:189 mean:346

Wait for Response: min:11 max:123 median:46 mean:43
Prepare for Request: min:0 max:69 median:9 mean:17
```

## Step 3: Fuzz Compilers with M_s and M_u

To fuzz compilers:

```sh
cd /root
mkdir -p workspace.fuzz
cd workspace.fuzz
python3 /root/driver/runUniqueCrash.py --seeds-dir /root/seeds --repeat-times 60 --wdir $(pwd) --seed 12345678 --fuzzers mu.s mu.u --duration 86400
```

Options explanation:

- `--seeds-dir`: Specifies the directory of seed programs.
- `--wdir`: Specifies the working directory; mutants, logs, and temporary files are saved to this directory.
- `--seed`: Specifies the random seed; this usually does not matter, `12345678` is acceptable.
- `--fuzzers mu.s mu.u`: Specifies the fuzzers to run. `mu.s` means the supervised version of generated mutators, and `mu.u` means the unsupervised version. All possible choices for this option are: all, mu.s, mu.u, afl++, grayc, csmith, and yarpgen.
- `--duration 86400`: Specifies the duration (in seconds) to run each fuzzer, defaulting to 24 hours.

**Quick Verification**:
You may use `--duration 1200` to run for 20 minutes instead of the default 24 hours, as well as `--repeat-times 1` instead of the default 60 repeat times.

This will generate subdirectories `mu.s,gcc,0`, `mu.s,clang,0`, `mu.u,gcc,0`, `mu.u,clang,0`, and etc. They are named in the format `{fuzzer},{compiler},{repeat times}`.

### Step 3.1: Analyze Coverage Trend

Every ten minutes or when new branches are discovered, coverage data will be updated in `{subdir}/coverage-trend.txt`, where `{subdir}` is something like `mu.s,gcc,0` created in Step 3.

Each line of `coverage-trend.txt` is in the format `{time in seconds},{currently discovered branches},{total branches},{branch coverage ratio}`.

An example `coverage-trend.txt`:

```
0.3652334213256836,72568,4569446,0.015881137450798194
0.5427205562591553,77566,4569446,0.01697492431248777
...
```

### Step 3.2: Analyze Compiler Crashes
Upon encountering a crash (whether new or duplicate), it will be saved to `{subdir}/crashes.json`. To view them, you can run:
```sh
python3 /root/driver/showUniqueCrashes.py {subdir} --summary
```
This command will output something like:
```
[
  {
    "compiler": "/root/objects/llvm-project-trunk.obj/bin/clang",
    "options": "-O2",
    "traceback": [
      [
        null,
        "llvm::X86TargetLowering::LowerSET_ROUNDING(llvm::SDValue, llvm::SelectionDAG&) const",
        null
      ],
      [
        null,
        "(anonymous namespace)::SelectionDAGLegalize::LegalizeOp(llvm::SDNode*)",
        null
      ]
    ],
    "mutators": [
      "mu.s"
    ],
    "srcfiles": [
      {
        "mutator": "mu.s",
        "date": 58.185559034347534,
        "file": "/root/workspace/mu.s,clang,58/muyucoztSfmQ.c"
      },
  }
]

{
  "mu.s": 40
}
```

Fields in the above example output are:
1. `compiler`: Shows the crashed compiler's binary path.
2. `traceback`: Lists the top 2 stack frames of the compiler crash, excluding irrelevant helper functions such as `llvm::report_error`.
3. `srcfiles`: Lists all encountered mutants that can trigger this crash.

To replay an existing crash from the output, you can follow these steps:

1. Find the crashed compiler, e.g., `/root/objects/llvm-project-trunk.obj/bin/clang`.
2. Find a file from the output, e.g., `/root/workspace/mu.s,clang,58/muyucoztSfmQ.c`.
3. Start GDB to debug it:
    ```sh
    gdb -q --args /root/objects/llvm-project-trunk.obj/bin/clang -O2 /root/workspace/mu.s,clang,58/muyucoztSfmQ.c
    ```
4. Execute these GDB commands to follow the `cc1` process:
    ```gdb
    b exit
    b abort
    set follow-fork-mode child
    ```
5. Run until the crash and backtrace the stacks:
    ```gdb
    r
    bt
    ```
    Upon normal exit, you will see `__GI_exit` in the backtrace. If it indicates a crash, the backtrace will show the details of the crash.

**Special Note**:
DO NOT use the return code to determine if a compiler crash has occurred. Newer versions of GCC and Clang have implemented crash handling strategies that suppress the crash's return code, returning 1 (the same return code as a compile error). Instead, use GDB to check for crashes as instructed above.

## Step 4: Compare with Existing Fuzzers

### Step 4.1: Statistics for Each Fuzzer's Throughput and Compilable Mutant Ratio

To statistic throughput:
```
cd /root
mkdir -p workspace.throughput
cd workspace.throughput
python3 /root/driver/runThroughput.py --seeds-dir /root/seeds --wdir $(pwd) --seed 12345678 --repeat-times 1 --duration 86400
```

Options explanation:
* `--seeds-dir`, this specify the directory of seed programs
* `--wdir`, this specify the working directory, mutants, logs and temporary files are saved to this directory
* `--seed`, specify the random seed, this usually does not matter, 12345678 is ok
* `--repeat-times`, specify the repeat times, to avoid randomness of evaluation results, you may specify a larger value
- `--duration 86400`: Specifies the duration (in seconds) to run each fuzzer, defaulting to 24 hours.

**Quick Verification**:
You may use `--duration 1200` to run for 20 minutes instead of the default 24 hours. However, please note that this may not provide accurate results.
The coverage increase rate and initial rate for each fuzzer vary, so the figure for the first 20 minutes may look quite different from that of the entire 24 hours.

After this, you will see some new directories created in the working directory. Each directory corresponds to a running fuzzing instance. The number of directories will increase as a fuzzing instance finishes and a new one is scheduled:

```sh
$ cd /root/workspace.throughput; ls
afl++,gcc,0/
grayc,gcc,0/
yarpgen,gcc,0/
csmith,gcc,0/
mu.s,gcc,0/
mu.u,gcc,0/
```

Given the specific example `afl++,gcc,0/` shown above, this directory represents the 0th fuzzing instance of AFL++.

This step will consume up to `24 * 6 * n` CPU hours (where `n` is the number of repeat times). To enhance evaluation stability, each fuzzing instance will use two physical cores exclusively (a physical core usually has two logical cores).

After this script finishes, you may find the following output, which corresponds to Table 5. If your screen is flushed, you can retrieve the output via:

```sh
python3 /root/driver/runThroughput.py --wdir /root/workspace.throughput --summary
```

The output will look like this:

```
csmith: Time=5.14 #CompilableMutants=1.0 #Total=1.0 CompilableRatio=1.0
yarpgen: Time=0.19 #CompilableMutants=1.0 #Total=1.0 CompilableRatio=1.0
afl++: Time=0.00 #CompilableMutants=1.0 #Total=1.0 CompilableRatio=1.0
mu.u: Time=0.09 #CompilableMutants=0.0 #Total=1.0 CompilableRatio=0.0
mu.s: Time=0.13 #CompilableMutants=1.0 #Total=1.0 CompilableRatio=1.0
grayc: Time=0.09 #CompilableMutants=0.0 #Total=1.0 CompilableRatio=0.0
```

### Step 4.2: Evaluate Each Fuzzer's Branch Discovery and Bug Hunting Ability

To start:

```sh
cd /root
mkdir -p workspace.uniqcrash+coverage
cd workspace.uniqcrash+coverage
python3 /root/driver/runUniqueCrash.py --seeds-dir /root/seeds --wdir $(pwd) --seed 12345678 --repeat-times 60 --duration 86400
```

**Options:**

- `--seeds-dir`: Specifies the directory of seed programs.
- `--wdir`: Specifies the working directory. Mutants, logs, and temporary files are saved to this directory.
- `--seed`: Specifies the random seed. This usually does not matter; `12345678` is a suitable default.
- `--repeat-times`: Specifies the number of repeat times. To minimize the randomness of evaluation results, consider specifying a larger value.
- `--duration 86400`: Specifies the duration (in seconds) to run each fuzzer, defaulting to 24 hours.

**Quick Verification**:
You may use `--duration 1200` to run for 20 minutes instead of the default 24 hours, as well as `--repeat-times 1` instead of the default 60 repeat times.

After running the command, you will see some newly created directories like these (the number of directories will increase as a fuzzing instance finishes and a new one is scheduled):

```sh
$ cd /root/workspace.uniqcrash+coverage; ls
afl++,gcc,0/     afl++,gcc,1/     afl++,gcc,2/     ...
afl++,clang,0/   afl++,clang,1/   afl++,clang,2/   ...
grayc,gcc,0/     grayc,gcc,1/     grayc,gcc,2/     ...
grayc,clang,0/   grayc,clang,1/   grayc,clang,2/   ...
yarpgen,gcc,0/   yarpgen,gcc,1/   yarpgen,gcc,2/   ...
yarpgen,clang,0/ yarpgen,clang,1/ yarpgen,clang,2/ ...
csmith,gcc,0/    csmith,gcc,1/    csmith,gcc,2/    ...
csmith,clang,0/  csmith,clang,1/  csmith,clang,2/  ...
mu.s,gcc,0/      mu.s,gcc,1/      mu.s,gcc,2/      ...
mu.s,clang,0/    mu.s,clang,1/    mu.s,clang,2/    ...
mu.u,gcc,0/      mu.u,gcc,1/      mu.u,gcc,2/      ...
mu.u,clang,0/    mu.u,clang,1/    mu.u,clang,2/    ...
```

Given the specific example `afl++,gcc,0/` shown above, this directory represents the 0th fuzzing instance of AFL++ running with GCC.

To balance evaluation stability and evaluation time, each fuzzing instance will use one physical core (or two logical cores) exclusively.

Similar to steps 3.1 and 3.2, you can find `coverage-trend.txt` in each subdirectory, which stores the coverage data for that fuzzing instance. Additionally, `crashes.json` will store all found crashes.

#### Step 4.2.1: Plot Coverage Trend (Figure 7)

To plot the coverage trend:
```sh
cd /root
mkdir -p figures
python3 /root/driver/plotCoverageTrend.py --idir /root/workspace.uniqcrash+coverage --odir /root/figures
```
After running this, you will see two PDF files, `gcc-trend.pdf` and `clang-trend.pdf`, inside the `/root/figures` directory. These files correspond to the coverage trends for GCC and Clang, respectively, and each contains data for six fuzzers.

#### Step 4.2.3: Plot Unique Crash Trend (Figure 9)
To plot coverage trend:
```
mkdir -p /root/{figures,datas}
cd /root/workspace.uniqcrash+coverage
python3 /root/driver/showUniqueCrashes.py * > /root/datas/crashes.json
python3 /root/driver/plotUniqueCrashes.py /root/datas/crashes.json --odir /root/figures
```
After this, you will see two pdf file `gcc-crashes.pdf` and `clang-crashes.pdf` inside the directory `figures`, they are corresponding to gcc's unique crashes trend and clang's unique crashes trend, each consists of six fuzzers.

#### Step 4.2.4: Venn Graph for Unique Crashes (Figure 8)

To obtain the Venn graph data:

```sh
mkdir -p /root/{figures,datas}
cd /root/workspace.uniqcrash+coverage
python3 /root/driver/showUniqueCrashes.py * > /root/datas/crashes.json
python3 /root/driver/tableVenn.py /root/datas/crashes.json
```

This will output something like:

```
num of unique crashes:
grayc: 1
mu.s: 16
mu.u: 20
total: 29

venn graph (A:afl++ B:grayc C:mu.u D:mu.s E:csmith F:yarpgen):
BDC: 1
D: 9
DC: 6
C: 13
```

In this output:
- A is `AFL++`
- B is `GrayC`
- C is `\muCFuzz.u`
- D is `\muCFuzz.s`
- E is `Csmith`
- F is `Yarpgen`

For example, `BDC: 1` means in the Venn graph, the intersection of `B` (grayc), `D` (mu.s), and `C` (mu.u) contains 1 unique crash.

## Step 5: Check Reported Bugs

### Clang

- [69070](https://github.com/llvm/llvm-project/issues/69070) - clang: Assertion `StackTop == (unsigned)llvm::popcount(Mask) && "Live count mismatch"' failed.
- [69202](https://github.com/llvm/llvm-project/issues/69202) - clang: Assertion `Value != 0 && "Invalid CPUStr passed to CpuIs"' failed.
- [69212](https://github.com/llvm/llvm-project/issues/69212) - clang: Assertion `ConstraintID != InlineAsm::ConstraintCode::Unknown && "Failed to convert memory constraint code to constraint id."' failed.
- [69217](https://github.com/llvm/llvm-project/issues/69217) - clang: fatal error: error in backend: register rbp is allocatable: function has no frame pointer
- [69347](https://github.com/llvm/llvm-project/issues/69347) - clang: 18: segment fault in `llvm::SelectionDAGISel::CannotYetSelect(llvm::SDNode\*)`
- [69201](https://github.com/llvm/llvm-project/issues/69201) - clang: fatal error: error in backend: Cannot emit physreg copy instruction
- [73109](https://github.com/llvm/llvm-project/issues/73109) - clang: 18: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
- [62984](https://github.com/llvm/llvm-project/issues/62984) - clang-17 crashes at llvm::SelectionDAGISel::Select\_READ\_REGISTER
- [62987](https://github.com/llvm/llvm-project/issues/62987) - clangtk fatal error: cannot pop empty stack
- [63027](https://github.com/llvm/llvm-project/issues/63027) - clangtk crashes at ::FPS::processBasicBlock
- [63028](https://github.com/llvm/llvm-project/issues/63028) - clangtk crashes at llvm::X86InstrInfo::copyPhysReg
- [63032](https://github.com/llvm/llvm-project/issues/63032) - `clangtk` crashes at llvm::SelectionDAGISel::SelectInlineAsmMemoryOperands
- [63569](https://github.com/llvm/llvm-project/issues/63569) - clang crashes when all `xmm` registers are clobbered
- [62997](https://github.com/llvm/llvm-project/issues/62997) - clang-17 crashes at llvm::SelectionDAG::getNode
- [62989](https://github.com/llvm/llvm-project/issues/62989) - clang-17 crashes at CodeGen::CodeGenFunction::EmitAtomicExpr
- [62999](https://github.com/llvm/llvm-project/issues/62999) - clang-17 crashes at clang::CodeGen::CodeGenFunction::EmitFunctionProlog
- [63001](https://github.com/llvm/llvm-project/issues/63001) - Assertion `S1->getType() == S2->getType() && "Cannot create binary operator with two operands of differing type!"' failed.
- [63002](https://github.com/llvm/llvm-project/issues/63002) - clangtk: Assertion `!isNull() && "Cannot retrieve a NULL type pointer"' failed.
- [63516](https://github.com/llvm/llvm-project/issues/63516) - `clangtk`: Assertion `isValidGCCRegisterName(Name) && "Invalid register passed in"' failed
- [63517](https://github.com/llvm/llvm-project/issues/63517) - `clangtk`: Assertion `E->isPRValue() && E->getType()->hasPointerRepresentation()' failed
- [63521](https://github.com/llvm/llvm-project/issues/63521) - `clangtk`: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
- [63528](https://github.com/llvm/llvm-project/issues/63528) - `clangtk`: fatal error: error in backend: Broken function found, compilation aborted!
- [63548](https://github.com/llvm/llvm-project/issues/63548) - clang: Assertion `castIsValid(op, S, Ty) && "Invalid cast!"' failed.
- [63566](https://github.com/llvm/llvm-project/issues/63566) - clang: Assertion `Prior->FD->hasAttr<NoUniqueAddressAttr>() && "should not have reused this field's tail padding"' failed.
- [63568](https://github.com/llvm/llvm-project/issues/63568) - clang: Assertion `Value != 0 && "Invalid CPUStr passed to CpuIs"' failed.
- [63601](https://github.com/llvm/llvm-project/issues/63601) - clang: Assertion `SizeSoFar == Layout->getSizeInBytes() && "Layout of constant struct may be incorrect!"' failed.
- [63620](https://github.com/llvm/llvm-project/issues/63620) - `clang`: Assertion `E->isNonOdrUse() != NOUR\_Unevaluated && "should not emit an unevaluated operand"' failed.
- [63943](https://github.com/llvm/llvm-project/issues/63943) - [OpenMP] clang crashes: UNREACHABLE executed at /root/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:251!
- [63945](https://github.com/llvm/llvm-project/issues/63945) - clang: Assertion `!Info.isIncomplete() && "Access to an object of an incomplete type!"' failed.
- [69054](https://github.com/llvm/llvm-project/issues/69054) - clang: Assertion `isValidGCCRegisterName(Name) && "Invalid register passed in"' failed
- [69055](https://github.com/llvm/llvm-project/issues/69055) - clang: Assertion `IsValid && "Failed to parse output constraint"' failed.
- [69063](https://github.com/llvm/llvm-project/issues/69063) - clang: fatal error: error in backend: Broken module found, compilation aborted!
- [69066](https://github.com/llvm/llvm-project/issues/69066) - clang -O1: fatal error: error in backend: Broken module found, compilation aborted!
- [69071](https://github.com/llvm/llvm-project/issues/69071) - clang: Assertion `castIsValid(op, S, Ty) && "Invalid cast!"' failed.
- [69220](https://github.com/llvm/llvm-project/issues/69220) - clang: crash at clang::CodeGen::CodeGenFunction::EmitAtomicExpr
- [63547](https://github.com/llvm/llvm-project/issues/63547) - `clangtk`: Assertion `NumBits >= MIN\_INT\_BITS && "bitwidth too small"' failed.
- [63306](https://github.com/llvm/llvm-project/issues/63306) - `clangtk`: Assertion `NumBits <= MAX\_INT\_BITS && "bitwidth too large"' failed.
- [63550](https://github.com/llvm/llvm-project/issues/63550) - clang: Assertion `VarSize == CstSize && "Emitted constant has unexpected size"' failed.
- [63570](https://github.com/llvm/llvm-project/issues/63570) - `clangtk` crashes when processing a simple assignment loop wrapped by `#pragma omp unroll`
- [69214](https://github.com/llvm/llvm-project/issues/69214) - [Clang][OpenMP] Assertion `IsRegistered && "private var already registered as private"' failed.
- [69203](https://github.com/llvm/llvm-project/issues/69203) - clang: 18: continue to consume memory until OOM
- [63004](https://github.com/llvm/llvm-project/issues/63004) - clangtk assertion failure: clang::QualType clang::FunctionProtoType::getParamType(unsigned int) const: Assertion `i < getNumParams() && "invalid parameter index"' failed.
- [63518](https://github.com/llvm/llvm-project/issues/63518) - `clangtk` crashes at clang::Expr::IgnoreImpCasts
- [63527](https://github.com/llvm/llvm-project/issues/63527) - `clangtk` crashes at clang::StmtVisitorBase<llvm::make\_const\_ptr, (anonymous namespace)::StmtProfiler, void>::Visit(clang::Stmt const\*)
- [63567](https://github.com/llvm/llvm-project/issues/63567) - clang: Assertion `Diagnosed && "failed to diagnose bad conversion"' failed.
- [63715](https://github.com/llvm/llvm-project/issues/63715) - `clang-17`: Assertion `false && "Unexpected unary operator!"' failed.
- [69058](https://github.com/llvm/llvm-project/issues/69058) - clang: UNREACHABLE executed at /root/llvm-project/clang/lib/AST/ASTContext.cpp:10670!
- [69064](https://github.com/llvm/llvm-project/issues/69064) - [Clang][OpenMP] Assertion `Diagnosed && "failed to diagnose bad conversion"' failed
- [69065](https://github.com/llvm/llvm-project/issues/69065) - Assertion `Result.isAbsent() && "local created multiple times"' failed.
- [69084](https://github.com/llvm/llvm-project/issues/69084) - clang: version 18: Assertion `getNumParams() == params.size() && "NumParams overflow!"' failed.
- [69205](https://github.com/llvm/llvm-project/issues/69205) - clang: Assertion `isa<To>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
- [69208](https://github.com/llvm/llvm-project/issues/69208) - clang: 18: Crash at clang::Sema::AddAllocAlignAttr(clang::Decl\*, clang::AttributeCommonInfo const&, clang::Expr\*)
- [69213](https://github.com/llvm/llvm-project/issues/69213) - clang: 18: Assertion `Init < getNumInits() && "Initializer access out of range!"' failed.
- [69345](https://github.com/llvm/llvm-project/issues/69345) - clang: 18: Assertion `isStruct() && "Invalid accessor"' failed.
- [69468](https://github.com/llvm/llvm-project/issues/69468) - clang: 18: Assertion `getType(Base)->isPointerType() || getType(Base)->isArrayType()' failed.
- [69470](https://github.com/llvm/llvm-project/issues/69470) - clang: Assertion 'idx < size()' failed. In function `clang::Sema::tryCaptureVariable`
- [69874](https://github.com/llvm/llvm-project/issues/69874) - clang: 18: Assertion `isLValue()' failed.
- [86904](https://github.com/llvm/llvm-project/issues/86904) - [Clang][Crash] Assertion failure in Clang 18 when using pre-increment operator on `\_Complex double` struct fields
- [86906](https://github.com/llvm/llvm-project/issues/86906) - [Clang][Crash] Clang-18 Assertion `iter->getSecond() >= 0 && "Found a negative number of references to a VarDecl"' failed.
- [63520](https://github.com/llvm/llvm-project/issues/63520) - `clangtk`: Assertion `Block == Succ' failed.
- [69472](https://github.com/llvm/llvm-project/issues/69472) - clang: 18: Assertion `E->getType()->isSpecificBuiltinType(BuiltinType::Overload)' failed.
- [62711](https://github.com/llvm/llvm-project/issues/62711) - clang-15 crashes on \_\_builtin\_va\_arg
- [63863](https://github.com/llvm/llvm-project/issues/63863) - `clang-17`: Assertion `Init->containsErrors() && "Dependent code should only occur in error-recovery path."' failed.
- [69069](https://github.com/llvm/llvm-project/issues/69069) - [Clang][OpenMP] Assertion `detail::isPresent(Val) && "dyn\_cast on a non-existent value"' failed.
- [69085](https://github.com/llvm/llvm-project/issues/69085) - [Clang][OpenMP] Assertion `!isNull() && "Cannot retrieve a NULL type pointer"' failed.
- [69198](https://github.com/llvm/llvm-project/issues/69198) - [Clang][OpenMP] Assertion `OriginalTy->isArrayType()' failed.
- [69200](https://github.com/llvm/llvm-project/issues/69200) - [Clang][OpenMP] Assertion `!isNull() && "Cannot retrieve a NULL type pointer"' failed.
- [69352](https://github.com/llvm/llvm-project/issues/69352) - clang: Assertion `NumPositiveBits <= BestWidth && "How could an initializer get larger than ULL?"' failed.
- [62829](https://github.com/llvm/llvm-project/issues/62829) - clang-15 hangs on a simple syntax error
- [69194](https://github.com/llvm/llvm-project/issues/69194) - clang: 18: hangs on a simple syntax error
- [62830](https://github.com/llvm/llvm-project/issues/62830) - clang-15 hangs on a valid program
- [62834](https://github.com/llvm/llvm-project/issues/62834) - clang crashes on pointer qualified by \_\_seg\_gs when optimization is enabled
- [63102](https://github.com/llvm/llvm-project/issues/63102) - clang-17 crashes at llvm::isPotentiallyReachable
- [63762](https://github.com/llvm/llvm-project/issues/63762) - `clang-17`: Assertion `IterCnt++ < 1000 && "Iterative simplification didn't converge!"' failed.
- [69211](https://github.com/llvm/llvm-project/issues/69211) - clang: 18: Assertion `ProcessedCount[V] < 100 && "Seem to have processed the same Value a lot"' failed.
- [69059](https://github.com/llvm/llvm-project/issues/69059) - clang-18: Assertion `New->getType() == getType() && "replaceAllUses of value with new value of different type!"' failed.
- [63304](https://github.com/llvm/llvm-project/issues/63304) - `clangtk` hangs when processing expression with deep dependency and type `intptr\_t`
- [63307](https://github.com/llvm/llvm-project/issues/63307) - `clangtk` hangs when compiling a simple program that involves loading from or storing to array elements
- [69090](https://github.com/llvm/llvm-project/issues/69090) - clang: 18: hangs with a short program in Polly's runIslScheduleOptimizer
- [63305](https://github.com/llvm/llvm-project/issues/63305) - `clangtk` crashes at a large floating constant
- [69218](https://github.com/llvm/llvm-project/issues/69218) - clang: trunk: crashes at clang::APValue::printPretty, Assertion `isa<ArrayType>(CanonicalType)' failed.

### GCC

- [110500](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110500) - gcc: internal compiler error: tree check: expected class 'type', have 'exceptional' (error\_mark) in c\_parser\_omp\_clause\_allocate
- [110575](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110575) - gcc: internal compiler error: tree check: expected class 'type', have 'exceptional' (error\_mark) in build\_aligned\_type
- [110603](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110603) - [14 Regression] GCC, ICE: internal compiler error: in verify\_range, at value-range.cc:1104 since r14-255
- [110667](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110667) - gcc-14, ICE: internal compiler error: in replace\_reg, at reg-stack.cc:722
- [110682](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110682) - [12/13/14/15 Regression] ICE: internal compiler error: in gimplify\_expr after error
- [110693](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110693) - internal compiler error: Segmentation fault with invalid gimple
- [110699](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110699) - [12/13/14 Regression] internal compiler error: tree check: expected array\_type, have error\_mark in array\_ref\_low\_bound, at tree.cc:12754
- [110736](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110736) - gcc: internal compiler error: in change\_stack, at reg-stack.cc:2709
- [110737](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=110737) - GCC: internal compiler error: Segmentation fault when processing \_\_builtin\_isinf and \_Atomic long double together
- [111809](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111809) - gimpleFE: unreferenced inline function with \_GIMPLE(ssa) definition causes ICE
- [111810](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111810) - rtlFE: nested inline RTL function cause ICE
- [111811](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111811) - [14/15 Regression] ICE with vector float bitfield after error
- [111816](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111816) - [gimple FE] ICE with \_GIMPLE(ssa) and 2 returns
- [111817](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111817) - ICE in expand\_asm\_stmt with invalid characters in inline-asm output operands
- [111818](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111818) - [11 Regression] ICE with \_\_builtin\_memcpy with volatile and constants
- [111819](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111819) - GCC: 14: internal compiler error: in fold\_offsetof, at c-family/c-common.cc:6877
- [111820](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111820) - [13 Regression] Compiler time hog in the vectorizer with `-O3 -fno-tree-vrp`
- [111821](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111821) - [11/12/13/14/15 Regression] OOM with packed struct and stack variable with copy
- [111833](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111833) - [13/14 Regression] GCC: 14: hangs on a simple for loop
- [111834](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111834) - GCC: 14: out of memory when \_\_builtin\_return\_address receive a large constant
- [111836](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111836) - ICE with undefined types and undefined variables with \_GIMPLE and starting at ssa
- [111855](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111855) - GCC: 14: internal compiler error: tree check: expected class 'type', have 'exceptional' (error\_mark) in useless\_type\_conversion\_p, at gimple-expr.cc:265
- [111856](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111856) - ICE with return type of typedef of void and `#pragma omp declare simd`
- [111862](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111862) - GCC: internal compiler error: tree check: expected class 'type', have 'exceptional' (error\_mark) in c\_parser\_omp\_clause\_reduction, at c/c-parser.cc:16234
- [111865](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111865) - [11/12/13/14 Regression] ICE with register decl and extern decl with the same asm name
- [111882](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111882) - [13 Regression] : internal compiler error: in get\_expr\_operand in ifcvt with Variable length arrays and bitfields inside a struct
- [111911](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111911) - [11/12/13 Regression] ICE with integer overflow converting to \_Bool
- [111912](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111912) - internal compile error when apply \_\_transaction\_atomic on logical or expr
- [111913](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111913) - [14 Regression] ICE with \_\_builtin\_popcount(X) + \_\_builtin\_popcount(Y) simplification
- [111914](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111914) - ICE with function pointer array as argument with non-constant size
- [111915](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111915) - [14 Regression] ICE in vect with option `-O2 -fno-tree-vrp -fno-tree-dominator-opts -fno-tree-ccp` since r14-2117-gdd86a5a69cbda4
- [111921](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111921) - [11/12/13/14/15 Regression] ICE with nested function after an error since r6-205-g5c4abbb8e80153
- [111922](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111922) - [11/12/13/14 Regression] ICE in cp with -O2 -fno-tree-fre
- [112487](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112487) - [11/12/13 Regression] ICE in setup\_one\_parameter with conflicting types
- [112488](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112488) - [14 Regression] ICE in make\_ssa\_name\_fn with VLA inside type and inlining since r14-1142
- [112489](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112489) - GCC: 14: internal compiler error: in check\_loop\_closed\_ssa\_def, at tree-ssa-loop-manip.cc:647
- [112493](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112493) - [11/12/13/14/15 Regression] ICE in verify\_sra\_access\_forest with scalar\_storage\_order
- [112494](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112494) - ICE in ix86\_cc\_mode, at config/i386/i386.cc:16477
- [112495](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112495) - [11 Regression] ICE: verify\_gimple failed (after vectorizer) with named address space (\_\_seg\_gs )
- [112496](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112496) - [13/14 Regression] ICE: in vectorizable_nonlinear_induction, at tree-vect-loop.cc with bit fields
- [112499](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112499) - [11/12/13/14 Regression] ICE in output\_operand: invalid expression as operand with addition of 2 differences of address of labels
- [112500](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112500) - GCC: 14: internal compiler error: in expand\_asm\_stmt, at cfgexpand.cc:3419
- [112501](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112501) - ICE in extract\_insn, at recog.cc:2804 with \_\_builtin\_speculation\_safe\_value with short type on x86\_64
- [112502](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112502) - GCC: 14: internal compiler error: in get\_predictor\_value, at predict.cc:2695
- [112503](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112503) - GCC: 14: internal compiler error: Segmentation fault at gsi\_start\_edge(edge\_def\*)
- [112504](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112504) - ICE in operand\_equal\_p, at fold-const.cc:3313 with function return struct containing an array and comparing the array to the same thing
- [112505](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112505) - [11 Regression]  internal compiler error: in build\_vector\_from\_val, at tree.cc:2104 since r10-4076
- [112509](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112509) - [14 Regression] internal compiler error: in verify\_range, at value-range.cc:1132
- [112511](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112511) - ICE in type\_contains\_placeholder\_1: SRA vs \_BitInt
- [112512](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=112512) - ICE in lra\_split\_hard\_reg\_for, at lra-assigns.cc:1861 with using RAX as global register
- [114203](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114203) - [13 Regression] Miscompilation: A possible miscompilation in GCC 13 and 14 with option -Os
- [114206](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114206) - [11/12/13/14/15 Regression] recursive function call vs local variable addresses
- [114207](https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114207) - [12/13/14/15 Regression] modref gets confused by vecotorized code ` -O3 -fno-tree-forwprop` since r12-5439

## Step 6: Run Macro Fuzzer

We have also developed an optimized macro fuzzer for compiler fuzzing. However, it's important to note that this macro fuzzer requires `cgroup`, which is not supported in Docker environments. Therefore, you will need to transfer the macro fuzzer to a physical machine for execution.

### Initial Setup

Before using the macro fuzzer, you must first download and configure instrumented compilers that are specially designed for macro fuzzing. Perform the following steps:

1. **Set Up the Environment**:
   Navigate to the fuzzer's directory and run the setup script:
   ```sh
   cd /root/fuzzers/CForgeFuzzer
   bash setup.sh
   ```
   This script will download the latest versions of GCC and Clang, and compile them with a branch coverage instrumenter. The resulting binaries will be stored in the `objects` directory within `CForgeFuzzer`.

2. **Configure the Macro Fuzzer**:
   Modify the `configs.py` file to configure the fuzzer settings:
   - Update the `compilers_to_register` list with the paths to the newly compiled compiler binaries:
   ```
   CForgeFuzzer/objects/gcc-project-trunk.obj/install/bin/gcc
   CForgeFuzzer/objects/gcc-project-trunk.instrum/install/bin/gcc
   CForgeFuzzer/objects/llvm-project-trunk.obj/bin/clang
   CForgeFuzzer/objects/llvm-project-trunk.instrum/bin/clang
   ```

### Running the Macro Fuzzer

After completing the setup, you can start the macro fuzzer using the following commands:

```sh
mkdir -p /root/ws-macro
cd /root/ws-macro
python3 /root/fuzzers/CForgeFuzzer/Forger.py -j 10 --wdir $(pwd) --load-seeds-from ~/seeds
```

These commands prepare a new working directory and initiate the fuzzer with a concurrency of 10 jobs, using the specified working directory and seed files.