There is a newer version of the record available.

Published October 10, 2024 | Version v4
Dataset Open

Artifact of Search+LLM-based Testing for ARM Simulators

  • 1. ROR icon King's College London
  • 2. ROR icon University College London
  • 3. ROR icon University of California, Davis

Description

SearchSYS tool can be downloaded here: a239d3c (for this work).

To process the fuzzed test inputs, post-fuzzing, the *.sh here can run on the output/default/queue and crash folders.

The rest are the data collected during experiments: (1) the Excel file contains the full analysis while (2) *.tar.gz are all outputs from the fuzzing stage. The LLM data is taken from here. You can follow the instructions here to install and run SearchSYS for X86.

 

The record here details how to run SearchSYS on an ARM machine (arm64-ubuntu-2204).

 

Experiments

Requirements:

  • GCC-11
  • wget, zip, unzip (any version)
  • LLVM-13+ (we tested with LLVM-13, but any newer version should be okay)
  • Python3 (tested with Python 3.8 and 3.10)
  • Other packages:
sudo apt-get update
sudo apt-get upgrade
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get -y install gcc-11 g++-11 cpp-11 wget lsb-release gnupg software-properties-common
sudo rm /usr/bin/cpp /usr/bin/gcc /usr/bin/g++  /usr/bin/gcov  /usr/bin/c++
sudo rm /usr/bin/cc
sudo ln -s /usr/bin/cpp-11 /usr/bin/cpp
sudo ln -s /usr/bin/gcc-11 /usr/bin/gcc
sudo ln -s /usr/bin/gcc-11 /usr/bin/cc
sudo ln -s /usr/bin/g++-11 /usr/bin/g++
sudo ln -s /usr/bin/g++-11 /usr/bin/c++
sudo ln -s /usr/bin/gcov-11 /usr/bin/gcov
wget https://apt.llvm.org/llvm.sh
sudo chmod +x llvm.sh
sudo ./llvm.sh 13
sudo ln -s /usr/bin/llvm-config-13 /usr/bin/llvm-config
sudo apt-get install -y lld-13 llvm-13-dev clang-13
sudo apt-get install -y lld llvm llvm-dev clang
sudo apt-get install -y build-essential python3-dev automake cmake git flex bison libglib2.0-dev libpixman-1-dev python3-setuptools cargo libgtk-3-dev
sudo apt-get install -y ninja-build cmake
sudo apt-get install -y git make cmake llvm gdb coreutils
sudo apt-get install -y gcc-11-plugin-dev
sudo apt -y autoremove
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-13 1300 --slave /usr/bin/clang++ clang++ /usr/bin/clang++-13 
sudo update-alternatives --install /usr/bin/llvm-config llvm-config /usr/bin/llvm-config-13 1300
LLVM_CONFIG=/usr/bin/llvm-config-13

We attached all installed packages in list-of-installed-packages.log.

 

Reproduce ICSE-SEIP 2025 Evaluation:

To reproduce the experiments follow these commands (Unix system):

## CREATE A NEW USER (you can also use your own user but then you will need to amend /home/debian and debian, accordingly) ##
sudo adduser debian
# if needed run:
# sudo passwd debian
# and set some simple password
sudo usermod -aG sudo debian
sudo mkdir -p /home/debian
sudo usermod -d /home/debian debian
su - debian

## GET DATA - 7 set of LLM-generated C programs ##
cd /home/debian
wget "https://zenodo.org/records/10999115/files/LLM_test_inputs-4FuzzExp2.zip" # Get the data
unzip -n LLM_test_inputs-4FuzzExp2.zip -d /home/debian/experiment
cd /home/debian/experiment/ 
sed -i "s:/home/ubuntu/experiment-7/:/home/debian/experiment/:g" */input/*.txt
cat /home/debian/experiment/TinyLlama-cmin/input/test_input_574567374417673.txt
 
## COMPILE BINARIES - build the test binaries and tailor the paths in the seeds ##

### Model CodeBooga ###
cd /home/debian/experiment/CodeBooga  
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done 
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf CodeBooga-cmin/binary
cp -r CodeBooga/binary CodeBooga-cmin/binary
cd /home/debian/experiment/CodeBooga-cmin/input
sed -i 's\/CodeBooga/\/CodeBooga-cmin/\g' *txt

### Model Llama ###
cd /home/debian/experiment/Llama  
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf Llama-cmin/binary
cp -r Llama/binary Llama-cmin/binary
cd /home/debian/experiment/Llama-cmin/input
sed -i 's\/Llama/\/Llama-cmin/\g' *txt

### Model Magicoder ###
cd /home/debian/experiment/Magicoder
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf Magicoder-cmin/binary
cp -r Magicoder/binary Magicoder-cmin/binary
cd /home/debian/experiment/Magicoder-cmin/input
sed -i 's\/Magicoder/\/Magicoder-cmin/\g' *txt

### Model TinyLlama ###
cd /home/debian/experiment/TinyLlama
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf TinyLlama-cmin/binary
cp -r TinyLlama/binary TinyLlama-cmin/binary
cd /home/debian/experiment/TinyLlama-cmin/input
sed -i 's\/TinyLlama/\/TinyLlama-cmin/\g' *txt

### Model Phi ###
cd /home/debian/experiment/Phi 
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf Phi-cmin/binary
cp -r Phi/binary Phi-cmin/binary
cd /home/debian/experiment/Phi-cmin/input
sed -i 's\/Phi/\/Phi-cmin/\g' *txt

### Model gpt3.5-new ###
cd /home/debian/experiment/gpt3.5-new  
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf gpt3.5-new-cmin/binary
cp -r gpt3.5-new/binary gpt3.5-new-cmin/binary
cd /home/debian/experiment/gpt3.5-new-cmin/input
sed -i 's\/gpt3.5-new/\/gpt3.5-new-cmin/\g' *txt

### Model gpt3.5-old ###
cd /home/debian/experiment/gpt3.5-old
for prog in raw/*.c; do gcc -O3 "$prog" -o "$prog.o" && mv $prog.o binary/ || true; done
rm  binary/fuzz_* || true
rm  .DS_Store || true
cd /home/debian/experiment/
rm -rf gpt3.5-old-cmin/binary
cp -r gpt3.5-old/binary gpt3.5-old-cmin/binary
cd /home/debian/experiment/gpt3.5-old-cmin/input
sed -i 's\/gpt3.5-old/\/gpt3.5-old-cmin/\g' *txt
 
At this point all data for fuzzing is ready. You need now to get SearchSYS and build the SUT (gem5) from source.


## GET AFL++ SearchSYS ## cd /home/debian git clone https://github.com/karineek/SearchGEM5.git git clone https://github.com/AFLplusplus/AFLplusplus.git cd AFLplusplus git checkout f596a297c4de6a5e1a6fb9fbb3b4e18124a24f58 cp ../SearchGEM5/src/gem5-afl/afl-fuzz-init.c src/afl-fuzz-init.c AFL_USE_ASAN=0 make ### The compilation shall state that it is ARM architecture: ### ``` debian@node0:~/AFLplusplus$ AFL_USE_ASAN=0 make [*] Compiling AFL++ for OS Linux on ARCH aarch64 [+] ZLIB detected [!] Note: skipping x86 compilation checks (AFL_NO_X86 set). [+] shmat seems to be working. [+] Python 3.8.10 support seems to be working. [+] Everything seems to be working, ready to compile ... ```

then you need to check that AFL is build correctly:
```
Build Summary: [+] afl-fuzz and supporting tools successfully built [+] LLVM basic mode successfully built [+] LLVM mode successfully built [+] LLVM LTO mode successfully built [+] gcc_mode successfully built
```

Then you need to build gem5 with AFL++ instrumentation:

cd /home/debian
git clone https://github.com/BobbyRBruce/gem5-ssbse-challenge-2023.git
cd /home/debian/gem5-ssbse-challenge-2023
cp ../SearchGEM5/src/gem5-afl/SConstruct .
cp ../SearchGEM5/src/gem5-afl/SConscript src/

# If needed install:
sudo apt install python3-pip
sudo apt install scons protobuf-compiler libhdf5-dev libgoogle-perftools-dev libsystemc-dev 

# Then install the requirements for gem5:
pip install -r requirements.txt

# And build gem5 with AFL++ instrumentation:
CC=afl-cc CXX=afl-c++ scons build/ARM/gem5.opt -j 4 --no-cache
./build/ARM/gem5.opt -C

During the configuration, you need to see it built for ARM, hence (1) make sure you use build/ARM not X86, and check the configuration log to see that the right architecture has been properly selected for the build.

 

 

Files

Files (873.0 MB)

Name Size Download all
md5:1a1997ca4bbf84de70a7dc2d904f7d18
615 Bytes Download
md5:85cc44607baba7b8b0e3ab728fcaa800
2.5 kB Download
md5:2ab674dda04e86f4cd9950d4bf3f8fab
1.5 kB Download
md5:623fe09ac17081f0365666512f271d8c
8.1 MB Download
md5:320b2ca649d503e455f23d6e1f9e0aaf
315.0 kB Download
md5:21a99f3d74d1a5df39e8e98b8c556d80
509.7 kB Download
md5:1cd1f90a60f29c78416e62d37d2524d2
308.8 kB Download
md5:b67a8c68aa86b3087ac994f6cc2591e9
329.5 kB Download
md5:86e07e8267f623b37215d1ce306786c5
258.8 kB Download
md5:712c335dc0a6e0561a11193823963d43
605.1 kB Download
md5:62985cf55f89c6244c84d12e4f5814f4
53.1 MB Download
md5:455877a90c72ddc916504554c0e52813
942.1 kB Download
md5:2f2554f174dfca57c3fdff8860baf780
170.3 MB Download
md5:99246babd3a274f8f980649e9c009cd4
537.7 kB Download
md5:ede8386f29738a7026610d643964d8bb
194.2 MB Download
md5:50a27be799133431cc7e9a001f0c9d2d
437.7 kB Download
md5:9655ffab583e04361c64e7e8726feafe
211.7 MB Download
md5:99ac2788646999d741274e074b4fa430
471.7 kB Download
md5:f2cb4167ab365c6aa37ec06ea7045e86
230.8 MB Download