Published January 29, 2026 | Version v2
Computational notebook Open

Untrustworthiness in LLM-based Vulnerability Repair: Benchmark and Detection

Authors/Creators

Description

This is the replication package accompanying our paper "Untrustworthiness in LLM-based Vulnerability Repair: Benchmark and Detection."

Codebase structure

The project is structured as follows.

.
├── data/                            # the labelled trustworthiness datasets
├── src/                              # source code of the project
├── out_appatch/              # the intermediate data of ExtractFix and ZeroDay
├── out_primevul/             # the intermediate data of PrimeVul's test set
├── out_sven/                    # the intermediate data of SVEN
├── requirements.txt        # required Python libraries
 

Prerequisite

You need to install Gumtree to be able to run SusVF. Please check out the installation instructions here.

If you want to use OpenAI's embedding models, please set up the environment variable OPENAI_API_KEY.

Run SusVF

Step 1. Run Gumtree to extract diffs
python gumtree_main.py --data_file <data_file> --out_dir <out_dir>
where data_file is the path to a dataset file in data/, and out_dir is the directory that stores intermediate output.
 
Step 2. Run SusVF
python susvf_main.py --data_file <data_file> --out_dir <out_dir> --model <model>
where data_file is the path to a dataset file in data/, out_dir is the directory that stores intermediate output, and model is path to the NLI machine (e.g., gpt-4o-mini, Qwen/Qwen2.5-Coder-32B-Instruct)

Files

PublishedSusVF.zip

Files (89.0 MB)

Name Size Download all
md5:bf510cb7037801ca462ffafc439652ea
89.0 MB Preview Download