ASE26_AVR_study

# ASE26_DRSCV

## Downstream Repair of Supply Chain Vulnerabilities (DR-SCV): Dataset and Evaluation Artifacts

This repository provides the dataset and experimental artifacts for our ASE 2026 paper on downstream repair of supply chain vulnerabilities.

---

## 1. Introduction

Third-party library vulnerabilities are often difficult to address through upstream fixes due to version constraints and the high maintenance cost of patch adoption. Consequently, repairing vulnerabilities directly in downstream projects has become an increasingly important strategy.

To support research in this setting, we construct a dataset for **Downstream Repair of Supply Chain Vulnerabilities (DR-SCV)** and provide evaluation artifacts for Automated Vulnerability Repair (AVR) tools.

A key challenge in DR-SCV is that the vulnerability root cause often resides in third-party libraries and is therefore not accessible or localizable within downstream project code. This repository is designed to facilitate evaluation under this realistic constraint.

---

## 2. Dataset Overview

The dataset consists of downstream proof-of-concept (PoC) cases derived from real-world projects.

- **24 upstream projects**
- **63 downstream projects**
- Covers diverse supply chain vulnerability scenarios
- Includes downstream contexts where root cause code may be unavailable

Each dataset instance represents a DR-SCV case, including:
- Vulnerability information
- Downstream project context
- Proof-of-concept (PoC) triggering the vulnerability

---

## 3. Repository Structure

```text
ASE26_DRSCV/
├── dataset/                         # Downstream repositories with PoCs, upstream repositories, and upstream PoCs
├── prompt/                          # Prompt files used in different evaluation settings
│   ├── prompt1/                     # Prompts for RQ1 and RQ2
│   └── prompt2/                     # Prompts with injected root-cause functions(RQ3)
├── evaluated_tools_cli_scripe/      # CLI scripts for the evaluated repair tools(The tool versions used in our experiments should be downloaded)
│   ├── Agentless_JAVA/              # Our adapted Java-compatible version of Agentless
│   ├── OpenHands/                   # CLI scripts for OpenHands
│   ├── ReinFix/                     # CLI scripts for ReinFix
│   └── SWE-agent/                   # CLI scripts for SWE-agent
├── code/                            # Scripts for running experiments and evaluations(RQ1-4)
├── result/                          # Generated patches(RQ1-3) and evaluation outputs examples(RQ4)
├── LICENSE                          # MIT License
└── README.md                        # Project documentation
```

## 4. Evaluation Framework

We propose a multidimensional evaluation framework tailored to DR-SCV, which jointly considers:

- **Patch Quality**
  - Correctness and validity of generated patches

- **Preservation Integrity**
   - Detection of misleading behaviors (e.g., modification of test cases)

This framework enables systematic evaluation of AVR tools in downstream repair scenarios.

---

## 5. Quick Start

## 5.1 Requirements

Before starting, ensure your development environment meets the following requirements:

### Prerequisites

- **Java**: Version 1.8  
- **Python**: Version 3.12 

### 5.2 Downstream Project POC Usage

### Example (run a specific test class)

```bash
mvn test -Dtest=XStreamEventSerializer_ESTest
```

The PoCs of our dataset:

| targetID | pocClassName(Dtest arg) |
|----------|--------------|
| CVE-2017-7957_cqrs-lottery-master/cqrs-example | XStreamEventSerializer_ESTest |
| CVE-2017-7957_source | XMLConfig_ESTest.class |
| CVE-2018-1324_common-util | ZipUtils_ESTest |
| CVE-2018-1002202_brigen-base | ZipDelegaterZip4j_ESTest |
| CVE-2019-12415_poi-examples | CustomXMLMapping_Test |
| CVE-2019-12415_PoiSamples | CVE_2019_12415_Test |
| CVE-2020-13956_crawler-jsoup-maven | CSDNLoginApater_ESTest |
| CVE-2020-13956_wechat-ssm | HttpUtils_ESTest |
| CVE-2020-26217_cqrs-lottery-master/cqrs-example | XStreamEventSerializer_ESTest |
| CVE-2020-26217_source | XMLConfig_ESTest.class |
| CVE-2020-26258_cqrs-lottery-master/cqrs-example | CVE_2020_26258_ComparisonTest.class |
| CVE-2021-21341_cqrs-lottery-master/cqrs-example | CVE_2021_21341_Test.class |
| CVE-2021-21341_source | CVE_2021_21341_PoC_Test.class |
| CVE-2021-23899_json-sanitizer | JsonSanitizer_ESTest |
| CVE-2021-29425_FastJoin | FileReader_ESTest.class |
| CVE-2021-31812_Framework-Mobile | Pdf_ESTest.class |
| CVE-2021-31812_PDFConvert14 | CVE_2021_31812_ESTest |
| CVE-2021-35516_JavaUtils-master | ZipPwdUtil_ESTest |
| CVE-2021-37714_phoenix_interface | InterfaceAPI_ESTest |
| CVE-2021-37714_pickaxe | Scraper_ESTest |
| CVE-2021-39144_cqrs-lottery-master/cqrs-example/ | CVE_2021_39144_Test.class |
| CVE-2021-39144_source | XMLConfig_ESTest.class |
| CVE-2022-41966_cqrs-lottery-master/cqrs-example | XStreamEventSerializer_ESTest |
| CVE-2022-41966_source | XMLConfig_ESTest |
| CVE-2022-42889_java | SearchController_ESTest |
| CVE-2022-45688_base-starter | XMLUtil_ESTest |
| CVE-2022-45688_virtress | Converter_ESTest.class |
| CVE-2023-1370_microservice-with-jwt-and-microprofile | TokenUtil_ESTest.class |
| CODEC-263_DBlog-master/blog-core | PasswordUtil_ESTest |
| CODEC-270_BurpCrypto-master | AesUtil_ESTest |
| CODEC-270_DBlog-master/blog-core | PasswordUtil_ESTest |
| IO-611_FastJoin | FileReader_ESTest.class |
| IO-611_velocity-engine/velocity-engine-core | FileResourceLoader_ESTest |
| LANG-1385_ewallet/wallet-base | NumberUtil_ESTest.class |
| LANG-1385_wechat-ssm | MyNumberUtils_ESTest.class |
| LANG-1484_jjdz7-drony-refactor/Web | Validator_ESTest.class |
| LANG-1645_ewallet/wallet-base | NumberUtil_ESTest.class |
| LANG-1645_wechat-ssm | MyNumberUtils_ESTest |
| TEXT-215_geoportal-esri/geoportal | Val_TEXT215_Test |
| TEXT-215_geoportal-server/geoportal | Val_TEXT215_Test |
| Zip-263_CarStoreApi/account/account-web | ZipUtil_ESTest |
| Zip-263_ZingClient | ZFile_ESTest |
| CVE-2023-43642_flow | CVE_2023_43642_Testcase2Test |
| CVE-2023-1370_axon-server-se/axonserver | CVE_2023_1370_Testcase3 |
| CVE-2022-42004_serritor | CVE_2022_42004_DownstreamTest |
| CVE-2022-42004_vsqdeveloper-serritor | CVE_2022_42004_DownstreamTest |
| CVE-2020-13973_son-editor/son-validate-web | CVE_2020_13973_Test |
| CVE-2020-13956_apache/httpcomponents-client | HttpUtils_CVE_2020_13956_Test |
| CVE-2019-10086_bean-query | CVE_2019_10086_Testcase2 |
| CVE-2021-27568_json-configuration | JsonConfigurationTest |
| CVE-2022-25845_geek_framework | DownstreamFastjsonCVE25845Test |
| CVE-2022-25845_base-starter | ReadDataPoCTest |
| CVE-2023-34453_UltraPlaytime | CVE202334453PoCTest |
| CVE-2022-29631_ucloud-java-sdk | CrlfInjectionPocTest |
| CVE-2015-2156_webbit | NettyCookieDecodePrintTest / NettyMalformedCookieDecoderTest / HttpRequestWrapperCookieVulTest |
| CVE-2022-22976_gerenciador-viagens | CVE_2022_22976_Test |
| CVE-2017-7957_rpki-commons | CVE_2017_7957_Test |
| CVE-2021-43859_rpki-commons | CVE_2021_43859_Test |
| CVE-2021-23900_OmegaTester | CVE_2021_23900_Test |
| CVE-2018-1002201_neo | CVE_2018_1002201_Test |
| CVE-2018-1000632_tcpser4j | CVE_2018_1000632_Test |
| CVE-2018-1002201_elasticsearch-maven-plugin | CVE_2018_1002201_Test |
| CVE-2018-15756_mirage/mirage-core | CVE_2018_15756_Test |
| CVE-2022-22976_RuoYi-Vue-Multi-Tenant/multi-tenant-server | CVE_2022_22976_Test |
| CVE-2021-23899_OmegaTester | CVE_2021_23899_Test |

### 5.3 Upstream Project POC Usage

Refer to the approach used in https://github.com/WordDealer/ASE2024-Magneto

## 6. how to use our Evaluation Framework

#### 6.1 Execution on the Dataset

Run the in-house tool on the full set of 63 datasets.

#### 6.2 Evaluation Framework Usage

Use the evaluation framework, where the output is required to be a single standardized patch file with a `.diff` or `.patch` suffix.

### 6.3 Verify Whether the Test Patches Can Be Successfully Compiled

Using **SWE-agent** as an example:

```bash
./run_mvn_install_all.sh \
  --root /VESTA/resource \
  --patch-dir /VESTA/resource/patch_swe-agent/patch_swe-agent/swe-agent \
  --checker ./mvn_install_one_agentless.py \
  --result ./result.txt \
  --pass-dir ./swe-agent_installed_patch \
  --label swe-agent-2-track
```

### 6.4 Verify Whether the Test Patches Can Pass All Original Test Suites

Using **SWE-agent** as an example:

```bash
python run_mvn_test_all.py \
  --patchs=/VESTA/resource/patch_scripes/swe-agent_installed_patch/ \
  --repo=/VESTA/resource/ \
  --output=./swe-agent_patch_test_success_2_track
```

### 6.5 Verify Whether the Test Patches Can Pass the PoC Tests

Using **SWE-agent** as an example:

```bash
python patch_Dtest_all.py \
  --yaml swe-agent-gpt-4o-tasks.yaml \
  --validator ./patch_validator.py \
  --repo_root /VESTA/resource \
  --patch_root /VESTA/resource/patch_scripes/swe-agent_installed_patch \
  --out_dir ~/swe_success_patch
```

### 6.6 Verify the Integrity of the Test Patches

Using **SWE-agent** as an example:

Count how many patches modify test files:
```bash
python count_test_in_patch.py swe-agent/
```

Count how many patches modify PoC files:
```bash
python count_Dtest_in_patch.py --dir swe-agent/ --rules Dtest_class.yaml
```

---

## 7. Disclaimer

This dataset is intended for research purposes only. The included vulnerability information and proof-of-concept (PoC) cases are provided solely to support evaluation research and should be handled responsibly.