Dataset for the paper "When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness"
Authors/Creators
Description
This dataset is associated with the paper titled “When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness”. The study examines whether LLM-generated refactorings of individual Java files are ready for adoption in the surrounding repository. Ten production files were selected from each of 15 open-source Java repositories (150 files) and refactored by five LLM configurations (Claude Opus 4.8, DeepSeek v4 Pro, Gemini 3.1 Pro Preview, GPT-5.5 and Qwen 3.6 Plus) under two generation conditions (a single prompt and a LangGraph-based multi-agent pipeline), yielding 1,500 generation attempts and 1,455 eligible refactoring candidates. Each candidate is assessed for contract/dependency risk, detected broken-caller evidence, a behavioural-equivalence indicator, targeted tests and adoption readiness. The data are stored in a Microsoft Excel file that consists of several worksheets. Below is a brief description of each worksheet.
| No. | Worksheet | Description (as shown on the Zenodo record) |
| 1 | README | The 'README' worksheet provides an overview of the study design, the LLM configurations, the risk indicators and the four adoption-readiness categories (Low-risk, Review-needed, High contract/dependency risk and Inconclusive), the analytical cohorts, the data flow between worksheets, missing-value handling, the statistical methods, data and code availability, and the limitations. |
| 2 | Paper_map | The 'Paper_map' worksheet identifies the worksheet and cell range of every table, figure, cohort figure and research-question result reported in the paper, with a direct link to each location and notes on values that differ between the paper and the dataset. |
| 3 | Codebook | The 'Codebook' worksheet defines the terms and analytical cohorts used in the paper and every column of the raw and processed worksheets, maps the logged field names to the paper terminology, and indicates which fields are used in the paper. |
| 4 | Study_Setup | The 'Study_Setup' worksheet presents the subject systems (Table 1), the pipeline stages, the LLM configurations, the model invocation settings (Table 4), the analytical cohorts and the repository profiles of the 15 Java repositories. |
| 5 | Raw_Selected_Files | The 'Raw_Selected_Files' worksheet contains the 150 selected production Java files (ten per repository) with their repository-exposure indicators, including LOC, PMD smell count, fan-in, fan-out, public and caller-used APIs, downstream callers and the dependency-risk score, with values unchanged. |
| 6 | Raw_Single_Prompt | The 'Raw_Single_Prompt' worksheet contains the 750 single-prompt generation attempts (150 files × 5 LLMs) across 64 logged fields, with the logged values unchanged. |
| 7 | Raw_Multi_Agent | The 'Raw_Multi_Agent' worksheet contains the 750 multi-agent generation attempts (150 files × 5 LLMs) across 64 logged fields, with the logged values unchanged. |
| 8 | Generation_Attempts | The 'Generation_Attempts' worksheet contains the 1,500 generation attempts with eligibility, contract and overall risk, broken-caller evidence, the behavioural-equivalence indicator, risk-reason coverage, targeted-test outcome, adoption-readiness category and artifact availability. It is the source data for RQ2–RQ5. |
| 9 | RQ1_Exposure | The 'RQ1_Exposure' worksheet presents Tables 5–7, which report the repository exposure of the 150 selected files by project and overall, the Kruskal–Wallis comparisons across projects with Holm-adjusted p-values, and the Spearman correlations among the exposure indicators, followed by the file-level source data. |
| 10 | RQ2_Risk_Indicators | The 'RQ2_Risk_Indicators' worksheet presents Tables 8–10, which report the prevalence of High contract/dependency risk, detected broken-caller evidence and the behavioural-equivalence indicator among the 1,455 eligible candidates, the risk mechanisms among the 864 reason-covered candidates and the threshold sensitivity analysis, followed by the source data for Figure 3 and the candidate-level data. |
| 11 | RQ3_Comparisons | The 'RQ3_Comparisons' worksheet presents Tables 11–14, which report the paired single-prompt versus multi-agent comparisons over 705 complete condition pairs (McNemar and Wilcoxon tests), the mechanism-level comparisons, and the model comparisons over complete five-model blocks (Cochran’s Q, Friedman and Holm-adjusted post-hoc tests), followed by the pair-level and block-level data. |
| 12 | RQ4_Adoption_Readiness | The 'RQ4_Adoption_Readiness' worksheet presents Tables 15–17, which report the four adoption-readiness categories with Wilson confidence intervals, the coverage of each verification-evidence layer (including 395 test-eligible candidates and 65 conclusive test outcomes) and the categories per LLM, followed by the candidate-level evidence data. |
| 13 | RQ5_Associations | The 'RQ5_Associations' worksheet presents Tables 18–19, which report file-level and candidate-level associations between source-file characteristics and the three outcomes (Spearman’s ρ and Cliff’s δ with Holm-adjusted p-values), the risk-mechanism overlap and edit-telemetry analyses, followed by the file-level and candidate-level data. |
| 14 | Raw_Data_Manifest | The 'Raw_Data_Manifest' worksheet lists the original raw data files of the study (results exports, repository profiles, risk-reason, test and edit-telemetry records), with the number of records, primary identifiers, related research questions and availability status of each file. |
| 15 | Artifact_Manifest | The 'Artifact_Manifest' worksheet records, for each of the 1,500 generation attempts, which artifacts are retained (raw response, candidate source, diff, profiles, verification report, compile and test logs). Candidate source artifacts are retained for 187 eligible candidates and complete artifact bundles for 166 eligible candidates. |
Files
Files
(2.0 MB)
| Name | Size | |
|---|---|---|
|
md5:f35fd577443f56e2ef7f128a3dc4c0ee
|
2.0 MB | Download |