Dataset for the Paper: Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair

This dataset is associated with the paper titled "Where Did the Repair First Go Wrong? Localizing the Origins of Silent Failures in Agentic Vulnerability Repair". It consists of a Microsoft Excel file with eleven worksheets, described in items (1)–(11), and the raw execution traces, provided as five compressed archives and described in item (12).

(1) The 'Overview' worksheet lists the worksheets, the main counts of the dataset, the identifiers of tasks, agent frameworks, and base models, and the correspondence between the values used in the data and the terms used in the paper.

(2) The 'By framework' worksheet reports the verification outcomes for each of the six agent frameworks, broken down by base model: the number of valid traces, traces that passed L0–L3, traces that failed L0 or L1, and silent failure candidates.

(3) The 'By model' worksheet reports the same verification outcomes for each of the six base models, broken down by agent framework.

(4) The 'Selected_Tasks_SecurityEval' worksheet lists the 75 sampled SecurityEval tasks, with one row per task. Each task records its identifier, CWE, a short description, its source, and whether each base model and each agent framework produced at least one valid trace.

(5) The 'Selected_Tasks_CVEfixes' worksheet lists the 75 sampled CVEfixes tasks, with the same columns.

(6) The 'Raw_Verification_Units' worksheet contains the 3,684 valid traces, with one row per trace. Each trace records its task, dataset, agent framework, base model, the result of each verification level (L0–L3), and the resulting outcome.

(7) The 'RQ1' worksheet reports the SAGE localization output for the 95 confirmed silent failures (RQ1). It contains summary tables for the localization status, code provenance by origin type, the origin turn relative to the introducing write, and missed detection opportunities, followed by a case-level table with the status, introducing write, code provenance, reconstruction condition, origin turn, origin type, confirmed deficiency, defect location, scores at the origin turn, and missed detection opportunities of each case.

(8) The 'RQ2_summary' worksheet reports the consistency of the SAGE output across three scoring runs of the primary judge on the 60-case subset (RQ2): the number of cases localized in all three runs, agreement on the origin type and the origin turn, and Krippendorff's alpha.

(9) The 'RQ2_Units_results' worksheet contains the case-level SAGE output of the three scoring runs for the 60-case subset.

(10) The 'RQ2_CrossJudge_results' worksheet reports the output of the second judge on the same subset and its agreement with the primary judge (RQ2): the localization status of both judges, agreement on the origin type and the origin turn, score differences at the origin turn, and agreement across all four ratings, followed by the case-level table.

(11) The 'RQ3' worksheet reports the agreement between SAGE and the two annotators (R1 and R2) on code provenance and the introducing write (RQ2, RQ3): by confirmed deficiency, reconstruction condition, agent framework, and base model; the overall percent agreement and Cohen's kappa; the confusion matrices; the consensus subset; and the case-level labels of R1, R2, and SAGE. It also reports the consistency of the origin across the three scoring runs and between the two judges for the same subgroups.

(12) 'SAGE_execution_traces_batch1.tar.gz' to 'SAGE_execution_traces_batch5.tar.gz' contain the raw execution traces. Extracting all five yields a single folder, 'SAGE_execution_traces', with one directory per execution, organized as batch / base model / agent framework / task identifier. Note: For anonymity, user names, local paths, host names, and folder names were replaced with placeholders (e.g., 'user', 'hpc.example.org', 'folder_1'), and API keys with '<REDACTED_SECRET>'.