There is a newer version of the record available.

Published March 6, 2025 | Version 1.0.0
Software Open

Replication Package: A Comprehensive Analysis of ChatGPT's Influence on Pull Request Outcomes

  • 1. ROR icon University of Nevada, Las Vegas
  • 2. ROR icon Mbarara University of Science and Technology

Description

The use of large language models (LLMs) like ChatGPT in software development has surged, yet their role in collaborative workflows, particularly in patch decision-making within pull requests (PRs), remains underexplored. This study addresses this gap by analyzing developers' ChatGPT interactions across 285 merged PRs, capturing 645 ChatGPT-generated code snippets and 3,486 patches from these PRs, systematically comparing them to assess integration patterns. Using this data, we developed PatchTrack, a classification tool that determines whether ChatGPT-generated patches were applied, not applied, or not suggested, identifying 115 applied patches, 64 not applied, and 106 cases with no patch suggestions. Additionally, we examine 53 closed PRs to assess how ChatGPT influences PR rejection. PatchTrack achieves 97.5% accuracy96.5% precision96.7% recall, and a 96.6% F1-score.

Files

PatchTrack.zip

Files (41.3 MB)

Name Size Download all
md5:43bf56220713d343e05105c517c629e2
41.3 MB Preview Download

Additional details

Software

Repository URL
https://github.com/unlv-evol/PatchTrack
Programming language
Python , Jupyter Notebook
Development Status
Active