Published August 25, 2025 | Version v1.0.1

Replication Package: A Comprehensive Analysis of ChatGPT's Influence on Pull Request Outcomes

  • 1. ROR icon University of Nevada, Las Vegas
  • 2. ROR icon Mbarara University of Science and Technology

Description

The rapid adoption of large language models (LLMs) like ChatGPT has introduced new dynamics in software development, particularly within pull request workflows. While prior research has examined the quality of AI-generated code, little is known about how developers actually use these suggestions in real-world collaboration. We analyze 338 pull requests from 255 GitHub repositories containing self-admitted ChatGPT usage, including 645 AI-generated snippets and 3,486 developer-authored patches. We introduce PatchTrack, a tool that classifies whether ChatGPT patches were applied, not applied, or not suggested, enabling fine-grained analysis of AI-assisted decisions. Full adoption of ChatGPT code is rare: the median integration rate was 25%. A qualitative analysis of 89 pull requests with integrated patches revealed recurring patterns of structural integration, selective extraction, and iterative refinement, showing that developers typically treat ChatGPT’s output as a starting point rather than a final implementation.

Files

Files (41.4 MB)

Name Size Download all
md5:efd2598a694784f096b591a1d0073846
41.4 MB Download

Additional details

Software

Repository URL
https://github.com/unlv-evol/PatchTrack
Programming language
Python , Jupyter Notebook
Development Status
Active