Mining Factors in Review Comment Generation

Mining Factors in Review Comment Generation

The appendix of complete results are shown in this repository as previewed.

1. Brief Introduction

Link: Zenodo Repository

Paper Title: Enhancing Code Review Automation by Mining Factors in Review Comment Generation

This study conducts a detailed comparison of the following aspects:

• Paradigms:

  • Pre-training and Fine-tuning (CodeT5)

  • Zero-shot Prompt Learning (GPT-4)

• Input Settings:

  • Exploration of 29 diverse input settings, formulated by combining representations of code changes, review tags, and more.

• Structural Information:

  • Introduction of a change-aware GAT component that consolidates information from both the old and the new ASTs.

Additionally, the study provides:

• A comprehensive evaluation methodology incorporating METEOR and BERTScore.

• A versatile and traceable dataset named CodeReviewCommentNet (CRCN).

• Insightful observations regarding existing gaps and suggesting potential paths forward.

2. Artifact Structure

The artifact is organized into the following main components:

In this Repository (Codes and Results):

• CRCN (codes): Scripts for generating datasets.

• pretraining_and_finetuning (codes): Scripts and results related to the "pretraining and finetuning" paradigm, including experiments with the graph component and baselines.

• zero_short_prompt_learning (codes): Scripts related to the "zero-shot prompt learning" paradigm.

• evaluation: Codes and results of various evaluations.

Dataset Repository:

• CRCN (dataset_short): The processed short dataset.

• CRCN (dataset_long): The processed long dataset.

• CRCN (raw_data): The raw crawled dataset.

Model Repositories:

• Baselines

• Change Aware Graph Component

• Pretraining and Finetuning (long)

• Pretraining and Finetuning (short) (C1-C5)

• Pretraining and Finetuning (short) (L1-L8)

• Pretraining and Finetuning (short) (S1-S8)

• Pretraining and Finetuning (short) (V1-V8)

Note 1: Our "zero-shot prompt learning" implementation is based on the APIs of GPT-4, hence it does not necessitate concrete models.

Note 2: Please refer to the comments in the specific files to determine the name and order of each individual experiment.