Published January 25, 2026 | Version v1
Other Open

Scripts, Software and Dataset for "Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework"

Authors/Creators

  • 1. ROR icon Universidade Estadual do Ceará
  • 2. Universidade Estadual do Ceara Centro de Ciencias da Saude
  • 3. ROR icon Universidade Federal do Ceará

Description

# Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework

This repository is the replication package of the paper
"Benchmarking LLM Commit Message Generation through a Developer-centric Pairwise Preference Framework".

The purpose of this repository is to provide the code, data and scripts required to reproduce the
experiments and metrics described in the paper.

## Repository contents

- `automated_metrics/` — Scripts for automatic metric calculation and statistics (e.g., METEOR).
- `diff_mining/` — Diff data, samples and mining scripts used in experiments.
- `dataset_experiment/` — Dataset of diffs evaluated during our commit message preference analysis.
- `elo/` — Scripts and results related to preference ranking analysis, numbered scripts should be executed in ascending order.
- `CoMEHT/` — The back and frontend implementations of our Commit Message Evaluation Helper Tool for replication purposes.

Files

Files (1.3 MB)

Name Size Download all
md5:b035e975a5ef8cc950dc14ef2f213e12
1.3 MB Download