Replication Package for Evaluating a Retrieval-Augmented Conversational Agent for OSS Newcomer Support

This repository contains the replication package for a study evaluating JabRAG, a retrieval-augmented conversational agent designed to support technical newcomer questions in the JabRef open source software project.

The package includes (1) a curated dataset of newcomer questions, (2) the manual assessment protocol used during dataset construction, (3) survey instruments and anonymized survey responses collected from JabRef maintainers, (4) the exact prompt used by the RAG system, and (5) the GitHub App implementation used to deploy JabRAG in GitHub Discussions.

This artifact supports the empirical results reported in the associated ICPC RENE track paper and enables independent inspection of the dataset, evaluation materials, and system implementation. All data is provided in raw form, and no aggregation or preprocessing beyond what is documented in the paper is assumed.

The repository is intended for replication, secondary analysis, and methodological transparency in studies of conversational agents and open source software maintenance