Agreement Between Large Language Models and Humans in Research Proposal Review — Data and Code

This repository contains the data and code required to reproduce the analyses, statistical results, and figures presented in the associated manuscript. Files are organized by function and described below.

All research proposals are anonymized and labeled with non-identifying identifiers (A, B, C, …). Reviewer identities were never provided to the authors. To prevent inadvertent disclosure, all free-text review content from both human reviewers and large language models (LLMs) has been removed; only the numerical evaluation data required to reproduce the reported analyses are included.

Data files

Human_raw_scores.csv

Individual numerical scores assigned by human reviewers, one row per reviewer × proposal × criterion. Used to compute panel-level summary statistics and the inter-reviewer reliability metrics reported in the manuscript.

Human.csv

Proposal-level human reference scores (one row per proposal) used as the human benchmark against which LLM scores and rankings are compared.

LLM_data_combined_clean_filtered.csv

All numerical scores generated by the evaluated LLMs. Preprocessed to remove evaluations in which a model failed to return one or more required numerical scores (see Data provenance and known limitations for counts).

review_criteria.txt

The evaluation criteria and rating scales. See the note under Data provenance regarding the 2023 vs. 2024 criterion naming.

Data dictionary

Human_raw_scores.csv

Human.csv

LLM_data_combined_clean_filtered.csv

Data provenance and known limitations

We document the following so that users can interpret the data accurately.

Two review cycles, combined. The 28 proposals come from two internal seed grant cycles: 15 from 2023 and 13 from 2024 (Cycle column). For 2024 proposals, proposal-level means in Human.csv are the official institute panel means; for 2023 proposals they are computed from the individual ratings in Human_raw_scores.csv.

Reviewers per proposal. Panels ranged from 3 to 6 reviewers. Reviewer identities were never provided; the human inter-reviewer reliability is therefore estimated with a one-way random-effects model (ICC(1,1)), which is the appropriate model when each proposal is rated by a different, unidentified set of reviewers.

Six 2024 reviews not present at the individual level. For six 2024 proposals (G, R, T, W, X, Z), one reviewer's scores were submitted without written comments and are not included in Human_raw_scores.csv. For these proposals, Human.csv carries the official institute panel means, so the panel mean in Human.csv and the mean recomputed from Human_raw_scores.csv differ slightly. The reproducibility check in 06_Human_data.ipynb confirms exact agreement for all proposals with complete individual records and reports the expected small differences for these six.

Out-of-scale LLM Overall ratings. A small number of responses (≈1.2% of reviews, almost entirely from gpt-3.5-turbo) rated the overall recommendation on a 1–5 scale rather than the requested 1–3 scale, in a format the parser could not distinguish. The analysis code masks values outside the valid range before computing any Overall-based result; the composite Score does not use Overall and is unaffected.

Excluded LLM evaluations. Evaluations in which a model failed to return one or more required numerical scores were removed prior to analysis. The file provided here is the post-exclusion (analyzed) dataset.

Code notebooks

Two groups of notebooks are provided: (i) the LLM evaluation pipeline and (ii) statistical analysis and figure generation.

LLM evaluation pipeline (Notebooks 1–4)

Documentation of the methodology used to generate the LLM evaluations. These use synthetic examples and contain no confidential data, API credentials, or real proposal text.

1. 01_pipeline_overview.ipynb — architecture, configuration, criteria, output format, evaluation matrix.
2. 02_prompting_strategies.ipynb — one-shot and chain-of-thought prompting; example selection; text vs. vision input.
3. 03_response_parsing.ipynb — regex extraction of ratings and comments; error handling; decimal ratings.
4. 04_example_evaluation.ipynb — end-to-end workflow on synthetic data.

Statistical analysis and figures (Notebooks 5–6)

1. 05_Data_Processing.ipynb — ANOVA and effect sizes; empirical absolute score differences; ICC(2,1) and Spearman correlations vs. the human panel; Monte Carlo comparisons; scatter, slope, and bias figures.
2. 06_Human_data.ipynb — ICC(1,1)/ICC(1,k) for human reviewers with bootstrap confidence intervals; Monte Carlo single-reviewer vs. leave-one-out panel rank agreement; consistency check of Human.csv against Human_raw_scores.csv.

Reproducibility

Running 05_Data_Processing.ipynb and 06_Human_data.ipynb against the included data reproduces the statistical results and figures in the manuscript. Notebooks 1–4 document the evaluation pipeline using synthetic examples.

Requirements

pandas
numpy
scipy
statsmodels
scikit-learn
matplotlib

The LLM evaluation pipeline notebooks (1–4) additionally use the packages listed in requirements.txt.

Citation

If you use this data or code, please cite the associated manuscript and this archive:

[Author list]. Agreement Between Large Language Models and Humans in Research
Proposal Review. [Year]. Zenodo. https://doi.org/[concept DOI]

License

Data are released under CC BY 4.0; code is released under the MIT License.