# Corpus reproduction package

**Paper:** *Systematic Ablation Reveals Hidden Failures in Multi-Agent AI for Science*
**Authors:** Valerio Bianchi & Dirkjan Schokker
**Submission:** Nature Machine Intelligence

## Purpose

This directory enables a third party (referee, replicator, future reader) to **reproduce the exact 200-paper evaluation corpus** used in the paper. The corpus PDFs themselves are NOT redistributed because most of the papers are subject to publisher copyright. Instead this package provides:

1. `corpus_papers.csv` — the authoritative manifest with one row per paper
2. `download_corpus.py` — a small Python script that downloads each paper from its source URL
3. This README

The single source of truth for the corpus identity is **`corpus_papers.csv`**. Anything that produces or consumes the corpus should agree with that file.

## Corpus composition

| Bucket | N | Source | Role in the experiments |
|---|---:|---|---|
| `core` | 50 | Bioinformatics, veterinary epidemiology, microbiome, AMR / One Health | Question generation and primary evaluation |
| `noise` | 150 | arXiv preprints (mixed domains, deliberate distractors) | Retrieval noise / scaling factorial |
| **Total** | **200** | | |

All 250 expert-curated questions in the paper were generated against the 50 core papers; the 150 noise papers are added to corpora at scale to test retrieval robustness.

## CSV schema

`corpus_papers.csv` has 11 columns:

| column | description |
|---|---|
| `paper_id` | Internal integer ID. 1–50 = core, 51–200 = noise. Stable across all experiments. |
| `bucket` | `core` or `noise` |
| `filename` | The PDF filename used in `publication/data/corpus/{bucket}/` |
| `title` | Paper title (core) or `arXiv:<id>` placeholder (noise) |
| `domain` | Subdomain tag from `corpus_metadata.json` |
| `doi` | DOI without URL prefix, when known |
| `pmcid` | PMC accession when known |
| `arxiv_id` | arXiv ID with version, for noise papers |
| `source_url` | Direct download URL (PMC, publisher OA, or arXiv) |
| `license` | One of `PMC-OA`, `CC-BY`, `arXiv-non-exclusive`, `case-by-case`, `unknown` |
| `redistributable` | `yes` if the license clearly permits redistribution; `case-by-case` otherwise |

Counts:

- `redistributable = yes`: 42 papers (PMC Open Access subset + 4 CC-BY publisher pages)
- `redistributable = case-by-case`: 158 papers (150 arXiv non-exclusive + 8 publisher OA pages whose redistribution policy varies)

The 150 arXiv papers are marked `case-by-case` because authors choose their own license at submission time and the corpus metadata does not record per-paper choices. In practice nearly all are downloadable from arXiv directly.

## How to reproduce the corpus

```bash
# 1. Clone or unpack this directory
cd corpus_deposit

# 2. Install the only dependency
python3 -m pip install requests

# 3. Download every paper into ./papers/{core,noise}/
python3 download_corpus.py corpus_papers.csv --output ./papers/

# Optional: download only one bucket
python3 download_corpus.py corpus_papers.csv --output ./papers/ --only core
python3 download_corpus.py corpus_papers.csv --output ./papers/ --only noise
```

The script:

- Skips papers that already exist on disk
- Validates that the downloaded bytes start with `%PDF`
- Rate-limits at 0.5 s between requests
- Reports `[OK]`, `[SKIP]`, `[FAIL]` per paper and a summary at the end
- Has no dependencies beyond `requests` and the standard library

After a successful run, `./papers/core/` should contain 50 PDFs and `./papers/noise/` should contain 150 PDFs, totalling roughly 500 MB.

## Why the PDFs are not bundled

Of the 200 papers, only 42 are clearly safe to redistribute (PMC Open Access subset and a few CC-BY publisher pages). The other 158 are either:

- arXiv preprints whose redistribution license depends on the author's choice (we cannot guarantee compliance for all 150)
- Open Access publisher pages whose terms vary
- A small number of papers whose canonical download URL has expired and now needs manual retrieval

Bundling all 200 PDFs would create a redistribution liability. Bundling the 42 GREEN-class papers separately would split the corpus identity across two sources. The chosen approach — ship the manifest plus the downloader, do not redistribute any PDFs — preserves a single source of truth while staying inside copyright.

## Caveats

- **Link rot.** Publisher URLs occasionally change. If a download fails, look up the paper by its DOI or PMC ID at <https://doi.org> or <https://www.ncbi.nlm.nih.gov/pmc/>.
- **Silverchair tokens.** A small number of NAR Genomics / OUP Academic URLs use ephemeral access tokens. The CSV uses the publisher's static fallback URL where one is known.
- **Unknown licenses.** ~7 core papers do not have a verified open-access source in the manifest. Their `source_url` is a Google Scholar query that should surface the paper. These need manual retrieval.
- **arXiv versions.** The corpus uses specific arXiv version IDs (`v1`, `v2`, etc.). The downloader strips the version from the URL because arXiv serves the latest version at the canonical URL — if exact version reproducibility matters, replace `https://arxiv.org/pdf/{id}` with `https://arxiv.org/pdf/{id}{vN}` in the script.
- **Checksums.** No SHA256 manifest is shipped. Adding one is recommended for a future revision and would let a third party verify byte-level correspondence.

## Citation

If you use this corpus, please cite both the paper and the corpus deposit:

```bibtex
@article{bianchi2026mentori,
  title   = {Systematic Ablation Reveals Hidden Failures in Multi-Agent AI for Science},
  author  = {Bianchi, Valerio and Schokker, Dirkjan},
  journal = {Nature Machine Intelligence},
  year    = {2026},
  doi     = {TBD},
  note    = {Data and code deposit: https://doi.org/10.5281/zenodo.19476756}
}
```

## Contact

For licensing questions or download failures: open an issue against the accompanying code repository, or contact the corresponding author at the e-mail on the manuscript.
