paper2md v0.4.0 — initial public release

paper2md is a PDF→Markdown extraction pipeline tuned for scientific journal articles. It targets the layout failure modes that general-purpose converters miss on scientific corpora: two-column reading order, tables without box rules, equation-heavy text, mixed reference styles (numbered / author–year / footnote), and pre-2000 scanned PDFs with degraded OCR.

Designed for fully offline, reproducible extraction. Runs on Apple Silicon (LM Studio + MLX) or NVIDIA (vLLM + Qwen3-VL-32B-Instruct). Optional --provider openai / --provider anthropic for runs against frontier APIs.

Highlights

Two layout engines behind a single CLI

• --layout-source mineru (default) — MinerU's pipeline backend (PaddleOCR + layout)
• --layout-source marker — marker + surya OCR with paper2md's own table finder
• --layout-source hybrid — marker's body text + MinerU's figure / table layout spliced by figure/table number
• --auto-layout-source — per-paper detector routes scans to marker, born-digital to mineru

Vision-LLM post-passes

• Hook 0: citation synthesis from page 1 + DOI
• Hook 1: per-table VLM rewrite + per-table sidecar .md
• Hook 1.5: orphan-caption table rescue
• Hook 2: author-caption matching with freeform fallback + dup-detect post-pass
• Hook 3: sparse-page rescue (opt-in)

Reproducibility

• Full run: block in YAML frontmatter: paper2md version, Zenodo DOI, model identifier, system prompt, sampling parameters, package versions, compute backend, hostname, full CLI invocation
• AI / VLM disclosure section in USAGE.md §18 with template paragraph for publications

Semantic asset naming

• assets/table_{id}_p{page}_{idx}.{md,jpg} — sidecar .md + matching .jpg share stems
• assets/figure_{id}{letter?}_p{page}.{ext} — multi-panel figures get the letter suffix
• Dots in ids (A.4) become _ in filenames; link text keeps the original .

Quality scoring + manual edits

• A–F grade + per-table / per-figure quality sub-scores
• --vlm-tables-force for publication-grade sidecars on individual papers
• Standalone vlm-table CLI for one-off image → markdown / CSV transcription
• --replace-table / --replace-fig / --revert-edit for manual fixes from a user-provided crop
• --recover-from-mineru / --confirm-recovery for audit-flagged tables MinerU detected but the hybrid splice dropped

Reference rescue

• Deterministic body cleanup passes (line-number strip, footnote consolidation, section merge, in-section tidy)
• Crossref → OpenAlex API fallback as the canonical solution for hard ref-section cases (any paper with a DOI)
• Per-journal rescues (APS last-ref-bleed, page-boundary mash, missing numbered headings) behind --use-journal-rescue

Metadata / copyright frontend

• 6-API resolution: OpenAlex, Unpaywall, Europe PMC, OSTI / DOE PAGES, arXiv, Crossref
• safe_to_distribute classification in copyright: YAML
• Optional swap to OA copy via --prefer-oa-source

Batch mode

• --batch over folder / glob; supplement auto-pairing via _SI regex; per-paper subdirs
• manifest.jsonl aggregates one line per paper
• Configurable parallelism via --workers; --paper-timeout per-paper safety net
• Failure isolation; resume on re-run (default behavior)

HDF5 bundling (--hdf5) — self-contained .h5 with main markdown + supplement + every asset, gzip-compressed for text, raw bytes for images.

Mac fixes shipped in this release

• MineruNotInstalledError preflight with install hint when the mineru CLI is missing
• KMP_DUPLICATE_LIB_OK=TRUE auto-set on Darwin to avoid the PyTorch / PaddlePaddle libomp duplicate-load abort

MinerU 3.1.7 pinned, with runtime version check warning on drift.

Install

Apple Silicon:

conda env create -f environment-mac.yml && conda activate paper2md  
pip install -e . --no-deps
pip install "mineru[core]==3.1.7"   # if using --layout-source mineru/hybrid 
# Start LM Studio with Qwen3-VL-32B-Instruct-MLX (8-bit recommended)
python src/paper2md.py paper.pdf -o outputdir

NVIDIA / CUDA:

conda env create -f environment-gpu.yml && conda activate paper2md
pip install -e . --no-deps
pip install "mineru[core]==3.1.7"
vllm serve Qwen/Qwen3-VL-32B-Instruct --port 8000 \
--max-model-len 32768 --gpu-memory-utilization 0.65 &
python src/paper2md.py paper.pdf -o outputdir

See docs/USAGE.md for the full CLI reference, hook-by-hook behavior, and troubleshooting.

Tests

python -m pytest tests/ -q # 800 tests, no GPU/network required

Citation

Stewart, S. T., & Claude (Anthropic, Opus 4.7). (2026). paper2md (v0.4.0) [Software]. MIT License. https://doi.org/10.5281/zenodo.20262917

License

MIT — see LICENSE.

Developed by Sarah T. Stewart (Arizona State University) with Claude Code (Anthropic, Opus 4.7).