Published November 10, 2026 | Version 2.1.0

OCR Pipeline for Historical Print Periodicals using Mistral AI

  • 1. Research Library for the History of Education

Description

A Jupyter-based batch OCR pipeline for processing scanned documents (PDF) and images (PNG, JPEG) using the Mistral OCR API, pinned to OCR 4 (mistral-ocr-4-1) for corpus processing, with a Tesseract cross-check that detects hallucinated text in the OCR output. Designed for (historical) print periodicals; handles printed text, forms, tables, and (with limitations) handwritten content (modern scripts [historical German scripts like Kurrent remain challenging - a separate pipeline is planned]).

New in 2.1.0

* Model updated to mistral-ocr-4-1 (mistral-ocr-4-0 has been retired by Mistral)
* Fallback for retired models: if the pinned version is no longer available, the pipeline lists the OCR models currently offered by the API and lets you choose a replacement - the model is never switched silently
* Tesseract cross-check in seperate notebook: every page is read a second time with Tesseract (locally, no API costs) to detect hallucinations, above all on blank pages, where Mistral OCR tends to invent text (repeated sentences or formulas, Chinese text, statements like "The image contains no text") - often with a confidence score close to 1.0
* Manual review in the notebook: flagged pages are shown with scan, Mistral text and Tesseract reading; Keep/Remove decisions are saved immediately, removed text is kept and can be restored
* Fix: large PDFs that are split into parts are now correctly skipped once processed, and interrupted runs resume with the missing parts

Features

* Structured text extraction (headings, paragraphs, footnotes) in Markdown format
* API confidence scores per page (default) or per word (optional), with per-page details in a JSON sidecar
* Optional layout-block extraction: typed blocks (title, text, caption, table, ...) with bounding boxes
* Pinned model version - a corpus is never processed with silently changing model versions
* Automatic PDF splitting for large documents (>50 MB or >1000 pages)
* SQLite checkpoint system for resumable processing
* Zero Data Retention by default for Mistral OCR
* Multilingual OCR support (Mistral OCR 4 covers 170 languages)

Quick Setup

1. Open the repository in a Jupyter-compatible environment (e.g., VSCode with Jupyter extension)
2. Get a Mistral API key from console.mistral.ai
3. Rename .env.template to .env and add your API key
4. Place your documents (PDF) or images (PNG, JPEG) in data/input/
5. Run notebooks/ocr_pipeline.ipynb
6. Optional: install Tesseract with German language data and run notebooks/tesseract_crosscheck.ipynb to check the results for hallucinations

Output

* Markdown files with preserved document structure
* Plain text files for downstream processing
* JSON metadata (model version, confidence scores, page count) plus optional sidecars for per-page confidence details and layout blocks
* Cross-check: page table, per-document reports, and cleaned copies of the texts with hallucinated pages removed (original output is never modified)

Cost

* Experiment accounts available without cost via console.mistral.ai
* $4 USD per 1000 pages (Mistral OCR 4 API, as of June 2026)
* Tesseract cross-check runs locally without API costs
* Requires mistralai>=2.7.0 (Python SDK v2); the cross-check additionally requires Tesseract, pytesseract and ipywidgets

Files

ocr-mistral-periodicals-v2.1.0.zip

Files (69.2 kB)

Name Size
md5:d4e34f0f6b801143daee26a653bcbfeb
69.2 kB Preview Download

Additional details

Related works

Is supplement to
Software: https://github.com/ma-wi-lo/pubs/tree/1.1.0 (URL)

Software

Repository URL
https://github.com/ma-wi-lo/pubs
Programming language
Python , Jupyter Notebook
Development Status
Active