POPP Datasets : Datasets for handwriting recognition from French population census

POPP datasets

This repository contains 3 datasets created within the POPP project (Project for the Oceration of the Paris Population Census) for the task of handwriting text recognition. These datasets have been published in Recognition and information extraction in historical handwritten tables: toward understanding early 20th century Paris census at DAS 2022.

The 3 datasets are called “Generic dataset”, “Belleville”, and “Chaussée d’Antin” and contains lines made from the extracted rows of census tables from 1926. Each table in the Paris census contains 30 rows, thus each page in these datasets corresponds to 30 lines.

The structure of each dataset is the following:

• double-pages : images of the double pages
• pages:
  • images: images of the pages
  • xml: METS and ALTO files of each page containing the coordinates of the bounding boxes of each line
• lines: contains the labels in the file labels.json and the line images splitted into the folders train, valid and test. The double pages were scanned at a resolution of 200dpi and saved as PNG images with 256 gray levels. The line and page images are shared in the TIFF format, also with 256 gray levels.

Since the lines are extracted from table rows, we defined 4 special characters to describe the structure of the text:

• ¤ : indicates an empty cell
• / : indicates the separation into columns
• ? : indicates that the content of the cell following this symbol is written above the regular baseline
• ! : indicates that the content of the cell following this symbol is written below the regular baseline

We provide a script format_dataset.py to define which special character you want to use in the ground-truth.

The split for the Generic Dataset and Belleville have been made at the double-page level so that each writer only appears in one subset among train, evaluation and test. The following table summarizes the splits and the number of writers for each dataset:

Generic dataset (or POPP dataset)

• This dataset is made 4800 annotated lines extracted from 80 double pages of the 1926 Paris census.
• There is one double page for each of the 80 districts of Paris
• There is one writer per double page so the dataset contains 80 different writers.

Belleville dataset

This dataset is a mono-writer dataset made of 1470 lines (49 pages) from the Belleville district census of 1926.

Chaussée d’Antin dataset

This dataset is a multi-writer dataset made of 780 lines (26 pages) from the Chaussée d’Antin district census of 1926 and written by 10 different writers.

Error reporting

It is possible that errors persist in the ground truth, so any suggestions for correction are welcome. To do so, please make a merge request on the Github repository and include the correction in both the labels.json file and in the XML file concerned.

Citation Request

If you publish material based on this database, we request you to include a reference to paper T. Constum, N. Kempf, T. Paquet, P. Tranouez, C. Chatelain, S. Brée, and F. Merveille,Recognition and information extraction in historical handwritten tables: toward understanding early 20th century Paris census ,Document Analysis Systems (DAS), pp. 143- 157, La Rochelle, 2022.