OCR quality assessment: Beyond ground truth

Over the past years, considerable effort has been put into digitising library collections. As part of such efforts, the automatic extraction of text from images is of critical important for making collections searchable and usable for both quantitative and qualitative analyses. This is the goal of Optical Character Recognition (OCR). OCR is difficult and error prone, especially so when considering historical documents and texts. Challenges arise from the support (e.g., bad preservation or fading ink) to the contents (e.g., language change) of such documents. Therefore, the topic of OCR quality evaluation has been getting increased attention in the literature and is of critical importance to inform libraries in their activities.

Typically, OCR quality is evaluated using “ground truth”: a high-quality, manually verified transcription that can be compared with the OCR-extracted text of the same document. Such evaluation is necessarily done on small samples, given the effort and cost of creating ground truth. This has several limitations: the scale of library collections and their variety make it always difficult to create a representative sample, therefore limiting the reliability of ground-truth-based OCR evaluations. This is why another line of work has focused on OCR quality assessment without ground truth. If successful, these methods promise to reduce the costs of OCR quality assessment substantially, thus making it a more widespread and common practice among libraries. The goal of the ICT with Industry KB challenge is to experiment with several, recently proposed techniques to assess the quality of OCR without ground truth, using a high-quality dataset provided by the KB. The main technique in this group, which will serve as a strong baseline, is lexicon-based evaluation or dictionary lookup. Finally, the challenge also serves to consolidate ongoing work between the KB and the research community on these topics. The open release of the evaluation data and challenge results is planned to this end.