Evaluating Graph-RAG Systems for Historical Transport Networks: A Berlin Case Study

This research project develops and evaluates specialised Retrieval-Augmented Generation (RAG) approaches for historical knowledge graphs, using Berlin's public transportation system during the Cold War era (1945-1989) as a case study

While traditional RAG systems excel at processing unstructured text, they often struggle with the complex, highly structured temporal data found in historical databases. This study addresses that gap by establishing a comprehensive evaluation framework for "Graph-RAG" systems in the digital humanities.

Key Findings The research demonstrates that no single pipeline excels at all historical query types. While NL-to-Cypher approaches dominated factual retrieval, they struggled with interpretive synthesis. The study found that a multi-pipeline architecture, which intelligently routes user queries to the most appropriate retrieval strategy, achieved a marked improvement in answer quality compared to single-pipeline approaches. Additionally, the findings highlight significant limitations in applying standard community detection algorithms (like Leiden) to sparse, linear infrastructure networks.

Files in this Record:

• Abschlussbericht.pdf: The complete final research report (13 pages) detailing methodology, pipeline architecture, evaluation framework, and findings.
• question_design_methodology.md: A comprehensive guide to the user-centered evaluation framework, including theoretical grounding in digital humanities tool adoption.
• rubric.md: The detailed scoring criteria (0-3 scale) used for evaluating historical accuracy, context retention, and explanatory capability across 33 questions.
• berlin_transport_questions.csv: The complete taxonomy of 33 evaluation questions spanning 6 user personas, 5 difficulty levels, and 5 dimensional categories.
• user_personas.md: Six detailed user archetypes (from neighborhood historians to data journalists) representing the target audience for the public-facing system.

Related Resources

• Live Demo: https://berlin-transport-history.de/chat

• Code Repository: https://scm.cms.hu-berlin.de/baumanoa/graph-rag

Funding Acknowledgement Dieses Projekt wurde durch das BMFTR-Datenkompetenzzentrum HERMES gefördert.