Chat about SOEP Research Data: A RAG System for Interactive SOEP Data Exploration

The German Socio-Economic Panel (SOEP) represents one of the world's longest-running household panel studies, containing rich longitudinal data spanning over four decades. However, the complexity and scale of SOEP data present significant challenges for researchers in data discovery and exploration. This paper presents a prototype Retrieval-Augmented Generation (RAG) system designed to provide conversational access to SOEP research data through natural language queries. Built on OpenWebUI with PostgreSQL and Qdrant vector databases, and leveraging open-weight LLMs, our system enables researchers to interactively explore dataset descriptions, variable definitions, survey questions, and metadata. While currently covering a substantial portion of SOEP datasets, this prototype demonstrates the potential for conversational AI to enhance research data accessibility and streamline the data discovery process for panel data researchers.

Funded by Deutsche Forschungsgemeinschaft as part of NFDI under project number 442494171.