Data Infrastructure

This deliverable provides an overview of the data infrastructure setup used to store, manage and process the smart meter data delivered by Radius in the project. The infrastructure is designed to ensure reliable, reproducible, and scalable handling of large volumes of time-series and raw measurement data, forming a foundation for later analytics and data pipeline development. The main components include a Postgres Database for structured, relational data access, object storage for time-series parquet files, and an Airflow-based ingest adapter integrated to support both operational queries and batch processing during prototyping. These components are aligned with the data model described in the data specification document D2.1.

The initial prototyping uses local file storage, while remaining compatible with other data governance platforms such as EDDK and INILAB infrastructure. This design ensures that the data management workflow is traceable and compliant with the requirements of the Danish energy sector. It further ensures that the platform can integrate additional datasets or migrate ingestion sources with ease, given different infrastructure deployments. Future work based on this deliverable includes improvements to partitioning strategies, ingestion capabilities, and long-term data governance measures, which will evolve the platform into a scalable, hybrid architecture that supports extensive analytics and pipeline-based services as the project advances. Most of these results are presented in D4.1 and D4.2.