A transferable infrastructure to enable FAIR timeseries data

The field of earth system sciences relies heavily on the collection, analysis, and interpretation of data to understand complex spatiotemporal environment processes and predict future trends. Time series data, which captures measurements or observations at regular intervals over time, plays a crucial role in elucidating patterns, detecting changes, and informing decision-making in various environmental domains. However, the effective storage and management of time series data present significant challenges that necessitate the development of robust and scalable data infrastructures.

A well-designed data infrastructure is crucial to ensure the reliability, accessibility, interoperability, and sustainability of time series data across different domains and scales. It enables efficient data collection through automated sensing technologies, standardized data exchange protocols, and quality control procedures. Moreover, robust time series data infrastructures facilitate the integration of data from diverse sources into distributed data infrastructures on national and/or continental scales and advances the dissemination of data to a wide range of stakeholders including scientists, policymakers, resource managers, and the general public.

The focus lies on enhancing accessibility for end users, operators, system integrators, and maintainers. The time series management system encompasses all essential components. It features a user-centric web-based frontend, a versatile data integration layer, a robust time series database, efficient object storage, real-time quality control, and comprehensive data visualization capabilities. It supports modern and classical data transfer protocols, and ensures compliance with OGC standards for data access. Moreover, our fully integrated and containerized solution offers the convenience of swift and effortless deployment within minutes and allows the seamless integration with existing services such as databases, identity providers, and object storages.

This innovative infrastructure has the potential to significantly enhance environmental data management, fostering more efficient research and facilitating informed decision-making processes.

Requirements

• Simply FAIR
• Science- and Management friendly: Provide interoperable and reliable timeseries data enriched by metadata and with provenance information.
• User friendly: Easy to use user interface for people that produce or manage timeseries data streams, without requiring knowledge about underlying technologies like databases. Review datastream content in realtime.
• Admin friendly: A scalable and transferable container based solution that will smoothly integrate into typical scientific IT landscapes.
• Developer friendly:Common open source solutions structured by microservice architecture to keep it open and simple to extend for developers.

Features

• File based raw data ingest with common protocols (FTP, SFTP, S3)
• Stream based raw data ingest with MQTT (TLS required)
• HIFIS/Helmholtz AAI Login for end users - all people associated to institutions that are connected to GÉANT Edugain can log in from anywhere
• Grouping data and things by projects that are organized as HIFIS/Helmholtz AAI Virtual Organisation (VO)
• Instant data visualisation of incoming data in Grafana: New Datastreams automatically get a dynamic dashboard with no need but the ability for further configuration and customisation
• Integration layer to link datastreams to sensor metadata in the Sensor Management System (SMS)
• Dynamic OGC STA endpoints with SMS metadata integration per project to access data in a standardized way

Quickstart

1. Install Docker Engine (Community Edition - CE is enough) and Docker Compose
2. Install a git client and checkout the tsm-orchestration repository or unzip the attached archive
3. Change to the tsm-orchestration dirctory and run docker copmose up and give it some seconds to start up
4. Browse http://localhost

Techstack, dependencies and third party open source products

• TimescaleDB
• Django
• Docker CE
• Docker Compose
• Grafana
• Minio
• FROST
• Tomcat
• Mosquitto MQTT Broker
• Alpine
• CAdvisor
• Nginx
• Python Click
• NumPy
• Pandas
• FastAPI
• django-helmholtz-aai

Acknowledgements