High-Level Design of the Data Fabric - Interoperability framework for data and models

This Deliverable presents the high-level design of the Data Fabric to be prototypically implemented for the four Real World Labs (RWLs) in DIRECTED, in order to showcase the technical advancements supporting Disaster Risk Management (DRM) and Climate Change Adaptation (CCA). This will be executed through improved interoperability, data standards and governance. The findings are predominately based on work performed in Task 5.1 - Stocktaking of the Current Data Context and Task 5.2 - Define the project interoperability Use Cases. Hence, this deliverable presents for each RWL i) its expected data contributions and ii) its data and information needs. Furthermore, the interactions between data and among models within the Data Fabric are presented highlighting the interoperability aspects per RWL use-case. The complete detailed architecture and implementation of the Data Fabric is beyond the scope of this document and still to be developed and detailed in D5.2.

Based on interviews with the RWLs, review of previous work, and workshops with the RWL hosts and participants within the co-design approach of DIRECTED, the current state of data usage, information products and software solutions has been assessed. During this process, WP 5’s focus has been on technical gaps and barriers where the DIRECTED Data Fabric could provide a valuable contribution addressing stakeholder’s real needs. The specific statuses of each RWL are rather heterogeneous due to different regional and national data providers and different compositions of stakeholders and hosts in each RWL. However, common data needs and sources, such as the Copernicus Climate Data Store, emerged from the interviews and workshops. These general base data will be detailed in Section 2.4, as well as ethical aspects of data and data providers in Section 2.5. The diversity of data providers and users of the Data Fabric is a challenge, but also an opportunity to develop a high-level architectural definition for the Data Fabric implementation required to support DRM and CCA applicable in diverse contexts.

In order to support re-usability and applicability of the software components constituting the Data Fabric, the foundation of these components is open-source, and additions as well as new implementations will have the ability to be published as open-source components that can be deployed and extended free of any charge. The Data Fabric will be based on open standards (e.g. OGC) that are widely accepted and frequently used for geospatial data. The envisioned solution will entail web-based front-end components for discovery, access and publication of data and models, but will also provide endpoints that allow to integrate the new information products into existing solutions (e.g., QGIS) already established in the RWLs. The development of the Data Fabric will be carried out iteratively in close cooperation with the RWL hosts and participants, ensuring that the real needs of stakeholders are best met. It should be noted, however, that the Data Fabric will also come with limitations. While building upon open standards for data and model integration, it cannot be guaranteed that “any” data set can be seamlessly integrated nor that any model chain can be build ensuring meaningful results. The development will be bounded by the priority use cases of the RWLs favouring reuse-, extend- and interoperability where possible and affordable within the project’s scope.

The Data Fabric will provide means to document the provenance of information products generated and consumed. This metadata will support the reproducibility and auditability of the information product supporting decision taking. Furthermore, access to specific data and visualizations will be given to the Data Steward, a person in each RWL responsible for permissions (Section 2.2). This will enable end-users to assess input data sources regarding their representativeness, such as with climate indicators. These measures are meant to prevent misuse and misinterpretation of information products generated through the Data Fabric.