iHelp: Standardisation and Quality Assurance of Heterogenous Data II

This deliverable (titled “Standardisation and Quality Assurance of Heterogenous Data II”) describes the initial implementations and first prototypes of the iHelp Standardisation and Quality Assurance Mechanism. As introduced in D3.7 – “Standardisation and Quality Assurance of Heterogenous Data I”, the Standardisation and Quality Assurance Mechanism is a unified and integrated mechanism consisting of three (3) core sub-components, the Data Cleaner, the Data Qualifier, the Data Harmonizer, and two (2) integrated sub-components: the Primary Data Mapper and the Secondary Data Mapper, which are responsible for providing the mapping operations between the raw data resources and the Holistic Health Records (HHRs) resources. This holistic mechanism is the core component that seeks to provide various cleaning, pre-processing, harmonization, and mapping functionalities and services on the incoming raw data. Specifically, it provides to the wider research and healthcare community a wide range of innovative solutions for the cleaning, qualification, transformation, harmonization, and mapping of raw healthcare-related data.
The current document builds upon the initial design and specifications of D3.7 – “Standardisation and Quality Assurance of Heterogenous Data I”, aiming to provide the initial implementation, the first prototypes and a concrete overview of how the proposed mechanism integrates with the overall architecture of the iHelp platform and other components in the Data Ingestion pipeline, and specifically (i) how to retrieve the incoming data from the Data Gateways; (ii) how to interexchange data with the already identified project’s message bus; and (iii) how to send the final processed, transformed and HHR aligned data to iHelp’s Big Data Platform.
To support the aforementioned functionalities, the iHelp Standardisation and Quality Assurance Mechanism specifications exemplify the respective sub-components, the overall data pipeline and workflow, the internal functionalities supported by each sub-component, the interaction points with different components as well as the technical details that drive the implementation and deployment of this holistic mechanism. Finally, this document - in different subsections, e.g., in Section 5.2, - seeks to review the current state of the art in order to identify the baseline technologies and approaches for the realization of the implemented Standardisation and Quality Assurance Mechanism.