Demonstrator of the data analysis and data sharing on HEAP. Public deliverable 6.3

This document describes the upgrades and improvements of the Knowledge Engine within the
Human Exposome Assessment Platform (HEAP). The HEAP software platform is built on top of
the open-source Hopsworks platform. The document describes the overall upgrades and
improvements to the Knowledge Engine in Hopsworks (HopsFS mount, Git support, Airflow
upgrade) and the updates on implemented partner pipelines into the HEAP platform.

The goal is for the pipelines to fully utilise the benefits of horizontal scaling and sharing of data and code as
some of the base qualities of the platform. The implementation presented in this document is
coordinated with work package 3 - Large sample cohorts from population-based interventions,
work package 4 - Consumer exposure monitoring system, work package 8 - Epigenomics
Analysis, and work package 9 - Metagenomic analysis.

This deliverable describes the implemented Data Analysis and Machine Learning Pipelines using
the Data Analytics and Machine Learning (ML) capabilities of the Knowledge Engine. This
document describes the technical architecture, data flow, data analysis, and associated
metadata of the HEAP pipelines. The document also presents the final updates in features and
improvements to the Hopsworks platform as part of the Heap platform.