Project deliverable Open Access
This accompanying document for deliverable D3.2 Data Ingestion & Integration Components describes the mechanisms and tools that will be used in the BigDataGrapes platform to ingest data of different nature from multiple sources. Also, the document describes the tools that will be used for data integration across the different BigDataGrapes platform layers, as well as for long-term storage and preservation of data.
The document first introduces the big picture of the architecture of the BDG platform and where the ingestion components are positioned. Afterwards, the document describes the different nature of data, and which technologies can be used to facilitate the ingestion process.
Then, the data fusion aspect is described, focusing on how the data will be made available across the BigDataGrapes platform, and how the different BigDataGrapes components will communicate with each other, in an effective and fault-tolerant way.
Moreover, the document provides documentation links for all the described technologies along with links to tools that facilitate their setup & maintenance. Finally, the document includes links that point to the dockerized versions of the respective tools, as provided by the BigDataEurope (BDE, https://www.big-data-europe.eu/) Project, which is starting point regarding the technical solutions that BigDataGrapes will built upon (as it has ben described in details in the BigDataGrapes DoA).
D3.2 - Data Ingestion & Integration Components.pdf