Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.
Published July 31, 2020 | Version 3.0
Project deliverable Open

BigDataGrapes D3.2 - Data Ingestion & Integration Components

Description

This accompanying document for deliverable D3.2 Data Ingestion & Integration Components describes the mechanisms and tools that will be used in the BigDataGrapes platform to ingest data of different nature from multiple sources. Also, the document describes the tools that will be used for data integration across the different BigDataGrapes platform layers, as well as for long-term storage and preservation of data.

The document first introduces the big picture of the architecture of the BDG platform and where the ingestion components are positioned. Afterwards, the document describes the different nature of data, and which technologies can be used to facilitate the ingestion process.

Then, the data fusion aspect is described, focusing on how the data will be made available across the BigDataGrapes platform, and how the different BigDataGrapes components will communicate with each other, in an effective and fault-tolerant way.

Moreover, the document provides documentation links for all the described technologies along with links to tools that facilitate their setup & maintenance. Finally, the document includes links that point to the dockerized versions of the respective tools, as provided by the BigDataEurope (BDE, https://www.big-data-europe.eu/) Project, which is starting point regarding the technical solutions that BigDataGrapes will built upon (as it has ben described in details in the BigDataGrapes DoA).

Files

D3.2 - Data Ingestion Integration Components_v3_(Submitted to EC).pdf

Files (1.0 MB)

Additional details

Funding

BigDataGrapes – Big Data to Enable Global Disruption of the Grapevine-powered Industries 780751
European Commission