Published October 22, 2024 | Version v1
Conference paper Open

Evolution of Extract-Transform-Load (ETL) processes towards data product pipelines

  • 1. ROR icon Tecnalia
  • 2. ROR icon Centro Tecnológico de Investigación, Desarrollo e Innovación en tecnologías de la Información y las Comunicaciones (TIC)

Description

The rise of data as a first-class asset has led to creating infrastructures and tools designed to enhance organizations’ abilities to monetize them internally. One of the most powerful tools have been ETLs, which govern the internal data operations, assisting in companies in their quest to becoming data-driven. Lately, these horizons have expanded with the apparition of new ecosystems for data exchange, such as Data Spaces or initiatives like Gaia-X or SIMPL, allowing companies to monetize data externally, e.g. sharing or selling them. However, traditional ETLs fall short to serve this purpose. In this article, we try to offer a technological comparison of how current ETL tools are prepared to address the new concept of data pipeline aimed at achieving a data product. Furthermore, this comparison is proposed within the framework of a project like DATAMITE, which allows it to be provided with real scenarios and use cases, in which its benefits and applicability can be accurately appreciated.

Files

3685651.3686662.pdf

Files (1.5 MB)

Name Size Download all
md5:82a10ac2aa4daf79e55849f70ea9c24c
1.5 MB Preview Download

Additional details

Funding

European Commission
DATA Monetization, Interoperability, Trading & Exchange Grant agreement ID: 101092989