Interoperability of EO cloud computing services

This document represents all presentations, held during ESA Phiweek at ESRIN (Frascati / IT) at the side event "Interoperability of EO Cloud Computing Services With the openEO API".

Following side event's description was published by the Phiweek's organizers.

Planned outcome of openEO

Copernicus and other novel Earth Observation (EO) programmes are generating data of unprecedented quality and volume. To scope with this, Petabyte-scale EO data centres and cloud computing services have been set up over the last decade, resulting in a variety of customised EO processing platforms. Nowadays, available services are ranging from Data-as-a-Service (DaaS) to Platform-as-a-Service (PaaS), are tailored by their user's needs and are often closed-source up to different degrees (e.g. DIAS, VITO, EODC, Copernicus Global Land Service, Google Earth Engine, Amazon Web Services, …). This heterogeneity, the diverse user demands as well as the emergence of the various service offerings makes it currently difficult for end users to compare results from different platforms, or costs of offerings.

openEO is a user-driven open source project that develops an API (a language) for communication between users of diverse programming environments (e.g. web browser, Jupyter notebooks, RStudio) and various EO service providers. It aims at cross-cloud platform interoperability (i) to merge heterogeneous user communities of different platforms, now developing combinable workflows and thus to enable more holistic and specialised cloud computing approaches in the EO sector, (ii) to allow users an easier switch between service providers, (iii) to implement cross-platform communication strategies and thus to develop de-centralised workflows, considering the capabilities of individual service providers, and (iv) to allow the comparison of the platforms processing results, capabilities and pricing. For the communication between clients and EO cloud providers, the workflows' commands are chained to standardised process graphs in a JSON format and transferred via web request to an interface at the service provider. The jobs are then translated by the openEO back-end to the platform's local syntax and are either executed as batch jobs, or lazy evaluated as part of secondary web services for web-based access or executed synchronously in case of lightweight jobs. With openEO, former needed many-to-many connections between clients and the cloud providers are reduced to many-to-one connections.

The openEO API entails processes from all aspects of the EO data life cycle. It prepares the data as a 'virtual data cube' – independently of the back-ends storage data structure. The openEO processes support following data manipulation: (i) EO data can be subsetted, (ii) dimension can be removed or added by computation, (iii) resampling and aggregation processes allow e.g. reprojecting or rescaling of EO data, (iv) pixel-based math processes are available (e.g. sorting algorithms, unary functions), and (v) the capability of processing user-defined functions (UDFs) will allow openEO to meet extremely specialised user demands. All currently available processes are listed and described at https://processes.openeo.org.

While openEO is designed as language neutral, clients for Python, R, and JavaScript are currently developed. Compatible service providers are momentarily VITO, EODC, mundialis, Sinergise, EURAC Research, JRC and Google Earth Engine; further connections are planned. The openEO API is conceived to be used as a template for further service providers to easily connect to the freely available API (https://github.com/Open-EO/). A stable version was released in July 2019, which will be demonstrated in this workshop.

Aim of the workshop

This workshop will provide participants with an overview of using openEO on different cloud platforms. Service providers will gain insights in implementing the open source API on their system and thus connecting to the joint user community.

The side event is split into 2 sessions of 2 hours each, showing live demonstrations and covering various aspects of the openEO API. The first session will concentrate on the clients perspective, demonstrating its use via Python, R, and web interfaces on various service providers (VITO, Mundialis, EURAC Research, Google Earth Engine). The second session will deal with the installation of openEO instances on service provider’s premises and chosen standards of the openEO API.