D2.2 A report on the coordination with WP1 on the incorporation of monitoring statistics and benchmarking results in registry releases

The objective of Deliverable 2.2 is to report the integration of benchmarking activities in the ELIXIR Tools Platform ecosystem. During the execution of the ELIXIR-EXCELERATE project the original tasks of WP2 have evolved to the generation of a integrated platform, OpenEBench, that supersedes the scope of initial Task 2.3 (section 5). For this reason, this first deliverable after the availability of the release of OpenEBench, presented in ELIXIR All Hands meeting, June 2018, contains also a detailed presentation of the management of Tools Monitoring metrics in the platform. Rationale and motivations about OpenEBench as ELIXIR integrated platform is described in section 8.1. OpenEBench aims to become a central hub for information about tools performance, covering three main aspects: i) scientific performance, ii) software quality, and iii) technical performance. Scientific performance is largely based in the work of scientific communities. OpenEBench provides a central data warehouse where such communities can expose their benchmarking activities for experts and non-experts users covering software developers, community participants, researchers, and/or funding bodies. The data model  supporting OpenEBench efforts was outlined in a previous deliverable (D2.1: Creation of a database warehouse infrastructure for storing and organizing data for online performance assessment experiments) and will not be described here. Future developments of OpenEBench will focus in the generation of an automated platform for supporting scientific benchmarking activities driven by communities. This development will be largely related to the evolution of the ELIXIR platform for software deployment (Biocontainers), and the deployment for solutions for workflow specification and execution from ELIXIR’s interoperability and compute platforms. Tools monitoring activity in OpenEBench is the central issue in D2.2. We describe the series of metrics that are being currently implemented (Section 8.2), and show how data can be obtained from the platform (Section 8.3). OpenEBench have developed a comprehensive web portal (https://openebench.bsc.es), the main access point to the platform for end-users. Also, data is accessible via a series of RESTful APIs for programmatic consumption. This is particularly relevant since this the main way other ELIXIR platforms like the Tools Registry bio.tools will access benchmarking data. Finally, a series of HTML widgets, ready to be embedded into other platforms portals are made available. OpenEBench was released in the occasion of ELIXIR All Hands 2018 Conference in Berlin, but it is still in constant evolution (see section 8.6).

The original aim of D2.2 was to document the relationships with ELIXIR-EXCELERATE WP1, in particular the bio.tools registry. This close relationship has indeed happened, and now bio.tools shows OpenEBench widget in its tool cards, and all tools registered in bio.tools are being monitored in OpenEBench, the information completed with data from other sources, and made available for reuse by bio.tools. However, the scope has now been extended (section 8.5) to a general engagement of benchmarking activities with other ELIXIR platforms and activities. In particular, we gather data from other software related activities, in particular from Biocontainers/Bioconda and Galaxy initiatives. We collect and expose information following the bioschemas standards promoted by the ELIXIR Interoperability platform, and we are in close contact with the compute platform to build the benchmarking execution services according to their standards. Following the ELIXIR Software development best-practice group, OpenEBench is a fully open source project (Section 8.7) from day 1 with an ecosystem of publicly available repositories for the different platform components1.