ELIXIR-EXCELERATE D2.4: Final report on the impact of ELIXIR infrastructure for hosting scientific benchmarking and technical monitoring results

The objective of Deliverable 2.4 is to report on the current status of OpenEBench as the ELIXIR Benchmarking platform considering all work carried out in the H2020 ELIXIR-EXCELERATE project. This report will constitute a fundamental piece to establish the roadmap for OpenEBench further developments. During EXCELERATE, the platform has become an integral service for hosting both scientific and technical benchmarking data for bioinformatics tools, workflows and web-services. The scientific benchmarking component has focused on existing and newly created community-led efforts as community agreed datasets and metrics act as proxies of the current challenges in a given research area.

This final report builds on previously deliverables e.g. D2.1: Creation of a database warehouse infrastructure for storing and organizing data for online performance assessment experiments1, D2.2: A report on the coordination with WP1 on the incorporation of monitoring statistics and benchmarking results in registry releases2, and D2.3: A report on the features and nature of novel data which are needed within online benchmarking experiments in different subareas3; and updates them in order to review the current infrastructure.

Two important aspects for the infrastructure are understanding the true nature of the data sets used by communities for their scientific benchmarking efforts as well as organizing the interactions with those communities. We have recently released the first stable version (1.0.1) of the OpenEBench scientific data model after extensive testing and adoption based on gathered feedback using real world data. Moreover, we have extensively documented and provide real examples to scientific communities for helping them to get organized as well as to facilitate their incorporation into OpenEBench. We have also consolidated our three-level models to facilitate the incorporation of scientific communities depending on their needs and maturity level. Level 1 and 2 are already implemented and are being used by different communities while level 3 has been a prototype but we depend on other efforts to fully implement it e.g. EOSC-Life.

In the technical monitoring side, we have consolidated the internal data management infrastructure, and extended the available metrics. Besides, we have been working as part of a community effort to translate and re-interpret, when needed, the FAIR data principles to FAIR principles applied to research software. This effort is important to us because it will offer a reference framework to position our set of technical and quality metrics for bioinformatics research software, which include tools, workflows and web-applications.