Evaluating the quality and reliability of open bibliometric data for country-wide research assessment purposes in Switzerland - insights from project TOBI

Despite strong international relevance of research done at Swiss Higher Educational Institutions (HEIs), scientometric services in Switzerland remain relatively underused in the areas of research assessment and research management. This is striking especially in comparison with countries like Germany or the Netherlands. Each of those countries hosts a competence center for bibliometrics with country-wide relevance and has some system in place to track the national scholarly metadata in a standardized way. There is no such centralized body in Switzerland and the issue is further complicated by the distributed nature of governance in the country. Consequently, the institutions need to decide by themselves how and how much they want to use bibliometrics, which often turns out to be too big of an investment and leads to a missed opportunity. However, the current lack of existing country-wide structures also presents a chance to design and introduce new solutions for quantitative research assessment in a way that would reflect the values put forward by DORA and CoARA and would be adapted to the local requirements.

The ETH Library of ETH Zurich has been active in the domain of scientometrics since 2020 and has been promoting the responsible use of scientometrics in research assessment. It is currently leading two projects in this topic, co-funded by "Swissuniversities", a federally funded association of Swiss HEIs. In this presentation, we would like to share insights from one of them: "Towards Open Bibliometric Indicators" (TOBI), which started in March 2023.

The anchor of the TOBI project are the DORA Recommendations #11-#14, which call for transparency and openness in data sources and methods used by organizations that supply metrics. We strongly believe that it means basing research assessment on open data, as only open data allows for a full reproducibility of calculations and curation. However, as long as the quality of the data remains unclear, biases may uncontrollably distort the calculated indicators and, in the worst case, render the results unreliable. If such biases are unknown to the analyst or decision-maker, distorted metric values may be misinterpreted and could lead to wrong decisions. Commercial, closed datasets--while suffering from the problems such as black-box curation, prohibitive licenses or being owned by big publishers--have the benefit that they have been widely used and scrutinized, so that the limitations of standard indicators are known to the scientometric community.

TOBI aims to undertake a detailed evaluation of quality of the scholarly metadata contained in various open bibliometric data sources, in a subset related to the research conducted at Swiss HEIs. The respective subsets are being analysed in terms of available metadata, data completeness, disambiguation quality, coherence, biases and identification of potential gaps, considering different facets (publication types, research fields, etc.) whenever possible. The outcomes of project TOBI will be shared as reports, open source code and datasets, empowering the Swiss HEIs to become more active in the domain. In the presentation we will share the interim insights from the project and discuss challenges we have encountered.