OPUS Deliverable 3.1: Indicators and Metrics to Test in the Pilots

This report is deliverable D3.1 of the OPUS project on Indicators and Metrics to Test in the Pilots. The report proposes a first draft of a researcher assessment framework (RAF) to assess researchers in an academic context. This includes assessing researchers applying for positions at a research-performing organisation (RPO), assessing researchers in their career development and progression at an RPO, and assessing project applications and progression by researchers at an RPO or research-funding organisation (RFO). The framework includes an Open Science dimension, whereby Open Science practices are explicitly recognised and rewarded. The implementation of the framework is supported by accompanying interventions for RPOs and RFOs, which are described in deliverable D2.1 of OPUS on Interventions to Test in the Pilots. Selected indicators/metrics from the framework and related interventions will be tested in pilots at 3 RPOs and 2 RFOs in OPUS.

The RAF builds on key policy developments in research assessment and Open Science as identified in deliverable D1.2 of OPUS on State-of-the-Art on an Open Science Ecosystem:

● San Francisco Declaration on Research Assessment (DORA)

● Leiden Manifesto for Research Metrics

● Hong Kong Principles

● Recommendations by the Open Science Policy Platform (OSSP)

● Recommendations on Science and Scientific Researchers and Open Science

● Agreement on Reforming Research Assessment

● Research Evaluation in a Context of Open Science and Gender Equality

● Conclusions on Research Assessment and Implementation of Open Science

● European Framework for Research Careers including European Charter for Researchers

The RAF also builds on key frameworks in research assessment and Open Science with a focus on developing new indicators/metrics and supporting Open Science as identified in D1.2:

● Researcher Development Framework (RDF)

● Evaluation of Research Careers Fully Acknowledging Open Science Practices

● Next-generation Metrics

● Recommendations of the OSPP on Next-Generation Metrics

● Mutual Learning Exercise on Open Science on Altmetrics and Rewards

● Open Science Monitor

● Indicator Frameworks for Fostering Open Knowledge Practices in Science and Scholarship

● A Pathway towards Multidimensional Academic Careers

The RAF further builds on key recommendations for a framework for researcher assessment from D1.2 which incorporates the recognition and reward of Open Science practices by researchers:

● Develop a comprehensive RAF of indicators/metrics for RPOs and RFOs

● Include both research and non-research activities by researchers in the RAF

● Include both a generic and Open Science approach to assessment in the RAF

● Include both a quantitative and qualitative approach to assessment in the RAF

● Integrate relevant indicators/metrics from existing frameworks into the RAF

The RAF has been developed in close collaboration with key stakeholders in research assessment and Open Science. This includes project partners and especially the pilot organisations1 which are already implementing researcher assessment and Open Science and will pilot the RAF in OPUS. This also includes members of the Advisory Board2 in OPUS and key external stakeholders3 as identified in D1.1 of OPUS on State-of-the-Art on an Ecosystem for Open Science. This lastly includes the GraspOS project and other relevant stakeholders4. This first draft of the RAF will be further developed in iterations, including testing and feedback from the pilots, targeted feedback from key stakeholders, and an open consultation with the wider research community.

The RAF consists of a framework to reform researcher assessment and incentivise and reward Open Science linked to the full spectrum of activities carried out by researchers. The RAF essentially offers a wide range of indicators/metrics for RPOs and RFOs to develop new or update existing researcher assessment frameworks. The RAF is divided into 4 main categories for research, education, leadership, and valorisation activities, which are further subdivided into relevant subcategories consisting of specific groups of indicators/metrics. The RAF is furthermore divided into generic and open dimensions, whereby the indicators/metrics are either related to activities not specified in terms of openness or related to activities focused specifically on Open Science. The framework is intended to be universally applicable across countries, across disciplines, and across organisations, whereby RPOs and RFOs can tailor the framework to their own interests and needs.

The report begins with the guiding principles behind the RAF and describes the overall structure and implementation of the RAF at RPOs and RFOs in Section 2. The report next presents the generic RAF and lists the generic indicators/metrics for researcher assessment at RPOs and RFOs in Section 3. The report then presents the Open Science RAF and lists the indicators/metrics to recognise and reward Open Science practices in researcher assessment at RPOs and RFOs in Section 4. The report lastly offers conclusions and next steps for OPUS in Section 5 and provides the full RAF with all generic and Open Science indicators/metrics in table format in Appendix 1.