OPUS Deliverable 2.3 Intermediate Evaluation of Interventions Tested in the Pilots

The OPUS project is developing coordination and support measures for organisations to reform research assessment, with a focus on recognising and rewarding researchers for Open Science. The main aim of the project is to develop a framework for researcher assessment, consisting of indicators/ metrics with a focus on Open Science and corresponding interventions for organisations to implement the framework. These are being tested in pilots and refined in consultation with the research community.

This report is deliverable D2.3 Intermediate Evaluation of Interventions Tested in the Pilot. The report builds on D2.1 Interventions to Test in the Pilots, which proposed a first draft series of interventions to support implementation of a researcher assessment framework (RAF) at research-performing organisations (RPOs) and research-funding organisations (RFOs). The report also builds on D2.2 Baseline Audit of Interventions to Test in the Pilots, which presents the interventions selected by each pilot organisation, put into their institutional context along with targets for when they will be implemented at M9 and M18. This report provides an evaluation of progress towards meeting the targets for the interventions outlined in D.2.2.

The report is closely aligned with D3.3 Intermediate Evaluation of Indicators and Metrics Tested in the Pilots which similarly outlines progress towards meeting targets set on indicators and metrics. This in turn builds on D3.1 on Indicators and metrics to test in the Pilots and D3.2 on Baseline Audit of Indicators and Metrics to Test in the Pilots.

The main aim of setting a baseline and targets is to monitor and support the progress of the pilots in implementing selected indicators/metrics and interventions at the pilot organisations. The baseline for the interventions at M0 and targets for M9 and M18 of the pilots were typically based on existing policies and activities, discussions with the pilot management team, and/or discussions with senior management at the pilot organisations. The baselines and targets are expressed in nominal and absolute values, or percentage increases to reflect the progress at the pilot organisations.

The report has been developed in collaboration with the organisations running pilots in the OPUS project (Table 1.1). The three RPOs (UNL, UCY, and UNIRI) and two RFOs (UEFISCDI and RCL) selected and tailored interventions from D2.1 and corresponding indicators from D3.1 to test in each pilot. The pilot period in the OPUS project runs for 18 months (M) from January 2024 (M0) until June 2025 (M18). Pilot progress is being monitored and this report represents the interim evaluation covering M0-M9. A final evaluation will take place at M18 of the pilots.

Table 1.1: Pilot Organisations in the OPUS Project

The report structure is based on the pilots and begins in Section 3 with a description of the methodology used to carry out the co-monitoring with pilots to track progress towards meeting targets for interventions and indicators/metrics. The report next presents a brief summary of each pilot, together with their baseline and targets for their selected interventions at M9 and M18, and actual progress at M9, including a brief narrative progress update covering M0-9 (UNL in Section 3, UCY in Section 4, UNIRI in in Section 5, UEFISCDI in Section 6, and RCL in Section 7). Lastly, the report offers some conclusions and proposes next steps for the OPUS project in Section 8.

It should be noted that the report does not go into detail on the implementation of selected interventions and indicators/metrics in the pilots as this will be presented in action plans describing the pilots by each of the pilot organisations. Full details can be found in the deliverable D4.1: Action Plans to Implement the Pilots - Initial and deliverable D4.3 Action Plans to Implement the Pilots – Final.