Published November 6, 2024 | Version v1
Project deliverable Restricted

OPUS Deliverable 3.3 Intermediate Evaluation of Metrics Tested in the Pilots

  • 1. JISC

Description

The OPUS project is developing coordination and support measures for organisations to reform research assessment with a focus on recognising and rewarding researchers for Open Science. The main aim of the project is to develop a framework for researcher assessment, consisting of indicators/ metrics with a focus on Open Science and corresponding interventions for organisations to implement the framework, which will be tested in pilots and refined in consultation with the research community.

This report is deliverable D3.3 of the OPUS project Intermediate Evaluation of Metrics Tested in the Pilots. The report builds on D3.1 on Indicators and Metrics to Test in the Pilots, which proposed a first draft of a researcher assessment framework (RAF) to assess researchers and recognise and reward Open Science in researcher assessment. The report also builds on D3.2 of the OPUS project on Baseline Audit of Metrics to Test in the Pilots which presented the indicators/metrics selected by each pilot organisation in the OPUS project and their corresponding baseline and targets for the pilots.

This report is closely aligned with D2.3 Intermediate Evaluation of Interventions Tested in the Pilots which outlines progress towards meeting targets set on interventions which should be read in before this deliverable. That in turn builds on D2.1 on Interventions to Test in the Pilots which proposed a first draft series of interventions to support implementation of a researcher assessment framework (RAF) at research-performing organisations (RPOs) and research-funding organisations (RFOs) and D2.2 on Baseline Audit of Interventions to Test in the Pilots.

The report has been developed in collaboration with the organisations running pilots in the OPUS project as listed in Table 1.1. The three RPOs (UNL, UCY, and UNIRI) and two RFOs (UEFISCDI and RCL) have selected and tailored indicators/metrics from D3.1 and corresponding interventions from D2.1 to test in each of their pilots. The pilot period in the OPUS project runs for 18 months (M) from January 2024 until June 2025. Pilot progress is being monitored and this report represents the interim evaluation covering M0-M9. A final evaluation will take place at M18 of the pilots.

 

Pilot # Organisation Acronym Country
Pilot #1 Nova University of Lisbon  UNL Portugal
Pilot #2 University of Cyprus  UCY Cyprus
Pilot #3 University of Rijeka  UNIRI Croatia
Pilot #4 Executive Agency for Higher Education, Research, Development, and Innovation Funding  UEFISCDI Romania
Pilot #5 Research Council of Lithuania  RCL Lithuania

Table 1.1: Pilot Organisations in the OPUS Project

The report structure is based on the pilots and begins in Section 2 with a description of the methodology used to carry out the co-monitoring with pilots to track progress towards meeting targets for interventions and indicators/metrics. The report next presents each pilot, together with their baseline and targets for their selected indicators/metrics at M9 and M18, and actual progress at M9, followed by a narrative progress update covering M0-9 (UNL in Section 3, UCY in Section 4, UNIRI in in Section 5, UEFISCDI in Section 6, and RCL in Section 7). Lastly, the report offers some conclusions and proposes next steps for the OPUS project in Section 8.

It should be noted that the report does not go into detail on the implementation of selected interventions and indicators/metrics in the pilots as this will be presented in action plans describing the pilots by each of the pilot organisations. Full details can be found in the deliverable D4.1: Action Plans to Implement the Pilots - Initial and deliverable D4.3 Action Plans to Implement the Pilots – Final.

Files

Restricted

The record is publicly accessible, but files are restricted to users with access.

Additional details

Funding

European Commission
OPUS – Open Universal Science 101058471
UK Research and Innovation
OPUS 10075391
UK Research and Innovation
OPUS - Open Universal Science 10054677

References

  • Webpage of the Open Universal Science (OPUS) project on CORDIS. Link: [https://cordis.europa.eu/project/id/101058471].
  • O'Neill, Gareth (2023) Indicators and Metrics to Test in the Pilots. Deliverable D3.1 of the Open Universal Science (OPUS) project.
  • O'Neill, Gareth, Emma Day, and Fleur Lebhardt (2023) Baseline Audit of Indicators and Metrics to Test in the Pilot. Deliverable D3.2 of the Open Universal Science (OPUS) project.
  • Clare, Helen (2024) Intermediate Evaluation of Interventions Tested in the Pilots. D2.1 of the Open Universal Science (OPUS) project.
  • Day, Emma (2023) Interventions to Test in the Pilots. Deliverable D2.1 of the Open Universal Science (OPUS) project.
  • O'Neill, Gareth, Emma Day, and Fleur Lebhardt (2023) Baseline Audit of Interventions to Test in the Pilots. Deliverable D2.2 of the Open Universal Science (OPUS) project.
  • Website of Nova University of Lisbon (UNL). Link: [https://www.unl.pt].
  • Website of University of Cyprus (UCY). Link: [https://www.ucy.ac.cy].
  • Website of University of Rijeka (UNIRI). Link: [https://uniri.hr].
  • Website of Executive Agency for Higher Education, Research, Development, and Innovation Funding (UEFISCDI). Link: [https://uefiscdi.gov.ro].
  • Website of Research Council of Lithuania (RCL). Link: [https://www.lmt.lt].
  • Venda, Ana, Andreia Zorrinho, and Raquel Vega (2023) Action Plans to Implement the Pilots – Initial. Deliverable D4.1 of the Open Universal Science (OPUS) project
  • Venda, Ana, Andreia Zorrinho, and Raquel Vega (2024) Action Plans to Implement the Pilots – Final . Deliverable D4.3 of the Open Universal Science (OPUS) project.
  • European Commission (2023) Proposal for a Council Recommendation on a European Framework to Attract and Retain Research, Innovation, and Entrepreneurial Talents in Europe. Link: [https://eurlex.europa.eu/legal-content/EN/TXT/?uri=COM:2023:0436:FIN].
  • Website of Global Health and Tropical Medicine Research Centre (GHTM). Link: [https://ghtm.ihmt.unl.pt].
  • Website of Information Management Research Centre (MagIC). Link: [https://magic.novaims.unl.pt].
  • Website of Foundation for Science and Technology (FCT). Link: [https://www.fct.pt]. Accessed 31 October 2023.
  • Website of KIOS Research and Innovation Centre of Excellence (KIOS CoE). Link: [https://www.kios.ucy.ac.cy].
  • Website of University of Rijeka Faculty of Law (PRAVRI). Link: [https://www.pravri.uniri.hr].
  • Webpage of National Research, Development, and Innovation Funding Programme on 'Centres of Excellence' (UEFISCDI CoE). Link: [https://uefiscdi.gov.ro/centre-de-excelenta-coe].
  • Webpage of National Funding Programme on 'Independent Scientific Research Projects by Groups of Scientists'. Link: [https://www.lmt.lt/lt/mokslo-finansavimas/kalendorinis-kvietimu-planas/2287/kvietimas-701].