Openness Profile: Modelling research evaluation for open scholarship

Knowledge Exchange (KE) has published the report ‘Openness Profile: Modelling research evaluation for open scholarship

The report describes mechanisms and approaches to improve and incentivize the recording, the evaluation and the recognition of contributions to Open Scholarship practice. The report presents the Openness Profile and how it can help address existing gaps in the assessment of open science.

The Openness Profile is a digital resource, a portfolio of a research contributor's outputs and activities, accessible in a single place. Academic and non-academic open scholarship activities become visible and more easily recognised. The Openness Profile is modelling how research evaluation in an open science context can be improved. Expected benefits are highlighted and requirements listed. Recommendations are provided to various stakeholders how to establish the Openness Profile as research evaluation routine.

Over 80 individual stakeholders from 48 different organisations provided input to this report on research assessment and open scholarship. The work and writing were done by consultants Fiona Murphy and Phill Jones of MoreBrains Cooperative, together with the KE Open Scholarship Research Evaluation task & finish group.

Background

As part of Knowledge Exchange's (KE) work on Open Scholarship, we are exploring the topic of Research(er) Evaluation, with the aim to identify and support recognition of contributions to Open Scholarship, academic and non-academic, that are essential to make Open Scholarship truly work. Our approach is to raise awareness on the lack of recognition in current evaluation practice and work towards a possible solution.

The Openness Profile

The KE Research Evaluation task & finish group worked to improve awareness, listing all academic and non-academic contributions that are essential to Open Scholarship and should be recognised when evaluating research. The group also works on the 'Openness Profile', a tool that is meant to allow evaluation of currently ignored contributions that are essential for Open Science.

By demonstrating the immaturity of current research evaluation practice, and by developing the Openness Profile tool, KE supports researchers as well as non-researchers to get credit for all their contributions that make Open Science possible. Our ambition is that recognition of these essential activities becomes part of standard research evaluation routine.

For the development of the Openness Profile, we have been exploring a concept that would recognise contributions to Open Science by documenting these as digital objects connected to a persistent identifier. We envisage the Openness Profile as a collection of documented contributions with a DOI , linked to the contributor’s ORCID iD.

There is a concern among the Open Science movement that there is a discrepancy between open science policy and the practices of current researchers. Also, there needs to be a system in place for rewarding research contributors for the Open Science activities they do but are invisible to contemporary research evaluation protocols. The Openness Profile would act as an incentivizing and rewarding tool, as well as a means of turning the OS into standard research evaluation routine.

Report ‘Openness Profile: Modelling research evaluation for open scholarship'

The new report describes mechanisms and approaches to improve and incentivize the recording, evaluation, and recognition of contributions to Open Scholarship practice, this report will be of interest to you. As a follow-up on the report ‘Openness Profile: Defining the concepts’ (January 2020, DOI: 10.5281/zenodo.3607579), it presents how the Openness Profile can help address existing gaps in the assessment of open science.

The Openness Profile is a digital resource, a portfolio of a research contributor's outputs and activities, accessible in a single place. Academic and non-academic open scholarship activities become visible and more easily recognised. Relying on persistent identifiers to combine already existing information with additional (including narrative) testimony of achievements, the usage of the Openness Profile could become a standard reference model in open research evaluation routine.

Over 80 individual stakeholders from 48 different organisations provided input to this report on research assessment and open scholarship. The work and writing were done by consultants Fiona Murphy and Phill Jones of MoreBrains Cooperative, together with the KE Task & Finish group on Research Evaluation.

The work on the Openness Profile relates to issues such as:

• The need to accelerate the transition to open - operationalising and normalising open scholarship practices has proven challenging.
• Conflicting ambitions combined with strong network effects that punish those who deviate from sector norms around research assessment and practice.
• The economic nature of challenges, either financial or relating to actors’ incentives, associated with the transition to open scholarship.
• Distortion of researcher behaviour due to over-reliance on traditional metrics.
• Underfunded and underdeveloped funder grant information systems. Poor adoption of PIDs and little to no interoperability with downstream stakeholders.
• Key contributors to the academic knowledge ecosystem being under-recognised
• Research being organised with ‘well defined’ rules that do not include ‘open’-related criteria.

The potential to improve open research evaluation practice as well as the requirements to implement the Openness Profile are addressed, including recommendations for stakeholders.