The Puzzle of Research Evaluation: Opportunities and obstacles on the way to full Open Scholarship

In spite of high-level policies and broad enthusiasm for open scholarship, limitations in recognizing individual contributions remains an obstacle to full implementation. Within the Open Scholarship community there is increasing awareness that (1) better recognition and reward of open practices, especially in the context of researcher evaluation, is a necessary condition for broad adoption of open scholarship, (2) top-down guidance, which is aimed at a wide range of disciplinary practices, will require substantial cultural change to shift the priority of research evaluation toward principles of open scholarship^[1] and (3) bottom-up open scholarship initiatives and practices must emerge from local contexts to enable relevant, sustainable change^[2]. 

Developed by the Knowledge Exchange^[3], the Openness Profile concept is a middle-out response to the above challenges. The concept and related workflows were refined on the basis of semi-structured interviews and a subsequent set of focus groups. Overall, the Openness Profile has been received enthusiastically among the wide range of stakeholders who participated. Researchers, infrastructure managers, and especially research funders, identified ways in which the Openness Profile could facilitate recognition of (while also incentivizing) open scholarship. Participants also welcomed the possibility to assess openness at different levels of aggregation, beyond its intended use for individuals, such as at the level of faculty, university, or country. All views and considerations from the interviews and focus groups, as well as suggested requirements to implement the Openness Profile in research evaluation practice, will be captured in a published report[4].

In addition to the above outcomes, our investigation revealed obstacles that hinder intervention in research evaluation practices. Stakeholders identify complexities in their local evaluation practices and concerns about the interests of multiple stakeholders that contribute to systems and infrastructures that underpin local evaluation events. This apparent impasse resembles findings from a recent EUA survey^[5], in which university leaders (in Europe) identify having both the autonomy and authority to change research evaluation in relation to open scholarship. However, they also cite the complexity of research evaluation as well as the popularity of quantitative metrics as barriers to change.

In this presentation, I introduce the Openness Profile, a middle-out resource for research evaluation, along with both the opportunities it invokes and the obstacles it reveals. The presentation concludes with a proposed summit meeting to bring bottom-up initiatives in dialog with top-down policy, which is intended to engage participants in ideas and actions aimed at moving toward better alignment of research evaluation and open scholarship.  

^[1]  Evaluation of Research Careers fully acknowledging Open Science Practices 2017: https://ec.europa.eu/research/openscience/pdf/os_rewards_wgreport_final.pdf

^[2] Incentives and Rewards to engage in Open Science Activities 
https://rio.jrc.ec.europa.eu/sites/default/files/report/MLE-OS-Report-3%20.pdf  

^[3] Knowledge Exchange Openness Profile, accessed 09 October 2020 
https://www.knowledge-exchange.info/event/openness-profile     

^[4] Expected in first quarter 2021, final report will be announced here: 
https://www.knowledge-exchange.info  

^[5] Research Assessment in the Transition to Open Science 2019 
https://www.eua.eu/component/attachments/attachments.html?id=2444