LIBER and LA Referencia Webinar: Making Research Assessment a Reality: Showcasing Concrete Open Science Advancements in Latin America and Europe

LIBER and LA Referencia Webinar - Making Research Assessment a Reality: Showcasing Concrete Open Science Advancements in Latin America and Europe  

On 13 May 2024 LIBER and LA Referencia co-organised a webinar aiming at providing a panel of concrete case studies, inspiring examples and useful contacts for building a framework fostering Open Science practices when evaluating research. The event aimed to introduce several methods and opportunities for aligning practices, fostering communities and partnerships at a global, regional, national and/or local level, and bringing together decision-makers, strategists, researchers and research communities, research staff support and librarians that are willing to join forces. 

The webinar heard from a wide range of speakers from Latin America and global networks within the Research Assessment and Research Data ecosystems, that take Open Science practices into account. 

Andrea Mora, LA Referencia President, and Giannis Tsakonas, LIBER Vice President, opened the session and welcomed the participants. As Ginanis Tsakonas stated : “The matter of research assessment is very critical for the progress of Open Science. Research assessment is dependent on many factors, including equity in visibility, academic freedom, and social contribution. To this end, we reckon that Research Libraries have a lot to offer to our LIBER members and have the capacity, knowledge and interest to promote alternative paradigms of research assessment that respect the particular conditions of disciplines and cultures. We explore the potential of open infrastructures that broaden the data and information that is considered in decision making, we help our institutions to shape policies that include diverse research practices, we challenge the use of traditional tools and metrics that neglect or oppress several types of research outputs, we develop services that work for the benefit of our researchers and of our societies. We are joining our forces and uniting our voice with our colleagues from around the globe and we commit to continue working with them to make Open Science the default paradigm”.  

Three speakers presented various approaches of the research evaluation in an Open Science perspective:  

https://coara.eu/ Eva Mendez 

Eva Mendez (Universidad Carlos III de Madrid, Spain - CoARA Steering Board member)  opened the discussion with her presentation Translating Values into Practice: Showcasing Concrete Advances in CoARA. She highlighted how assessment processes are relying predominantly on journal- and publication-based metrics and can be a hurdle to the recognition of diverse contributions and may negatively affect the quality and impact of research. She pointed out that Responsible Research Assessment can pave the road towards a reform and that the academic world should adapt the set of indicators to the field, project, type, or entity in order to help build new measures. The four main commitments would be : recognise the diversity of contributions to, and careers in, research, in accordance with the needs and the nature of the research ;  base research assessment primarily on qualitative evaluation for which peer-review is central, supported by responsible use of quantitative indicators ; abandon the inappropriate uses in research assessment of journal and publication-based metrics, in particular the inappropriate uses of journal impact factor (JIF) and h-index ; avoid the use of rankings of research organisations in research assessment. She ended up presenting COARA WG and the National chapters. She underlined the major role played by libraries. Reading  https://zenodo.org/communities/coara_action_plans , considering signing the Barcelona declaration and discovering or joining some projects such as https://graspos.eu (next Generation Research Assessment to Promote Open Science), https://opusproject.eu (Reform the assessment of research towards a system that incentivise researchers to practice #OpenScience) and https://pathos-project.eu  (collect concrete evidence of Open Science effects, and study the pathways of OS practices, including enabling factors and key barriers) is a step for fostering reshaping research evaluation. 

https://www.clacso.org/en/folec/ Laura Rovelli 

Laura Rovelli, coordinator of the Latin American Forum for Scientific Evaluation (FOLEC, in Spanish) of the Latin American Council of Social Sciences (CLACSO) presented ongoing reforms on responsible evaluation of research in Latin America and the Caribbean.  Starting from recalling international initiatives and national and regional open science guidelines, she described the initiatives in Argentina, Brazil, Chile and Colombia. Then, she focused on several Open and institutional infrastructure initiatives such as Latindex, Redalyc and the collaboration with LA Referencia. She ended up with major recommendations on reform scaling and scoping ; open, diverse and inclusive infrastructure; institutions and actors. She  concluded citing two fundamental references: https://www.clacso.org/no-solo-abiertas-sino-tambien-diversas-e-inclusivas-hacia-fuentes-de-informacion-sobre-investigacion-descentralizadas-y-federadas/  and 

https://unesdoc.unesco.org/ark:/48223/pf0000388985  

https://www.rd-alliance.org/ Francoise Genova 

Françoise Genova, after being charge of the French space astronomy experiments at the French space agency CNES and former director of the Astronomical Data Centre at the Strasbourg Observatory, is deeply involved with Research data Alliance (RDA). She presented the RDA Value for the Evaluation of Research, with concrete examples for WG outputs and reflexions. RDA’s reach across the international community, including research organisations and funders, and the diverse profiles of its individual members, can help to facilitate and align a global discussion of research evaluation, drawing on data experts from a range of domains. Research software experts are also active in the RDA and their involvement will be sought. Liaison is sought with the Research Software Alliance, with which RDA has established a strong collaboration.With regard to the value of the RDA community expertise, a number of RDA Groups have activities relevant to the many changes surrounding the Evaluation of Research. Evaluation of Research is embedded in a complex ecosystem. Building on the diversity of the RDA community and activities, possible RDA contributions to the debate faell in several categories: possible metrics and criteria, stakeholder representation, research outputs and their relevant properties, disciplinary aspects. The groups dealing with support activities contribute to the ecosystem’s progression by providing discussions and outputs which help researchers to engage in and with the transforming context. 

The most straightforward connections are related to possible metrics and criteria. Several Groups have been active in the domain, in particular the Data Usage Metrics WG and the Sharing Rewards and Credit (SHARC) IG. The IG Understanding and Capturing the Usage of Digital Research Infrastructure which is being currently discussed will also have the capacity to assess possible new criteria if it gathers enough support to be endorsed. 

The evolution of research assessment will be possible only with a contribution of all relevant stakeholders. The RDA contributes a rich palette of groups gathering key stakeholders at the international level. Two of the governance bodies are particularly relevant, the Organisational Assembly and the Regional Assembly, which gather respectively the RDA organisational members and representatives from the countries and regions interested in the RDA. The Funders Forum and its Interest Group enable connections with research funders. The Early Career and Engagement IG gives a voice to the early career researchers working in the domain. The Data Policy Standardisation and Implementation IG gathers publishers and other stakeholders involved in the definition and implementation of journal/publishers’ research data policies. 

One important aspect of the expansion of the Evaluation of Research is the recognition of research output diversity. RDA groups contributing to activities enhancing and supporting open data and more generally open research provide potentially important input on the ecosystem, in support of the adoption of open research practices by researchers and others involved in the research endeavour, a necessary element for the community to accept and engage in changes to the Evaluation of Research. RDA Groups dealing with researcher engagement, education and training include Engaging Researchers with Data IG, Education and Training on Handling of Research Data IG, and CODATA/RDA Research Data Science Schools for Low and Middle Income Countries.  New activities and expertise are required for FAIR data and others involved in the research endeavour are experiencing higher profiles and increased demand in recognition of their new capacities, as represented by the Professionalising Data Stewardship IG and the Libraries for Research Data IG. The numerous activities that support data management, life cycle and FAIRisation also contribute to the transformation of the Evaluation of Research (e.g. Active Data Management Plans IG, Reproducibility IG, National PID Strategies IG, Vocabulary Services IG, Sensitive Data IG, FAIR Principles for Research Hardware, Virtual Research Environment IG, RDA/WDS Certification of Digital Repositories IG, Data Repository Attributes WG.) 

The webinar was facilitated by Cécile Swiatek Cassafieres (LIBER), with concluding remarks by Claudia Cordova from https://www.gob.pe/concytec. Concytec is the National Council for Science, Technology and Technological Innovation in Peru, the governing body of SINACYT, responsible for directing, promoting, coordinating, supervising and evaluating the actions of the State throughout the country in the field of science, technology and technological innovation; it guides the actions of the private sector; and it executes support actions that promote scientific and technological development in the country. Her valuable voice invited participants and academic communities to take advantage of the organisations and networks presented during this webinar, adopt or take inspiration from their methods and outputs, in order to become involved in a process of research reflection aimed at a more impotent opening up of research results and their open dissemination. 

The Q&A session led by Cécile Swiatek Cassafieres (LIBER) addressed the following issues, that you can discover by replaying the webinar: 

• The fundamental role of aligned metadata for opening up sharing research results openly; 

• the more effective way for decision makers to invite scientific researchers to commit themselves with new models of research evaluation, more specifically targeting open  science ; 

• the recommended first step(s) for those in charge of a research unit or an institution in order to engage the research teams into a renewed research evaluation framework ; 

• As partnering seems to be a key for renewing the research evaluation models and orienting them towards open science, advice for improving the global collaboration on this topic; 

• Some experience sharing with success, failure, hope and regret. 

Keywords from the Q&A session are : interoperability, metadata, alignment, shared practices, and knocking on the decision-makers doors in order to put the research reform on the political and strategic agenda at all levels. 

View the event recording: 

• Webinar recording (Español) YouTube 

• Webinar recording (English) YouTube