Exploration of Research Impact through IMeTo. Supporting Societal Technology Transfer

IMeTo (Impact Measurement Tool) is the Institute of Literary Research of the Polish Academy of Sciences (IBL PAN) pilot initiative within the GRAPHIA project. IMeTo aims to measure the impact of scientific activities, considering both scientific publications and activities defined by research assessment criteria in a given national context. IMeTo is designed to serve as a tool to assist managers of scientific institutions and teams to evaluate and shape the output of scientific organisations.

GRAPHIA, funded by the European Union through the Horizon Europe programme, aims to strengthen the European Research Area by developing a knowledge graph for the humanities and social sciences, enhanced by advanced AI and LLM-based services, while modernising the next generation of SSH research and innovation. The resulting graph integrates distributed data, using existing infrastructures and data resources such as the EHRI Collection Graph, E-RIHS DIGILAB KG, RESILIENCE ReIReSearch, CNRS KG Matilda, OpenCitations, Open Research Knowledge Graph and GESIS Knowledge Graph. The project involves 20 partners from 10 countries, including five participants in the European Strategy Forum on Research Infrastructures (ESFRI), underlining its importance for the European Research Area.

IMeTo focuses on knowledge and technology transfer in the humanities and social sciences, enabling researchers to more effectively communicate the results of their work and its societal impact. Knowledge and technology transfers are complex processes involving dissemination towards society and business to facilitate commercialisation or wider application. In the context of IMeTo, this transfer focuses on supporting researchers in demonstrating their studies' societal and economic impact. The tool will use pre-defined general and user-defined specific criteria to facilitate the assessment and presentation of the impact of scientific research. In addition, IMeTo will measure the effectiveness and success rate of disseminating research results to a wide audience, thereby enhancing inclusivity in science and promoting community engagement.

A key feature of IMeTo is the use of artificial intelligence (AI) and machine learning (ML) to automate impact assessment, as the tool is intended to help with processing large volumes of data (i.e. departments/universities/institutes' yearly output). To this end, the tool will integrate an open language model (LLM), such as Llama, Mistral or PLLuM, with fine-tuning capabilities. The role of the model will be to classify information in input documents and generate proposed impact descriptions using a hybrid language model approach:

• Analytical model: Extracts key information from documents by assigning relevant labels from typologies to the input data.

• Generative model: Generates impact descriptions based on the extracted information and assigned labels, producing coherent and well-formulated texts that take into account the relationships between different typology elements.

The development of IMeTo includes the annotation of existing scientific impact descriptions. This involves the creation of a set of typologies that reflect the specific characteristics of impact and allow the extraction of key impact description elements as labels, such as scientific activity, impact beneficiaries and intended use of research results.

The annotated data will come from the POL-on integrated network of information on science and higher education in Poland. The database contains descriptions prepared by higher education and research institutions detailing the relationship between research results or development activities (or artistic endeavours) and their impact on the economy, public administration, health care, culture, environmental protection, national security or other social progress factors. These descriptions are used to assess the quality of scientific activities of institutions in Poland.

IMeTo may be implemented as a plugin for Current Research Information Systems (CRIS) or as a standalone application capable of working with local datasets in various formats (e.g., txt, pdf). In addition, the IMeTo application will have an API that will allow researchers, research institutions, funding agencies and evaluators of scientific activities to access its functionalities.

IMeTo is a tool that specifically addresses the challenge of better understanding how SSH can be more accessible to the wider society [1]. Improving the impact of SSH on society depends on obtaining relevant and comprehensive data about how the research influences the public, and currently, as research shows [2, 3], this process needs to be data-driven. IMeTo will serve as a tool that can assist with analysing research output to maximise the impact potential of SSH institutions.

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the Agency. Neither the European Union nor the granting authority can be held responsible for them.