How does Our Research Influence Policy on Global Societal Changes? A Bibliometric Proof of Concept Targeting the Sustainable Development Goals of the United Nations

University leaders asked the library for new ways to measure societal impact and the university’s connectivity to society.

In this project we created a proof of concept for analysing the research quality and policy impact related to each of the 17 Sustainable Development Goals (SDGs), which the United Nations has set as challenges for the World. We have developed a tool that gives insight into the University’s performance, the excellence of that research performance, to what extent that research is freely accessible to society, and most importantly the extent it is adopted by (non-) governmental policy.

We present all this information in an interactive dashboard, which allows users to arrange the data from different perspectives. It allows university leaders to see the unique societal profile of their research, but also helps to develop new research strategies based on the societal narrative.

With a team of bibliometricians from nine universities in the AURORA-network, we created and reviewed 17 queries – one for each SDG – based on the UN policy text and indicators for each global goal. We collected the publications using Scopus, and used Scival to get the top 10% journal citations. Open Access data was harvested from Unpaywall/Impactstory, and policy mentions from Altmetric. First we used a manual workflow to track the entire process, but have now developed an automated workflow, which allows for rapid evaluation of other societal themed queries.

The dashboard generates unique insights, distributed in particular among two quadrants: “opportunities” and “strong SDGs”. The first quadrant shows above average research excellence (horizontal axis) combined with lagging citations in policy documents (vertical axis). The strong SDGs quadrant represents SDGs where both the research excellence as well as the policy citations are greater than average. For the “opportunities” we discovered that although 58% of the “Climate Action” research (SDG13) was published in the top 10% percentile of most cited Journals, only 8% of that research was used in policy by (non-)governmental organisations. For “Good Health and Well Being” 43% of the papers fell in the top 10% percentile, and 19% ended up in policy citations. The “Opportunities” quadrant thus represents excellent research that is still left largely unused by societal policy partners. Shifting the data perspective, we can see which universities have the most policy influence, and in a group of universities like AURORA, have a conversation on how to use their network to reach policymakers better.

The challenge for us now is to make the tool robust enough to support strategic decision-making, by increasing the recall and precision of the queries underlying the data collection.