Automated Extraction and Analysis of Open Science Metrics in Research Papers

The increased interest in open science and meta-science calls for robust tools, capable of collecting information from published papers. Being able to automatically collect such data at scale is invaluable in order to determine longitudinal trends and the influence of open-science policies publishers are adopting.

In this talk, we introduce an automated tool capable of automatically parsing paper PDFs to identify relevant open-science metrics with the aim of assisting researchers in this field. Our tool is extensible, capable of retrieving different types of information from published PDF files. Concretely, one use case we are currently investigating is automatically determining if the paper links to datasets or code that were produced as a result of the research. As another use case, our tool can analyze references from the bibliography to, for instance, determine if cited papers are open-access or even link to informal, non-scientific, resources. We are also building automated tooling to identify AI-hallucinated citations, an increasing and worrisome phenomenon in academia. As a proof-of-concept, we applied the tool to our own field of research, computer security, and found an increase in shared paper artifacts over time, which can be linked back to changing publication policies at major conferences.

By open-sourcing our tool, we encourage researchers from all fields to utilize the tool to uncover trends in open-science policies and, if needed, finetune its methodology to better suit their field. We hope that the tool can spark research ideas and discussion on conducting meta-science studies with a broader view than our research field leading to more accessible research.