Planned intervention: On Wednesday April 3rd 05:30 UTC Zenodo will be unavailable for up to 2-10 minutes to perform a storage cluster upgrade.
Published March 17, 2022 | Version v1
Preprint Open

Identifying science in the news: An assessment of the precision and recall of Altmetric.com news mention data

  • 1. Simon Fraser University, Canada
  • 2. University of Victoria, Canada
  • 3. Uniformed Services University, USA
  • 4. San Francisco State University, USA

Description

The company Altmetric is often used to collect mentions of research in online news stories, yet there have been concerns about the quality of this data. This study investigates these concerns. Using a manual content analysis of 400 news stories as a comparison method, we analyzed the precision and recall with which Altmetric identified mentions of research in 8 news outlets. We also used logistic regression to identify the characteristics of research mentions that influence their likelihood of being successfully identified. We find that, for a predefined set of outlets, Altmetric’s news mention data were relatively accurate (F-score = 0.80), with very high precision (0.95) and acceptable recall (0.70), although recall is below 0.50 for some news outlets. Altmetric is more likely to successfully identify mentions of research that include a hyperlink to the research item, an author name, and/or the title of a publication venue. This data source appears to be less reliable for mentions of research that provide little or no bibliometric information, as well as for identifying mentions of scholarly monographs, conference presentations, dissertations, and non-English research articles. Our findings suggest that, with caveats, scholars can use Altmetric news mention data as a relatively reliable source to identify research mentions across a range of outlets with high precision and acceptable recall, offering scholars the potential to conserve resources during data collection. Our study does not, however, offer an assessment of completeness or accuracy of Altmetric news data overall.

Files

preprint-submission.pdf

Files (524.0 kB)

Name Size Download all
md5:3302013d5df9047aa49709ade11b87f1
524.0 kB Preview Download

Additional details