Quality, impact, productivity: Researchers' conceptions of bibliometric indicators and altmetrics

Bibliometric measures are increasingly used in academic funding and hiring situations. Although rather vaguely described as “impact” or “performance” indicators, they often appear to be equaled with research quality. Altmetrics challenge established bibliometric evaluation concepts by promoting alternative notions of impact: Attention, visibility or societal relevance are some of the concepts that may be used to charge metrics with value and thus add to the many dimensions of quality. Research quality, however, is a hard to define concept. Researchers are likely to express different understandings of quality, influenced by their fields’ shared epistemic and cultural beliefs. Evaluative contexts that make use of indicators often crucially determine researchers’ career paths. They can thus be assumed to affect researchers’ conceptions of quality and consequently shape the way research is conducted, communicated and received. We argue that in order to use metrics responsibly, we need to understand the relation between quality and indicators. However, research in this regard is limited. Several survey and interview studies have investigated the use of and opinions on metrics by researchers from different fields and career levels (Aksnes & Rip, 2009; Aubert Bonn & Pinxten, 2021a, 2021b; Hammarfelt & Haddow, 2018; Langfeldt et al., 2021; Lemke et al., 2019; Ma & Ladisch, 2019). Their findings suggest which indicators researchers use for peer evaluation or self-marketing, and reasons for objecting the usage of indicators, such as limitations of the indicators’ validity or reactive effects on publishing practices. Hammarfelt & Haddow (2018) showed that scholars in the humanities experience that indicators do not reflect their understanding of quality. While humanities are known to be disadvantaged in bibliometric analyses, our study lays emphasis on the use of indicators with respect to different notions of quality outside the humanities. It explores how researchers in the life sciences assess and prioritize different bibliometric measures and altmetric data in terms of their suitability for evaluating scientific work. What are researchers’ conceptions of bibliometric and altmetric indicators regarding research quality? How is quality related to notions of impact and productivity?