Methodological shortcomings of bibliometric papers published in the journal Sustainability (2019-2020)

This paper has explored various issues relating to the methodology used in bibliometric analyses published in the journal Sustainability in 2019 and 2020, showing that many of the papers published lack the methodological rigour that would normally be required. In particular, the following limitations have been identified:

• Small samples: bibliometric analyses do not make sense “when the scope of review is specific” or “when the dataset is small and manageable enough that its content can be manually reviewed” (Donthu, Kumar, Mukherjee, Pandey, & Lim, 2021). In this regard, a total of 41 papers (20.1%) were detected that analysed samples of less than 200 documents, suggesting that such analyses could have been approached using other more appropriate techniques.
• Inadequate description of the search equation applied: not all the terms used in the search are broken down in detail, nor the fields in which the search is performed. 102 of the papers reviewed (50%) had shortcomings of this nature, making it difficult or impossible to reproduce the original bibliometric analysis.
• Lack of information on the indexes used within the Web of Science platform: 90 of the papers (44.1% of the total sample and 64.7% of the studies using WoS) did not indicate the specific sources from which the information had been extracted. Some of these papers did list the Web of Science Core Collection as the data source, but without specifying the specific databases searched. As highlighted by Liu (2019), this is an inadequate practice given that “many institutions may choose to subscribe to a customised subset of the whole core collection,” which significantly affects the reproducibility of the work.
• Lack of information on the document types and languages covered by the bibliometric analysis: up to 103 of the papers analysed (50.5%) did not break down either (or both) of these variables.

Considering these shortcomings, 181 of the 204 studies analysed (88.7%) have one or more methodological limitations which hinder or prevent their reproducibility. This shows that there is considerable room for improvement in the methodological quality of the bibliometric papers published in Sustainability, although there are no significant differences between the characteristics of the papers published in special issues and those published in regular issues. Accordingly, our results further underline the concerns raised by authors such as Jonkers & Derrick (2012) and González-Alcaide (2021) regarding the quality of bibliometric papers published in journals outside the IS field, or by Oviedo-García (2021) regarding the rigour of the peer review processes applied by MDPI.

One limitation of this study is that it is a case study of a single journal and it does not use a control group to determine whether the methodological shortcomings detected also occur in IS journals, and more specifically in the most productive journal in the field, Scientometrics. In terms of future studies, it would be interesting to extend the content analysis to other variables, such as the techniques used to analyse the results of these studies, or other aspects relating to their reproducibility or transparency. Finally, we suggest that these variables could be studied in other journals with extensive bibliometric production outside the IS field, as well as measuring their relationship with the formal characteristics of the papers, such as the number of authors, collaborations or citations received.

In short, there is a need to further study the characteristics of bibliometric analyses published in journals of all disciplines in order to promote increasingly rigorous use of this powerful quantitative methodology.