RepoSZ - PH Schwyz Repository
Published October 4, 2021 | Version v1
Preprint Open

How systematic are systematic literature searches?

Authors/Creators

  • 1. Pädagogische Hochschule Schwyz

Description

Systematic literature reviews and meta-analyses have gained importance due to the increasing rates of academic publications and the need to evaluate and integrate findings. The systematic literature search using databases and platforms is the essential first step in the process of conducting research syntheses. We investigated the replicability of literature searches conducted within two databases (ERIC, PsycINFO) and two search platforms (ProQuest, Web of Science). Fifteen researchers from different institutions performed identical searches guided by a manual. We found that ERIC searches were replicable, but the type of quotation marks used in the search string mattered. Despite PsycINFO being frequently reported in published reviews and meta-analyses, none of the researchers had direct access, but used alternative platforms (EBSCOhost or Ovid) to access PsycINFO. Using a specific platform (either EBSCOhost, Ovid, or ProQuest) to indirectly search the databases, PsycINFO and/or ERIC provided identical search results across institutions. However, the indirect search via platforms mostly delivered different numbers of hits compared to the direct search in the databases. Additionally, the searches via platforms identified publications that partially differed from the publication identified via direct searches in the databases. Web of Science searches were hardly replicable; results differed between institutions with regard to the number of hits and in the composition of the identified publications. To increase replicability, we suggest expanding reporting guidelines like PRISMA-S to require researchers to publish their full initial search hits.

Files

How systematic are systematic literature searches .pdf

Files (205.8 kB)

Additional details

Related works

Is published in
Conference paper: 10.33683/dida.22.05.83 (DOI)

References

  • Bornmann, L., & Mutz, R. (2015). Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references: Growth Rates of Modern Science: A Bibliometric Analysis Based on the Number of Publications and Cited References. Journal of the Association for Information Science and Technology, 66(11), 2215–2222. https://doi.org/10.1002/asi.23329
  • Cram, W. A., Templier, M., Université Laval, Pare, G., & HEC Montréal. (2020). (Re)considering the Concept of Literature Review Reproducibility. Journal of the Association for Information Systems, 21(5), 1103–1114. https://doi.org/10.17705/1jais.00630
  • Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods, 11(2), 181–217. https://doi.org/10.1002/jrsm.1378
  • Jacso, P. (2005). As we may search—Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science, 89(9), 1537–1547.
  • Lam, M. T., & McDiarmid, M. (2017). Increasing number of databases searched in systematic reviews and meta-analyses between 1994 and 2014. Journal of the Medical Library Association, 104(4). https://doi.org/10.5195/JMLA.2016.141
  • Landhuis, E. (2016). Scientific literature: Information overload. Nature, 535(7612), 457–458. https://doi.org/10.1038/nj7612-457a
  • National Research Council. (1999). A Question of Balance: Private Rights and the Public Interest in Scientific and Technical Databases (p. 9692). National Academies Press. https://doi.org/10.17226/9692
  • Newman, M., & Gough, D. (2020). Systematic Reviews in Educational Research: Methodology, Perspectives and Application. In O. Zawacki-Richter, M. Kerres, S. Bedenlier, M. Bond, & K. Buntins (Eds.), Systematic Reviews in Educational Research (pp. 3–22). Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-27602-7_1
  • Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, n71. https://doi.org/10.1136/bmj.n71
  • Patra, N. K. (2017). From Electronic Resources to Electronic Resource Management. In Digital Disruption and Electronic Resource Management in Libraries (pp. 13–25). Elsevier. https://doi.org/10.1016/B978-0-08-102045-6.00002-9
  • Pozsgai, G., Lövei, G. L., Vasseur, L., Gurr, G., Batáry, P., Korponai, J., Littlewood, N. A., Liu, J., Móra, A., Obrycki, J., Reynolds, O., Stockan, J. A., VanVolkenburg, H., Zhang, J., Zhou, W., & You, M. (2020). A comparative analysis reveals irreproducibility in searches of scientific literature [Preprint]. Scientific Communication and Education. https://doi.org/10.1101/2020.03.20.997783
  • PRISMA-S Group, Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B. (2021). PRISMA-S: An extension to the PRISMA Statement for Reporting Literature