Replicability and transparency in topic modelling: developing best practice guidelines for the digital humanities

The sharing of resources (that is, data and tools) is a growing trend among researchers; it saves time throughout the data collection and subsequent analysis, and builds collaborations between scholars and disciplines. The field of Digital Humanities is, for the most part, made up of researchers who rely on existing data and tools for their analysis. However, simply making resources available alongside paper submission is not enough. The average researcher (i.e., one that is not fully proficient in computational methods) should also be able to understand the steps taken to build the dataset, and the mechanism of the method applied to process and analyse the data. That is particularly true for studies with language in use, as the way data is collected, prepared, queried and analysed varies greatly according to the register type, language variety, research questions, etc. Putting procedures in place is therefore paramount – it all comes down to transparency and a critical mindset.

To illustrate our claim, we report on a case study applying topic modelling to the analysis of literary texts obtained from the Gutenberg Project. Using Jupyter Notebook (a web-based interactive environment that allows sharing live code and its documentation), we demonstrate how customization of the parameters throughout the whole process (e.g., data scraping, cleaning, tokenization rules, tagging, stopword selection, TM methods) affects the topic model output; and how that, in turn, can affect researcher interpretation. We argue that one way of achieving replicability and transparency is by offering, alongside the aforementioned resources, a guide that allows users to understand its mechanisms by reproducing its steps. We will also discuss how being aware (i.e., having sound knowledge of the process) is crucial for final data analysis and interpretation.