A comparison of three multidisciplinarity indices based on the diversity of Scopus subject areas of authors’ documents, their bibliography and their citing papers

In this paper, we compare the distribution of Elsevier Scopus subject areas of authors’ documents, their bibliographical references and their citing documents. We compute the complement of the Herfindahl–Hirschman index as a measure of multidisciplinarity. We analyse a sample of 120 researchers belonging to two groups, one from the Italian Institute of Technology (IIT, whose work is expected to be highly multidisciplinary) and one from the National Institute for Nuclear Physics (INFN, whose work is expected to be much less multidisciplinary). We show that the two groups are distinguishable through the measured index values. By using the subject areas of authors’ bibliographical references we obtain a better identification of the two groups than relying on the subject areas of the author’s documents. We then extend the analysis to 3317 researchers belonging to seven Italian Scientific-Disciplinary sectors (SSD) providing insights about the degree of multidisciplinarity within each SSD. The results seem interesting for assessing the multidisciplinarity of younger researchers with scarce scientific output and few citations.


Introduction
Nowadays, measuring and understanding multidisciplinary research is of prominent interest for both researchers and funders or evaluators (Wagner et al. 2011;Wang et al. 2015), also to choose proper assessment parameters that go along with the characteristics typical of a particular scientific field. Multidisciplinary works and consequent applications are generally considered to have a stronger impact on the society and the scientific development (Rafols and Meyer 2007). In spite of this growing interest, both defining and measuring multidisciplinarity is not a trivial task. Several definitions have been given to describe the multiple modalities in which disciplines can interact (Stokols et al. 2003;Choi and Pak 2006;Porter et al. 2007). For a systematic review, see Rousseau et al. (2019). We use the term multidisciplinarity in a broad sense, indicating that elements from different disciplines are present. The term "multidisciplinarity" is used in the following and the concept of interdisciplinarity is inherent to it. Since many years, a majority of quantitative indicators of the degree of multidisciplinarity of a researcher are based on bibliometric methods (Porter et al. 2007). These methods are commonly classified in bottom-up and top-down approaches (Wagner et al. 2011). Bottom-up approaches are based on grouping and forming sets of articles according to a criterion, like building a co-citation network (Boyack and Klavans 2010). This approach is suitable for finding emerging fields, in which there is no classification available, very few publications and no a priori taxonomy (Leydesdorff 2007;Leydesdorff et al. 2013). Top-down approaches are dependent on existing classifications available. They are suitable for a large-scale analysis, especially when dealing with big amount of data (Porter and Rafols 2009;Leydesdorff et al. 2013). In fact, published manuscripts are commonly included and indexed in several databases such as Elsevier's Scopus and Clarivate Analytics' Web of Science (WoS). A feature offered by such databases is that main topics or research areas are assigned to articles and journals via (semi-)automatic methods. Bibliometric data and their classification into topics are the raw material used to measure multidisciplinarity. In 2007, Stirling introduced a general framework for analysing diversity in science that takes variety, balance and disparity into account-all properties of a diversity measure (Stirling 2007). Not only the number of disciplines in articles (or in their reference lists) are taken into account, but also the distance between them: Stirling suggested to apply the Rao index developed in biology (Rao 1982) to measure research multidisciplinarity, defining the so-called Rao-Stirling index. The Rao-Stirling index is a popular indicator of multidisciplinarity, used in many works (Rafols and Meyer 2007;Leydesdorff and Rafols 2011;Leydesdorff et al. 2013;Wagner et al. 2011). However, creating a meaningful distance measure is far from being an easy task (Rafols and Meyer 2007). This requires the selection of a context (a set of papers), the identification of attributes on which a distance measure is based and so on.
In our work, we are going to analyse the behaviour of the Rao-Stirling index when no distance measure is used. This index was demonstrated to be equal to the Herfindahl-Hirschman index, also known as Simpson (Simpson 1949) index, very popular in economics (Rhoades 1993;Rousseau 2018), with different choices of parameters (Porter and Rafols 2009). In the existing literature, the Herfindahl-Hirschman index is often presented together and compared to the Shannon index [see Stirling (2007), Wang et al. (2015), Porter and Rafols (2009) and Moreno et al. (2016)]. To measure multidisciplinarity, we use the complement of the Herfindahl-Hirschman index (CHH), named Herfindhal's diversity in Porter and Rafols (2009). To define disciplines, we use the top-down classification provided by Scopus. The disciplines are named subject areas (SA) in Scopus. We use the 1 3 CHH index to analyse the differences in the distribution of subject areas in three cases: the subject areas of an author's documents, of his/her documents' bibliographical references and the citing publications of his/her documents. The evaluation is conducted on two sets of researchers and their publications in the years 2010-2018. The first set is made of researchers working at the Italian Institute of Technology (IIT) that is a Foundation established by Law no. 326/2003, and financed by the Italian State to conduct scientific research in the public interest, for the purpose of technological development 1 and according to its strategic plan is engaged in highly multidisciplinary research activities. These researchers, then, are expected to be highly multidisciplinary. The second set contains researchers affiliated with the National Institute for Nuclear Physics (INFN) 2 in Italy. These researchers are expected to be much less multidisciplinary. "Experiments on seven Italian Disciplinary Sectors" section extends the analysis of Moschini et al. (2019) by investigating the multidisciplinarity of 3317 researchers. This larger set contains all the researchers active in Italy in seven Scientific-Disciplinary Sectors (SSD). "Conclusions and further extensions" section puts forward possible extensions of the CHH index and summarizes the main results.

Distribution of subject areas
For all the sets of researchers in the paper (IIT, INFN, researchers in SSDs), we considered only the documents published in the period 2010-2018, as indexed in Scopus. Documents were accessed through the Author Scopus identifiers of each researcher profile on Scopus: the subject areas were retrieved via a script based on the Scopus Application Programming Interface (Elsevier 2019). The Scopus classification consists of 334 subject areas (SAs) that are grouped in 27 macro-categories. Scopus assigns a varying number of subject areas to the source medium in which a document appears (i.e., journal, conference proceedings, book, …), rather than to the document itself. The classification is assigned by Scopus manually at the time when a new journal is setup for Scopus coverage, based on the scope of the journal and its published articles. There is no regular process to update the classification, however possible changes can be requested from the publisher of the journal. This process may of course create some distortions in the analysis as publications (and/or their venues) can be increasingly assigned to more than one SA. Since Scopus assigns subject areas to journals or conferences and not to the article itself, the assumption is that an article inherits the SAs of the source where it is published. In our analysis, we distinguished three different ways to consider the subject areas that could potentially describe the disciplines belonging to the scientific production of a researcher: 1. SAs of the documents written by a researcher (DOC). 2. SAs of the bibliographical references of every document written by a researcher (BIB). 3. SAs of the publications that cite the documents written by a researcher (CIT).
We computed the percentage of publications falling in each subject area, for each of the three cases. Note that Scopus frequently assigns two or more SAs to a publication: in that case, the publication is counted one time for each SA, as the number of publications in each SA is being counted. The percentages were then normalized to range from 0 to 100. In the following, the subject area "Multidisciplinary" (MULT) is excluded: it indicates journals that publish articles coming from several disciplines (e.g., "Nature" is classified as MULT in Scopus), rather than inherently multidisciplinary articles and it was removed from the analysis. We are going to refer to a total of 333 SAs and 26 macro-categories. A researcher is described by three vectors of 333 components, each vector component representing a given SA. Publications co-authored by multiple authors who are in our set simply contribute to the vectors independently.

The index used in our work
To measure the multidisciplinarity of authors' research outputs we used the complement of the Herfindahl-Hirschman (CHH) index, also known as Simpson diversity index. The Herfindahl-Hirschman index is a very popular concentration index in economics, used to measure how the market share is distributed among companies. In our context, the marketshare distribution among companies translates into the subject areas related to a researcher, computed as a vector containing the percentage of publications categorized under each subject area, as explained in "Distribution of subject areas" section.
Let us call V a vector containing percentage of publications per SA and let us assume its values are normalised to range between 0 and 1. It is easy to see that less concentration means more multidisciplinarity. Let us define CHH(V) as the complement of the Herfindahl-Hirschman index, computed on a vector V whose components represent N subject areas, as: The higher the value, the less concentrated are the subject areas, hence more multidisciplinarity is achieved. The function described by Eq. (1) takes values in the range from 0 to (1 − 1 N ).

Experiments on IIT and INFN dataset
The first experiment relates to two sets of researchers. The first set contains 64 researchers affiliated with the Italian Institute of Technology (IIT). They are expected to show a high multidisciplinarity degree: their scientific output often combines, for example, robotics with life sciences, medicinal chemistry with biology, and so on. The second set contains 56 researchers affiliated with the National Institute for Nuclear Physics (INFN), whose work has inherently a narrower scope: hence, less multidisciplinarity is expected. They are senior researchers, i.e. with a number of publications at least larger than 20. We are interested in verifying if the CHH index can actually distinguish two sets of researchers with values of multidisciplinarity expected to be very different, as for our IIT and INFN sets: the index should show higher values on the IIT set compared to the INFN set. In the "Appendix" at the end of the paper, two tables report the Scopus subject areas of the authors' publications in descending order and confirm the higher multidisciplinarity of IIT authors w.r.t. the INFN authors. (1) The aim of our tests is to check the discriminating power of the index between the two groups of researchers, when considering (1) the subject areas of the author's documents, (2) the bibliographical references of author's documents, (3) or the citing documents of author's documents. The computed index is the CHH index on the vectors of percentages of publications falling in each Scopus subject area, for the 120 researchers, for the three cases DOC, BIB and CIT described in "Distribution of subject areas" section. Only documents published within the years 2010-2018 were considered, to avoid a potential correlation between multidisciplinarity and the age of a researcher. Each of the 120 researchers published documents before 2010: these documents were simply discarded. Figure 1 shows the results obtained after computing the CHH index, multiplied by 100 for convenience. In the plot, the IIT researchers are indicated in black colour, while the INFN researchers in grey. The CHH values are shown sorted. We notice that the range of index values for INFN researchers is much broader in the BIB and CIT case than in the DOC case. The mean value for the INFN researchers in the DOC case is much more different from the BIB and CIT case. Instead, the mean values for IIT researchers fall in a similar range in each of the three cases. Table 1 shows in details the range of indexes and the mean/ median values computed for the IIT and INFN researchers for the three cases. In Fig. 1, we would expect that all the IIT researchers stay on top of the INFN researchers, due to the fact that they are more multidisciplinary. We notice that the two sets are split up to a good extent in the three cases. However, there are researchers who cross the boundary of their own group, overlapping with the others. Interestingly, the same researcher is common to both DOC and BIB. The actual scientific production of these "overlapping" authors should be further investigated: their research could be not so monothematic or multidisciplinary as we thought initially. The limit between what can be multidisciplinary and what cannot be is unlikely a rigid and exact boundary: it is a promising sign, however, that researchers of a set overlap with researchers of the other set only near the boundary region.
If we select a few researchers less multidisciplinary and less "monodisciplinary" (i.e., at the boundary of the black and grey "regions" of Fig. 1), we note that most of them are the same researchers across the three cases: they have approximately the same ranking position in each case. Then, we extended our analysis to study how the relative ranking positions of the researchers change throughout DOC, BIB and CIT. We measured that 45.83% of the researchers remained in a range of ± 5 positions, comparing the ranking of DOC w.r.t. BIB. The percentage goes up to 60.83% for BIB w.r.t. CIT. If the range considered is ± 10 positions, the percentage is 62.5% and 80.83%, considering DOC w.r.t. BIB and BIB w.r.t. CIT. Also, the experiments show that the ranking changes less on the INFN set, although the size of the sets is not large enough to draw precise conclusions. Overall, only about 10% of the researchers maintain the same position in the ranking across the cases. By and large, it appears that researchers keep roughly their position (about 85% of the researchers stay in a range of ± 15 positions) in the three cases. We computed also the Kendall rank correlation among the vectors of percentages in the three cases. On the whole set of 120 researchers, Kendall's correlation is 0.78 and 0.81, for DOC w.r.t. BIB and DOC w.r.t. CIT, respectively. Considering the IIT set, the correlation decreases to 0.61 and 0.67, while on the INFN set it goes down to 0.54 and 0.58. This means that the order of the researchers generated by the index based on the three different vectors (DOC, BIB, CIT) varies largely within each set, but the overall order keeps the separation of the two sets, hence we measure higher correlation when considering all the researchers together. This happens because Kendall's correlation counts how many concordant or discordant couples are present: an increase in correlation means that the percentage of concordant couples (couples of researchers for whom the order does not change) is higher considering the whole sets of researchers together.
We observe that DOC, BIB and CIT may measure different kinds of multidisciplinarity: the citations indicate the visibility of an author's work among various disciplines rather than the multiple disciplines a work is based on. Integration of different disciplines is probably better shown by the references cited by an article in its bibliography (Porter et al. 2007): an authors' own publications would instead indicate how diverse the individual production is. From our study, the use of bibliographical references (the BIB case) showed a better separation between IIT and INFN than the DOC case: the mean/median are further apart and less researchers end up in the wrong region. The use of the bibliographical references gives good clues about multidisciplinarity and, in contrast to the CIT case, also allows us to measure younger researchers whose scientific output is still limited or who are not cited many times: the bibliography is normally composed by a large number of entries, making it suitable in these situations, too. It would also be interesting to check if a correlation exists between long reference lists (i.e., knowledge base) and being cited by a more multidisciplinary audience (examining the CIT case) and if the number of co-authors of a paper affects the degree of multidisciplinarity (Abramo et al. 2018a).

Experiments on seven Italian Disciplinary Sectors
"Experiments on IIT and INFN dataset" section shows that the CHH index computed on bibliographical references' data discriminates well the two sets of researchers with lower and higher multidisciplinarity. In this section, we extend the analysis to a larger set of 3317 Italian researchers, for which we had data at our disposal. In the Italian system, academic research areas are categorized into officially pre-assigned Scientific Disciplinary Sectors (SSDs), as defined by the Attachment A of the Ministerial decree n. 855 of October 30, 2015. The full list is also available on Wikipedia Italy (permalink https ://w.wiki/Byp). There are 367 SSDs, divided into 88 macro-sectors distributed over 14 main disciplinary areas. Every Italian researcher in the Italian academic system belongs to a SSD. Multidisciplinarity can be analysed then within a particular SSD classification, available for all the authors. Abramo et al. (2012Abramo et al. ( , 2018b measured multidisciplinarity through author's affiliations and SSDs. In our work, instead, we measure multidisciplinarity through the computation of the CHH index. The population of all the researchers in seven SSD is selected, as detailed in Table 2. Similarly to what described in "Distribution of subject areas" section, the subject areas of each author's publications were retrieved from Scopus to build the corresponding 3317 vectors (V) of subject areas, computed on the bibliographical references of the authors' documents (BIB case).
In general, when evaluating researchers, it would be important to define a comparison set made of researchers who showcase similar features, inherent to their own actual scientific production. Researchers belonging to the same SSD can be fairly compared if they have similar multidisciplinarity values and their scientific production spans similar areas. For each SSD, the CHH index on the subject areas of the bibliographical references' data is computed. Figure 2 shows the results. The boxplots of the SSDs are sorted in increasing order of the median value of the index. Some SSDs, such as ING-INF/05, MED/09, CHIM/03 and BIO/10, show a low level of dispersion of the CHH index. On the contrary, the dispersion of FIS/02 and MAT/05 is considerable. This means that within "Theoretical Physics (FIS/02)" and the "Mathematical Analysis (MAT/05)" disciplinary sectors, there are researchers with varying values of the CHH index. The scientific production of FIS/02 and MAT/05 researchers seems more multidisciplinary or at least their scientific production spans a broader set of disciplines compared to the other disciplinary areas considered. We analyse the profile of FIS/02 to understand in more details the behaviour of the index in these two SSDs. We choose to run in R-language the well-known k-means clustering method 3 to see how the set of FIS/02 researchers gets split into two clusters ( k = 2 ) according to their vectors of subject areas' distributions (BIB case).
The two panels in Fig. 3 illustrate the discipline distributions for the researchers grouped in the two clusters identified by k-means. For each of the two clusters, the average percentage of publications falling in each subject area is shown. The first group (upper panel in Fig. 3) contains researchers with a single main subject area related to PHYS (Physics), with code X3106; the second group (lower panel in Fig. 3) contains researchers whose activity spans more subject areas, linked to both PHYS and MATH (Mathematics). From these results, the whole set of researchers in FIS/02 is split in two clusters that show different publication profiles. The cluster in the lower panel shows a higher multidisciplinary level than the other: this explains the fact that the CHH index values related to FIS/02 in Fig. 2 span a wide range of values: within FIS/02, there are multidisciplinary and less multidisciplinary researchers, showing that this SSD consists of heterogeneous researchers that probably should not be grouped together. Similar reasoning and considerations apply to MAT/05, too. A hypothetical fair comparison among researchers within FIS/02 should account for the different aspects of their scientific production, as highlighted by the distribution of SAs. The CHH index and the inspection of the subject areas involved can possibly point out differences in scientific production, useful for evaluation purposes and for building a more homogeneous set of researchers sharing similar publication characteristics.

Possible extensions using Topic Clusters
As explained in "Experiments on IIT and INFN dataset" section, the CHH index well indicates researchers whose research is very "monodisciplinary". However, from the box plots in Fig. 2 (2017) and Small et al. (2014). We have recently started to evaluate the CHH index computed on vectors representing Topic Clusters. Some preliminary analysis conducted on Topic Clusters seems promising, in terms of distribution of index values and better representing the actual scientific production of a researcher. Future extensions of this work will investigate deeper Topic Clusters, analysing and comparing also other multidisciplinarity indices.

Conclusions
In literature, the CHH index has been often used to measure multidisciplinarity, although, to the best of our knowledge, not on Scopus subject areas (SAs). In our paper, first we analysed its performance on data related to IIT and INFN researchers. The two sets were chosen so that the former would potentially show a more multidisciplinary scientific production, while the latter less multidisciplinarity. We considered three cases of subject areas (SAs) from Scopus: the SAs of the documents written by an author, those of the bibliographical references of an author's documents and those of the citing documents of an author's documents. The two sets are differentiated to a good extent by the CHH 1 3 index. Using the SAs of the bibliography of the publications of a researcher seems the best way forward: the distinction between the two groups of researchers is higher than the one obtained using the SAs of the researchers' articles. Moreover, the larger number of bibliographical references would make the method suitable for researchers with a low number of publications and few citations. The analysis was extended to a large sample of 3317 Italian researchers, working in seven scientific fields to analyse the range of the multidisciplinarity index inherent to a specific area. The scientific fields are defined through a categorization of the Italian academic system, in which research areas translate into officially pre-assigned Scientific Disciplinary Sectors (Settore Scientifico Disciplinare, SSD). In two SSDs, the CHH index values span a broad range of values: we noticed that the researchers are grouped in the same SSD, in spite of showing a scientific production quite different in terms of subject areas. The use of the CHH index could help in defining a proper comparison set, not based (only) on SSDs when evaluating researchers, embracing also researchers from institutions for which the official disciplinary classification is not available. The evaluations we have carried out in this paper could be further extended to check the robustness of the whole disciplinary classification of the Italian academic system.