Using Structured Expert Judgment to predict a university ranking in the Humanities

This report is an assignment for the MOOC Decision Making Under Uncertainty: Applying
Structured Expert Judgment. The goal of the exercise it to familiarize with the application of Structured
Expert Judgment, the design and implementation of an elicitation and the reporting of findings. Structured
expert judgment uncertainty quantification is a methodology developed by R. Cooke [3] at TU Delft and
used in situations where decision making takes place despite a lack of information. This course is part of
the Training Plan of the MSCA project Unveiling the Ecosystem of Science: A Contextual Perspective on
the Many Roles of Scientists funded by the LEaDing Fellows Programme under the Marie Sklodowska-Curie
grant agreement No 707404.
The exercise aims at reporting a hypothetical ranking of Spanish universities in the field of Humanities.
While Structured Expert Judgment is usually applied in fields in which there is a lack of data, in the
particular example displayed here the starting point is different. We are analyzing a field in which there is
a lack of consensus towards which data and how such data should be used to assess performance in these
fields. The experts were directed to provide answers on the estimated rank a university should receive based
on their knowledge and opinion. No indication as to which indicator should they used was given, hence
the ranking should be considered as an overall ranking, comprising not only research performance, but also
teaching and knowledge transfer. Having said this, as an exploratory exercise, the results should not be in
any case considered as definite.