The effect of the scale of grant scoring on ranking accuracy
Description
We provide 4 R scripts to generate the 4 figures in the paper.
Figure 1. Representation of the multiple threshold model for k = 5 categories.
The x-axis shows the unobserved continuous scale in standard deviation units and the y-axis the density. The position of each of the 4 thresholds is shown as a vertical red line.
Figure 2. Correlation between the observed categorical score and the underlying continuous score.
The x-axis is the number of discrete categorical scores (k) and the y-axis shows the correlation between the observed categorical score (Y) and the underlying continuous score (u). The red horizontal line denotes a correlation of 0.95.
Figure 3. Loss of information relative to scoring on a continuous scale.
Each panel shows the loss of information (Equation [3]) when scoring a finite number of categories relative to the continuous score, as a function of the number of assessors (panels a to d) and the proportion of variation in scores due to the quality of the grant (x-axis).
Figure 4. Loss of information induced by scoring too few grants in extreme categories. The x-axis is the number of discrete categorical scores (k) and the y-axis shows the correlation (Rk) between the observed categorical score (Y) and the underlying continuous score (u). The correlation Rk is calculated under three scenarios defined by the variance (s2s) of the distribution of underlying scores. The grey horizontal line denotes a correlation of 0.95 or 0.99.