The effect of the scale of grant scoring on ranking accuracy

We provide 4 R scripts to generate the 4 figures in the paper.

Figure 1. Representation of the multiple threshold model for k = 5 categories.

The x-axis shows the unobserved continuous scale in standard deviation units and the y-axis the density. The position of each of the 4 thresholds is shown as a vertical red line.

Figure 2. Correlation between the observed categorical score and the underlying continuous score.

The x-axis is the number of discrete categorical scores (k) and the y-axis shows the correlation between the observed categorical score (Y) and the underlying continuous score (u). The red horizontal line denotes a correlation of 0.95.

Figure 3. Loss of information relative to scoring on a continuous scale.

Each panel shows the loss of information (Equation [3]) when scoring a finite number of categories relative to the continuous score, as a function of the number of assessors (panels a to d) and the proportion of variation in scores due to the quality of the grant (x-axis).

Figure 4. Loss of information induced by scoring too few grants in extreme categories. The x-axis is the number of discrete categorical scores (k) and the y-axis shows the correlation (R_k) between the observed categorical score (Y) and the underlying continuous score (u). The correlation R_k is calculated under three scenarios defined by the variance (s²_s) of the distribution of underlying scores. The grey horizontal line denotes a correlation of 0.95 or 0.99.