Classification measures

List of classification measures provided in the toolbox

Several classification measures are provided in the toolbox in order to evaluate the classification performances (in fitting, validation and for external test set of samples). The classification measures are provided as MATLAB structure (class_param) and can be found in the results section of the GUI, as well as in the results obtained in the command window of MATLAB (see the description of the MATLAB routines). The classification measures can be analysed by typing, in the MATLAB command window:

when fitting the model, or


when validating the model.

The calculated classification measures are described in this publication: Ballabio, D., Grisoni, F., Todeschini, R. (2018). Multivariate comparison of classification performance measures. Chemometrics and Intelligent Laboratory Systems, 174, 33-44 [link]. Here, an overview is given:

the confusion matrix (conf_mat)
is a square matrix with dimensions G x G + 1, where G is the number of classes. Each element n_gk represent the number of samples belonging to class g and assigned to class k. The last column collects the number of samples not assigned.

precision
represents the capability of a classification model to not include samples of other classes in the considered class. It can be measured as the ratio between the samples of the g-th class correctly classified and the total number of samples assigned to that class:
where n'_g is the total number of samples assigned to the g-th class and n_gg is the number of samples belonging to class g and correctly assigned to class g.

sensitivity
describes the model ability to correctly recognize samples belonging to the g-th class and is defined as:
where n_g is the total number of samples belonging to the g-th class. Samples of the g-th class which were not assigned are not considered for the sensitivity calculation.

specificity
characterizes the ability of the g-th class to reject the samples of all the other classes and is defined as:
where n'_k is the total number of samples assigned to the k-th class:
Not assigned samples are not considered for the specificity calculation.

accuracy (ac)
is the ratio of correctly assigned samples:
where n is the total number of samples. Not assigned samples are not considered for the accuracy calculation.

ratio of not assigned samples (not_ass)
is the ratio of the samples that have not been recognized and classified in the modelled classes. It is calculated as the sum of the last column of the confusion matrix divided by the total number of samples. Not assigned samples are not considered for the calculation of sensitivity, specificity, error rate and non error rate.

non error rate (ner)
is the average of the class sensitivities:

error rate (er)
is defined as follows:
where NER is the non-error rate.


[-> top]


ROC curves

ROC (Receiver Operating Characteristics) curves are graphical tools for the analysis of classification results. ROC curves can be calculated a) only in the graphical interface of the Classification toolbox for MATLAB, b) only when PLSDA is used. A ROC curve is a graphical plot of sensitivity and 1-specificity, for a binary classification system as its discrimination threshold is changed.
A single value of sensitivity and 1-specificity can be calculated from a contingency table and consequently each contingency table represents a single point in the ROC space. For each threshold value, a classification rule is calculated and the respective contingency table is obtained. The best possible classification method would yield a point in the upper left corner of the ROC space, representing maximum sensitivity and specificity, while a random classification gives points along the diagonal line from the left bottom to the top right corners. Summarising, ROC curves are calculated for each class, separately, by changing the threshold of assignations. An example of ROC curves calculated on three classes is shown here. The area under the ROC curve (AUC) can be used as estimator of the class discrimination; it is shown in the plot title for each class.


[-> top]