Machine Learning for Probabilistic Prediction

Prediction is the key objective of many machine learning applications. Accurate, reliable and robust predictions are essential for optimal and fair decisions by downstream components of artificial intelligence systems, especially in high-stakes applications, such as personalised health, self-driving cars, finance, new drug development, forecasting of election outcomes and pandemics. Many modern machine learning algorithms output overconfident predictions, resulting in incorrect decisions and technology acceptance issues. Classical calibration methods rely on artificial assumptions and often result in overfitting, whilst modern calibration methods attempt to solve calibration issues by modifying components of black-box deep learning systems. While this provides a partial solution, such modifications do not provide mathematical guarantees of prediction validity and are intrusive, complex, and costly to implement. This thesis introduces novel methods for producing well-calibrated probabilistic predictions for machine learning classification and regression problems. A new method for multi-class classification problems is developed and compared to traditional calibration approaches. In the regression setting, the thesis develops novel methods for probabilistic regression to derive predictive distribution functions that are valid under a nonparametric IID assumption in terms of guaranteed coverage and contain more information when compared to classical conformal prediction methods whilst improving computational efficiency. Experimental studies of the methods introduced in this thesis demonstrate advantages with regard to state-of-the-art. The main advantage of split conformal predictive systems is their guaranteed validity, whilst cross-conformal predictive systems enjoy higher predictive efficiency and empirical validity in the absence of excess randomisation.