Published June 23, 2022 | Version v1
Conference paper Open

StaDRe and StaDRo: Reliability and Robustness Estimation of ML-based Forecasting using Statistical Distance Measures

  • 1. Fraunhofer Institute for Experimental Software Engineering, Germany
  • 2. University of Hull


Reliability estimation of Machine Learning (ML) models is becoming a crucial subject. This is particularly the case when such models are deployed in safety-critical applications, as the decisions based on model predictions can result in hazardous situations. As such, recent research has proposed methods to achieve safe, dependable and reliable ML systems. One such method is to detect and analyze distributional shift, and then measuring how such systems respond to these shifts. This was proposed in earlier work in SafeML. This work focuses on the use of SafeML for time series data, and on reliability and robustness estimation of ML-forecasting methods using statistical distance measures. To this end, distance measures based on the Empirical Cumulative Distribution Function (ECDF), proposed in SafeML, are explored to measure Statistical-Distance Dissimilarity (SDD) across time series. We then propose SDD-based Reliability Estimate (StaDRe) and SDD-based Robustness (StaDRo) measures. With the help of clustering technique, identification of similarity between statistical properties of data seen during training, and the forecasts is done. The proposed method is capable of providing a link between dataset SDD and Key Performance Indices (KPIs) of the ML models.



Files (1.8 MB)

Name Size Download all
410.1 kB Preview Download
1.4 MB Preview Download

Additional details

Related works

Is identical to
Conference paper: 10.48550/arXiv.2206.1111 (DOI)


BIECO – Building Trust in Ecosystems and Ecosystem Components 952702
European Commission