D3.3 Assessment of regulatory requirements for patient level decision making

The EHDEN network has the potential to support the development of externally validated clinical prediction models at high speed and in a transparent manner. Under the incoming Medical Device Regulations in Europe (2017/745), clinical prediction models that are used to inform further testing or treatment decisions for patients are likely to be classified as medium to high risk medical devices and be regulated accordingly. The clinical evidence required to demonstrate performance and safety over a device’s lifetime and the intensity of post-market surveillance are greater than in the outgoing directives. This, in turn, is likely to lead to greater scrutiny by health technology assessment bodies and has prompted the development of evidence standards for the demonstration of clinical benefit. The development and use of such models must also adhere to other European and national regulations such as the General Data Protection Regulation.

To ensure that high-quality clinical prediction models developed through the EHDEN network can usefully inform clinical practice, it is essential that developers take into account these changing regulatory requirements. In this report we provide an overview of the incoming regulations and evidence standards for clinical algorithms, as well as best practice methods for clinical prediction model development. We discuss the challenges that this will pose for those developing clinical prediction models on the EHDEN platform and provide guidance for the future development of such models.

Models developed on the EHDEN network using OHDSI tools are well placed to provide substantial evidence on a model’s performance both in-sample and in one or more external datasets and to do so in a transparent manner following the principles of open science. However, it is also important to demonstrate clinical effectiveness against usual practice, ideally through a randomised controlled trial. Developers must also establish mechanisms to collect data on the real-world use of their clinical algorithms to support post-market surveillance and vigilance activities.

Some commentators have argued that European regulations require decisions made based on clinical algorithms to be explainable (conceptualisations of which differ), which may impede the use of certain machine learning methods. However, there remains considerable debate about to what extent European regulations demand this. Others argue that demonstrable performance benefits are of utmost importance.

It is also essential that those developing prediction models engage with healthcare practitioners, managers, and patients to ensure that any model developed is acceptable to clinicians and patients and can be integrated within existing clinical software systems. Finally, ethical concerns (e.g. due to unfair discrimination) need to be placed at the forefront of algorithmic development and application.