Designing Machine Learning Experiments using SLURM within a Cloud Trusted Research Environment

Trusted Research Environments (TRE) are secure platforms which enable researchers to access and analyse personal data within the ‘Five Safes’ framework. TREs have been in use for many years to enable the safe use of sensitive data in research yet are not fully capable of supporting researcher needs including big data and high computational demands for machine learning (ML) workloads. Researchers are increasingly applying a range of machine learning (ML) algorithms on de-identified personal datasets derived from healthcare (for example: electronic health records, routinely collected medical scans and diagnosis information). TREs are looking to deploy on-demand high-performance computing (HPC) environments using third-party cloud computing providers that enable batch processing pipelines such as SLURM (Simple Linux Utility for Resource Management). Slurm is a open source, scalable scheduling tool for HPC environments which can be used to launch and monitor jobs on assigned nodes. This ‘on-demand’ approach allows for cost optimisation of the TRE resources in comparison to the ‘always on’ physical computing environment. Also, each researcher can be provisioned their own HPC environment, providing full data and network isolation from other projects, and able to grow dynamically depending on compute requirements

This poster discusses how ML experiments can be freshly designed (or retrofitted) using SLURM within restrictive TRE platform. It provides examples of optimised approaches to implement the different steps of a standard ML experiment using batch processing whilst using secure, shared data storage. It will highlight how the training runtimes can be drastically reduced by adopting some of these measures.