The GenePattern Gateway for Genomic Medicine: Containerized cloud hybrid computing for biologists

Over the last two decades, research in biology has become increasingly demanding of compute resources. For example, gene expression analysis has evolved from single gene assays containing a small number of values, to single-cell RNA sequencing experiments producing matrices representing millions of individual cells, each with tens of thousands of transcript expression values. Biological researchers with many years of study in their fields now need access to high performance compute (HPC) clusters to analyze their data. However the use and programming of HPC systems is a specialized skill that requires training outside the scope of what most biologists receive.

GenePattern, www.genepattern.org, is a gateway providing implicit access to HPC systems for biologists and other non-programming scientists. It includes access to hundreds of tools for the analysis and visualization of multiple genomic data types. GenePattern has a web-based interface to provide easy access to these tools and allows the creation of multi-step analysis pipelines that enable reproducible in silico research. In recognition of the success of the electronic notebook, we have also released the GenePattern Notebook Environment, which extends the power of GenePattern in a Jupyter-based environment. The publicly available GenePattern servers currently support thousands of users and up to tens of thousands of analyses each month. 

To handle the scope and scale of modern genomic analyses, the GenePattern gateway uses a cloud hybrid model where analyses are distributed to public cloud HPC systems such as AWS Batch as well as academic HPC clusters such as the Expanse cluster at the San Diego Supercomputer Center. Management of jobs and the distribution of jobs to HPC are handled by the GenePattern server, allowing the biologists to focus on their data and its analysis. 

To manage configuration of the analyses across disparate HPC systems, GenePattern has embraced container-based systems. This has allowed GenePattern to support over 250 different analyses written in multiple programming languages. Here again GenePattern uses a hybrid model where different container systems including Docker and Singularity can be used, and in some case