Data archive for paper "Machine Learning Emulation of 3D Cloud Radiative Effects"

Overview

This is the data archive for the paper "Machine Learning Emulation of 3D Cloud Radiative Effects". It contains the paper’s data archive with all model outputs (see the results folder) as well as theSingularity image.

For the development repository (i.e. only containing source files without generated artefacts), please see https://github.com/dmey/ml-3d-cre.

For the Python tool to generate the synthetic data, please refer to the Synthia repository.

Requirements

• Linux
• Singularity >= 3
• Portable Batch System (PBS) job scheduler*

*Although PBS in not a strict requirement, it is required to run all helper scripts as included in this repository. Please note that depending on your specific system settings and resource availability, you may need to modify PBS parameters at the top of submit scripts stored in the hpc directory (e.g. #PBS -lwalltime=24:00:00).

Initialization

Deflate the data archive with:

./init.sh

Build the Singularity image with:

singularity build --remote tools/singularity/image.sif tools/singularity/image.def

Compile ecRad with Singularity:

./tools/singularity/compile_ecrad.sh

Usage

To reproduce the results as described in the paper, run the following commands from the hpc folder:

qsub -v JOB_NAME=mlp_synthia ./submit_grid_search_synthia.sh
qsub -v JOB_NAME=mlp_default ./submit_grid_search_default.sh
qsub submit_benchmark.sh

then, to plot stats and identify notebooks run:

qsub submit_stats.sh

Local development

For local development, notebooks can be run independently. To install the required dependencies, run the following through Anaconda conda env create -f environment.yml. Then, to activate the environment use conda activate radiation. For ecRad, the list of system dependencies are listed in tools\singularity\image.def and can be run with tools\singularity\compile_ecrad.sh.

Licence

Paper code released under the MIT license. Data released under CC BY 4.0. ecRad released under the Apache 2.0 license.