CleF - Climate Finder - A python based command line tool to query ESGF dataset hosted at NCI

CleF - Climate Finder is a dataset query tool developed by the CLEX CMS team, powered by ESGF and the NCI clef.nci.org.au database.

CleF searches the Earth System Grid Federation datasets stored at the Australian National Computational Infrastructure, both data published on the NCI ESGF node as well as files that are locally replicated from other ESGF nodes.

Currently it searches for the following datasets:

• CMIP5 in NCI projects: rr3, where NCI is the primary publisher and al33 for replicas
• CMIP6 in NCI projects: oi10 for replicas, fs38 where NCI is the primary publisher
• CORDEX (ESGF only) in NCI projects: rr3, where NCI is the primary publisher and al33 for replicas

The search returns both the path of data that is already available at NCI as well as information on data that is on external ESGF nodes but not yet available locally.

The search works like the ESGF search website, e.g. https://esgf.nci.org.au/search/esgf_nci. Results can be filtered by using flags matching the ESGF search facets.

Four optional flags are available for the cmip5 and cmip6 commands to change the output or submit a data request:

• clef --remote <dataset>  returns all the ESGF for the dataset matching the constraints, it is the equivalent of doing a search online on an ESGF node
• clef --local  <dataset> finds local files accessing directly the NCI’s clef.nci.org.au database so it will also return older versions or datasets that might be temporarily offline.
• clef --missing <dataset>  finds files on ESGF that haven’t been downloaded to NCI
• clef --request <dataset> create and pass to NCI a request to download the missing files

CleF is a python module, it can also be imported directly in your own python codes. It was designed specifically to work on the NCI servers and it is already installed and available in a shared conda environment on these.

This version (1.3.0) adds the cordex sub-command to query CORDEX data published on the ESGF.

For more information, tutorial and the full list of options check the Read the Docs documentation.

If you find a bug or have a comment on the code, please create an issue on GitHub.