Update of ESiWACE Application Software Framework prepared for the demonstrators (D3.2)
- 1. Max Planck Institute for Meteorology
- 2. Barcelona Supercomputing Center
- 3. The University of Reading
Description
Abstract
Software implementations of modern climate and numerical weather prediction models and their execution workflows require many pieces of software to be correctly installed and configured. Various mathematical, input/output, parallelisation and other software libraries are necessary to build and run executables of the model; plotting, statistics, file format conversion and other packages are required for post-processing; scripting language interpreters and filesystem utilities are essential for controlling different stages of the execution workflows and gluing them together. All these generate long lists of software dependencies that have to be resolved and accounted for during the deployment of the complete software stack required for numerical experimentation. This process is further complicated by the fact that the models are research tools and hence are under continuous development. It is often the case that introduction of changes in the model’s source code becomes an essential part of a research workflow. Thus, the software environment for the research workflow does not have to only fulfil the runtime requirements of the model execution workflow but also the requirements posed by software development needs. On top of that, given the resource requirements of some scientific projects, a single machine is not always enough to accommodate all required model runs. Thus, it becomes mandatory to make the software stack portable to whatever supercomputing environment becomes available for the numerical experiments.
This work aims to systematise requirements of a typical climate modelling workflow to the software environment, to identify common problems on the way to meeting them, and attempts to remedy the situation by providing recommendations on what solutions can be applied to make the preparations of the software environment for Earth system modelling easier.
There are no indications that the requirements to the software environments of the upcoming extreme scale HPC will decrease. On the contrary, the main challenges related to the increasing amount of input and output data of the models, workload balancing, and fault tolerance (see ESiWACE deliverable D3.8), will be exacerbated, requiring more and more sophisticated solutions, the implementation of which will inevitably lead to the further complications of the already difficult problem of the software dependencies resolution. At the same time, most experts assume that the basic concepts underlying the existing approaches to the software deployment and maintenance will be inherited in the next generation of HPC solutions. Thus, the investment into the development of the automatization tools for software installations promises long term benefits for the whole HPC community.
About this document
Work package in charge: WP3 Usability
Actual delivery date for this deliverable: 5 March 2018
Dissemination level: PU (for public use)
Lead author:
Max Planck Institute for Meteorology (MPI‐M), Sergey Kosukhin, Reinhard Budich
Other contributing partners:
Barcelona Supercomputing Center (BSC), Kim Serradell, Oriol Mula-Valls
The University of Reading (UREAD), Grenville Lister
Rewier:
German Climate Computing Center (DKRZ), Chiara Bearzotti
Notes
Files
D3.2-complete.pdf
Files
(1.0 MB)
Name | Size | Download all |
---|---|---|
md5:1f98906c466d2ebe5d0bed288bed880a
|
1.0 MB | Preview Download |