Irene
=====

# In a nutshell
* Full irene documentation available [here](https://www-tgcc.ccc.cea.fr/docs/)
* Current project name on irene: **gen7725**
* Version to be used: release 18.07
* Compiler: intel18.0.3.222
* MPI library: openmpi2.0.4
* Run to be performed on **two nodes**
* Binary available in `/ccc/cont003/home/gen7725/gen7725/soft/metalwalls_bin/2018-07/mw_18.07`
* job example in `computers/irene/job` copied below
* Welcome message in run.out not mentioning release 18.07... To be fixed in the next one. If you see the following, you are using release 18.07
```
MetalWalls 00.00
commit ba6a698b71aee8356f3033ea12da15cbc68b2db0
```

# Job example
```shell
#!/bin/bash
#MSUB -r MW2
#MSUB -n 96         # Number of tasks to use  
#MSUB -N 2
#MSUB -T 100        # Elapsed time limit in seconds
#MSUB -o std.log    # Standard output. %I is the job id
#MSUB -e std.err    # Error output. %I is the job id 
#MSUB -A gen7725    # Project's quota used
#MSUB -q skylake    # skylake for thin nodes, knl for KNL nodes, hybrid for GPU partition, xlarge for fat nodes
#MSUB -m work


set -x

module purge 
module load intel/18.0.3.222 mpi/openmpi/2.0.4

time ccc_mprun /ccc/cont003/home/gen7725/gen7725/soft/metalwalls_bin/2018-07/mw_18.07
```

# Job example with dependancy
To access the mw_utils utility required for jobs with dependancy the user has to add the following line in his .bashrc file
```
module use /ccc/cont003/home/gen7725/gen7725/soft/modules
``` 


```shell
#!/bin/bash
#MSUB -r Relevant_Name
#MSUB -n 96         # Number of tasks to use  
#MSUB -N 2
#MSUB -T 100        # Elapsed time limit in seconds
#MSUB -o std.log    # Standard output. %I is the job id
#MSUB -e std.err    # Error output. %I is the job id 
#MSUB -A gen7725    # Project's quota used
#MSUB -q skylake    # skylake for thin nodes, knl for KNL nodes, hybrid for GPU partition, xlarge for fat nodes
#MSUB -m work
#MSUB -w            # Dependancy option: several jobs with the same name can be submitted at once. 
                    # They will be executed in the order they have been submitted and start only when 
                    # the previous job is finished


set -x

module purge 
module load intel/18.0.3.222 mpi/openmpi/2.0.4 mw_utils

# mw_utils will detect the latest run, create the directory for the new one and copy all files
sim_dir=`mw_utils restart`
cd $sim_dir
time ccc_mprun /ccc/cont003/home/gen7725/gen7725/soft/metalwalls_bin/2018-07/mw_18.07
```

This job file has to put at the top directory that contains all runXXX directories that contain a trajectory (and not inside a run directory as it is usually the case). Jobs have to be submitted from that directory also. But several jobs can be submitted at once 
. They will be executed in the order they have been submitted and will start only when 
the previous job is finished.




# Benchmarking
#### Compiler comparison 
10 steps of Blue energy test-case with 18.07 version on 2 nodes

|Compilers | Time(s)|
|:--------:|:------:|
|intel17.0.6.256 |6.163|
|intel18.0.1.163 |6.376|
|intel18.0.3.222 |5.823|
|gnu4.8.5 |16.393|

Compiler intel18.0.3.222 is the fastest one


#### MPI lib comparison on 4 nodes

|MPI_lib        | Time(s)|
|:-------------:|:------:|
|intelmpi2018.0.1.163 |26.73|
|intelmpi2018.0.3.222 |26.44|
|openmpi2.0.2 |26.73|
|openmpi2.0.4 |25.44|

The default MPI library openmpi2.0.4 is the fastest one.


#### Scalability 
100 steps of Blue energy test-case with 18.07 version. Two nodes seem to be the appropriate amount of resources to run in production. Maybe go to 4 nodes in case of rush.

|#Node |  Time(s) |  Speedup  |  Eff(%)|
|:----:|:--------:|:---------:|:------:|
|1 |    44.28 |     1.00 |    100.0|
|2 |    26.91 |     1.65 |     82.3|
|4 |    16.34 |     2.71 |     67.8|
|6 |    12.90 |     3.43 |     57.2|
|8 |    12.63 |     3.51 |     43.8|
|10 |    11.59 |     3.82 |     38.2|

# Some words on file systems
4 file systems @TGCC:
* `home`: small, backed up => source,  software and small data
* `work`: large and performant, **no back up** => target file system for MW production
* `store`: huge and slow (tapes), archiving system => pack regularly run results in tarball to limit the number of files there in order to back up data
* `scratch`: very large and performant, periodic remove of non used files => not relevant for MW production   

```
module switch dfldatadir/gen7725
```
sets for each of these file systems, environment variables for private and shared spaces. For instance, for the work file system:
* `ALL_CCCWORKDIR` points to the shared space /ccc/work/cont003/gen7725/gen7725 associated to project gen7725
* `CCCWORKDIR` points to the private space /ccc/work/cont003/gen7725/haefemat of user haefemat associated to project gen7725
=> good practise to load the correct module and use these env variables in your scripts

