Occigen
=======

# In a nutshell
* Full occigen documentation available [here](https://www.cines.fr/calcul/materiels/occigen/)
* Current project name on irene: **pec7725**
* Version to be used: release 18.07
```
commit 7f1ffe228738405cf63fecfe6820165bc173b825
```

# 1 - Using the BDW partiton
* Compiler: intel/16.4.258
* MPI library: intelmpi/5.1.3.258
* Run to be performed on **four nodes**
* job example in `computers/occigen/job.slurm_BDW` copied below

# Job example
```shell
#!/bin/bash
#SBATCH -J benchmark
#SBATCH --constraint BDW28
#SBATCH --nodes 4
#SBATCH --ntasks-per-node 28
#SBATCH -e std.err                   # Error output. %I is the job id
#SBATCH -o std.log                   # Standard output. %I is the job id
#SBATCH -t 00:05:00                  # Elapsed time limit in seconds
echo $SLURM_NODELIST > nodelist
module purge
module load intel/16.4.258 intelmpi/5.1.3.258
env > run_env
srun --mpi=pmi2 -K1 -n $SLURM_NTASKS /home/nganfoud/mw2/mw
```
# Job with dependency example
For this you need to install first the mw_utils script that enables to create a restart directory:
```
cd scripts/mw_utils
python setup.py install --user
```
Add the following line in your $HOME/.profile file
```
export PATH=$PATH:$HOME/.local/bin
```

Then use the following script to launch your job. This script should not be located within the run directory but one level above such that the run directory can be created.  

```shell
#!/bin/bash
#SBATCH -J benchmark
#SBATCH --constraint HSW24
#SBATCH --nodes 4
#SBATCH --ntasks-per-node 24
#SBATCH -e std.err
#SBATCH -o std.log
#SBATCH -t 00:20:00

module purge
module load intel/16.4.258 intelmpi/2017.0.098

sim_dir=`mw_utils restart`
cd $sim_dir
echo $SLURM_NODELIST > nodelist
env > env_run

srun --mpi=pmi2 -K1 -n $SLURM_NTASKS /home/haefele/dev/mw2/mw

mv ../std.* .
```

Several jobs that depend among each others could be then submitted at once. Here is the submission of two jobs in a row. The first job is running while the second is pending due to dependency reason:
```
haefele@login2:~/runs/blue-energy_nidhal$ sbatch job.slurm 
Submitted batch job 5845217
haefele@login2:~/runs/blue-energy_nidhal$ sbatch --depend=afterany:5845217 job.slurm 
Submitted batch job 5845220
haefele@login2:~/runs/blue-energy_nidhal$ sq
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           5845220      allc benchmar  haefele PD       0:00      8 (Dependency)
           5845217      allc benchmar  haefele  R       0:12      8 n[1129-1130,1399,1401-1405]
```
Several keywords explained [here](https://slurm.schedmd.com/job_array.html) can be used to describe the kind of dependency you want. `afterany` waits only for the end of the job, successful or not. `afterok` waits for the end of the job and the job must be successful.

# Benchmarking
#### Compiler comparison 
10 steps of Blue energy test-case with 18.07 version on 2 nodes

|Compilers | Time(s)|
|:--------:|:------:|
|intel/15.6.233 |2.94183E+01|
|intel/16.4.258 |7.73117E+00|
|intel/17.0 |8.34772E+00|
|intel/17.1 |8.05659E+00|
|intel/17.2 |8.59797E+00|
|intel/18.0 |7.80357E+00|
|intel/18.1 |7.80353E+00|

Compiler intel/16.4.258 is the fastest one


#### MPI lib comparison on 4 nodes

|MPI_lib        | Time(s)|
|:-------------:|:------:|
|intelmpi/5.1.3.258 |4.65219E+00|
|intelmpi/2017.0.098 |4.80014E+00|
|intelmpi/2017.1.132 |4.70121E+00|
|intelmpi/2017.2.174 |4.67436E+00|
|intelmpi/2018.1.163 |5.22213E+00|
|openmpi/intel_2.0.0 |4.75916E+00|
|openmpi/intel_2.0.1 |5.11469E+00|
|openmpi/intel_2.0.2 |4.79311E+00|
|openmpi/intel_2.0.4 |4.80401E+00|

The library intelmpi/5.1.3.258 is the fastest one.


#### Scalability 
100 steps of Blue energy test-case with 18.07 version. 4 nodes seem to be the appropriate amount of resources to run in production. Maybe go to 6 nodes in case of rush.

|#Nodes |  Time(s) |  Speedup  |  Eff(%)|
|:----:|:--------:|:---------:|:------:|
|1 |   121.82 |     1.00 |    100.0|
|2 |    66.85 |     1.82 |     91.1|
|4 |    39.73 |     3.07 |     76.7|
|6 |    28.39 |     4.29 |     71.5|
|8 |    23.86 |     5.10 |     63.8|
|10 |    20.76 |     5.87 |     58.7|
|12 |    19.28 |     6.32 |     52.7|



# 2 - Using the HSW partiton
* Compiler: intel/16.4.258
* MPI library: intelmpi/2017.0.098
* Run to be performed on **four nodes**
* job example in `computers/occigen/job.slurm_HSW` copied below

# Job example
Very similar to the one above, only the line:
```shell
#SBATCH --constraint BDW28
#SBATCH --ntasks-per-node 28
module load intel/16.4.258 intelmpi/5.1.3.258
```
should be replaced by
```shell
#SBATCH --constraint HSW24
#SBATCH --ntasks-per-node 24
module load intel/16.4.258 intelmpi/2017.0.098
```

# Benchmarking
#### Compiler comparison 
10 steps of Blue energy test-case with 18.07 version on 2 nodes

|Compilers | Time(s)|
|:--------:|:------:|
|intel/15.6.233 |3.88620E+01|
|intel/16.4.258 |9.72410E+00|
|intel/16.8.266 |9.84837E+00|
|intel/17.0 |1.03389E+01|
|intel/17.1 |1.00830E+01|
|intel/17.2 |1.00221E+01|
|intel/18.0 |9.84669E+00|
|intel/18.1 |9.73797E+00|

Compiler intel/16.4.258 is the fastest one


#### MPI lib comparison on 4 nodes

|MPI_lib        | Time(s)|
|:-------------:|:------:|
|intelmpi/5.1.3.258 |5.67041E+00|
|intelmpi/2017.0.098 |5.65752E+00|
|intelmpi/2017.1.132 |5.66275E+00|
|intelmpi/2017.2.174 |5.69706E+00|
|intelmpi/2018.1.163 |5.74000E+00|
|openmpi/intel_2.0.0 |5.65877E+00|
|openmpi/intel_2.0.1 |6.08363E+00|
|openmpi/intel_2.0.2 |5.78514E+00|
|openmpi/intel_2.0.4 |5.66331E+00|

The library intelmpi/2017.0.098 is the fastest one.


#### Scalability 
100 steps of Blue energy test-case with 18.07 version. 4 nodes seem to be the appropriate amount of resources to run in production. Maybe go to 6 nodes in case of rush.

|#Nodes |  Time(s) |  Speedup  |  Eff(%)|
|:----:|:--------:|:---------:|:------:|
|1 |   153.83 |     1.00 |    100.0|
|2 |    84.95 |     1.81 |     90.5|
|4 |    48.11 |     3.20 |     79.9|
|6 |    35.31 |     4.36 |     72.6|
|8 |    29.02 |     5.30 |     66.3|
|10 |    24.63 |     6.25 |     62.5|
|12 |    23.45 |     6.56 |     54.7|
