###################################################
#                                                 #
# Steps to create and run a coupled configuration #
# Example with BENGUELA_LR                        #
# Detailed documentation can be found here:       #
#                                                 #
# https://www.croco-ocean.org/documentation       #
#                                                 #
# Copyright (c) 2018 S. Jullien                   #
# swen.jullien@ifremer.fr                         #
#                                                 #
###################################################

This file contains the following information sections: 

-> Coupling_tools CONTENTS 
-> Get the model source codes
-> RUNNING the coupled model
    -> Create configuration file system
    -> Compiling
    -> Pre-processing
    -> Running


-----------------------------------------------
Coupling_tools CONTENTS
-----------------------------------------------

* create_config: script to create your configuration directory tree 
                 and copy all the following scripts and files in the appropriate dir. 

* scripts_run: 
    run_env :source script to define a set of useful environment variables for your run
             needs to be edited carefully when you start working on your configuration
             it is called at the begining of all other scritps!
    run_cpl: script to launch a coupled run  
    run_frc_croco  |
    run_frc_wrf    | scripts to launch each model in forced mode
    run_frc_ww3    |

* scripts_croco:
    readme_download_CFSR             | readme and script to download and pre-process 
    Process_CFSR_files_for_CROCO.sh  | CFSR file to be used for CROCO ONLINE interp BULK
    readme_preprocess                | readme and scripts to use croco_tools classic 
    crocotools_param.m               | pre-processing (in matlab)
    start.m                          |
    make_ww3_grd_input_files_from_croco_grd.m: script to generate coord. and bathy. file 
                                               for WW3 from croco_grd.nc file
    cppdefs.h.frc                    |
    cppdefs.h.frc.cfsr.online        | examples of cppdefs.h files for the different
    cppdefs.h.oa                     | forced or coupled cases. They will be linked in
    cppdefs.h.ow                     | cppdefs.h during compilation (make_CROCO_compil)
    cppdefs.h.owa                    |
    make_CROCO_compil : script to define some variables and launch CROCO jobcomp
    croco.in.base  : CROCO namelist file in which some stuff (timestep, etc) will be 
                     replaced by run_frc_croco or run_cpl scripts  

* scripts_ww3:
    make_WW3_compil : script to define some variables and launch WW3 compilation
    script_make_CFSR_wind_for_ww3.sh                |
    script_make_WRF_wind_for_ww3.sh                 | scripts to create wind, current,
    script_make_CROCO_current_and_level_for_ww3.sh  | and level input files for WW3
    UV2T.sh                                         |
    inputs_ww3:
        switch_aw_BENGUELA     |
        switch_ow_BENGUELA     | switches for the different modes
        switch_owa_BENGUELA    |
        switch_frc_BENGUELA    |
        ww3_grid.inp.base :    grid input file in which some stuff (timesteps, etc)
                               will be replaced by run_frc_ww3 or run_cpl scripts  
        ww3_prnc.inp.wind      |   
        ww3_prnc.inp.level     | prnc input file for prerpating ww3 input files
        ww3_prnc.inp.current   |
        ww3_strt.inp :         strt input file for running ww3_strt
        ww3_shel.inp.base.frc  |
        ww3_shel.inp.base.Afrc | shel input files for the different modes in which
        ww3_shel.inp.base.ow   | some stuff (dates, etc) will be replaced by 
        ww3_shel.inp.base.aw   | run_frc_ww3 or run_cpl scripts 
        ww3_shel.inp.base.owa  |
        ww3_ounf.inp.base :    ounf input file in which dates will be replaced 

* scripts_wrf:
    readme_download_CFSR_data  |
    readme_wps                 | some useful readme for WPS
    readme.Vtable              | 
    configure.namelist.wps_BENGUELA : configure file to edit for running WPS and real  
    run_wps.bash    :          script to run wps (wrf pre-processing)
    run_real.bash   :          script to run real (wrf pre-processing)
    make_WRF_compil :          script to compile wrf
    inputs_wrf:
        configure.wrf.coupled    | examples of configure files for compiling wrf
        configure.wrf.uncoupled  | in forced and coupled modes
        readme.Vtable             |
        Vtable.CFSR_sfc_flxf06    | Vtables for CFSR data
        Vtable.CFSR_press_pgbh06  |
        namelist.input.base.complete    : namelist base in which stuff will be replaced
                                          when running real
        namelist.input.prep.BENGUELA.aw : namelist example for a coupled AW run 
        README.namelist : readme to know all the namelist options available
        myoutfields.txt : example of file that can be prescribed in wrf namelist to 
                          add/remove variable outputs

* scripts_oasis:
    create_oasis_grids_for_wrf.sh : script to create grids.nc and masks.nc files for  
                                    OASIS for WRF(call the the oasis function not 
                                    implemented in WRF yet). Will be called in run_cpl
    create_oasis_restart_from_calm_conditions.sh          | scripts to create restart 
                                                          |files for OASIS from calm or
    create_oasis_restart_from_preexisting_output_files.sh | pre-existing model files. 
                                                          | Can be called in run_cpl.
    from_croco.sh         |
    from_ww3.sh           | functions called by the previous scripts for each model
    from_wrf.sh           |
    to_wrf_stag_grid.sh   |
    create_oasis_toy_files.sh: script to create files that will be used by the toy 
                               model to mimic another model
    inputs_oasis:
        make.ada_my   | example files for OASIS compilation on ADA and DATARMOR clusters
        make.DATARMOR |
        namcouple.base.aw or oa or owa or ow : namelist files for the different coupled 
                                               modes in which stuff will be replaced by
                                               run_cpl script  
        namcouple.base.aw.debug : example of a namelist file with debug options
        namcouple.base.aw.nointerp : example of namlist with given interpolation file
        namcouple.base.aw.toyatm : namelist files for the different coupled modes 
                                   in which stuff will be replaced by run_cpl script 
                                   when running with a toy model

* toy_model: toy model sources and namelists. This toy model can be used to test 
             coupling, it mimics a model by exchanging prescribed fields with oasis.
             For example if you are using CROCO and want to couple it with a wave 
             model, you should first try to couple it with the toy wave model. 

-----------------------------------------------
Get the model source codes
-----------------------------------------------
- CROCO: https://www.croco-ocean.org/download/croco-project/
- WRF: http://www2.mmm.ucar.edu/wrf/users/
- WW3: https://forge.ifremer.fr/projects/ww3/
- OASIS3-MCT: https://verc.enes.org/oasis/download/oasis-registration-form


-----------------------------------------------
RUNNING the coupled model
-----------------------------------------------

* Create configuration file system
  --------------------------------
- Be sure that you work in a bash environment (if not just do: bash)
- Edit and run create_config
- Edit and source run_env

* Compiling
  ---------
- First, compile OASIS:
        cd $HOME/oasis/OASIS3-MCT_3.0_branch/oasis3-mct/util/make_dir/
        Edit your make.YOURMACHINE file for paths, libraries, compilers and options
        Edit make.inc to include your make.YOURMACHINE
        make realclean -f TopMakefileOasis3
        make -f TopMakefileOasis3 > oasis_make.out
        Check in your oasis compile directory if all the libraries are present: 
             libmct.a  libmpeu.a  libpsmile.MPI1.a  libscrip.a

- WW3 compilation:
        Edit and run make_WW3_compil (in ww3_in)
        WW3 has to be compiled again if you want to change physical/dynamical switches

- CROCO compilation: this should be done, once CROCO config is set up...
        Edit and run make_CROCO_compil (in croco_in)
        CROCO has to be compiled again if you change physical/dynamical options 
        in cppdefs.h or domain or parallelization settings in param.h

- WRF compilation: 
        go in your WRF source dir and execute ./configure.wrf
        choose distributed memory option associated to your machine
        cp configure.wrf configure.wrf.uncoupled
        then set your configure for coupled case: 
                cp configure.wrf configure.wrf.coupled
                Edit configure.wrf.coupled by adding:
                    OA3MCT_ROOT_DIR = $(OASISDIR)
                    ARCH_LOCAL : for OASIS add -Dkey_cpp_oasis3
                    INCLUDE_MODULES : add before netcdf include:  
                        -I$(OA3MCT_ROOT_DIR)/build/lib/mct \
                        -I$(OA3MCT_ROOT_DIR)/build/lib/psmile.MPI1 \
                    LIB_EXTERNAL : add before netcdf library:     
                        -L$(OA3MCT_ROOT_DIR)/lib -lpsmile.MPI1 -lmct -lmpeu -lscrip \
        then go back in your config dir ($hconf) and go in wrf_in: set make_WRF_compil
        create a job in which you can execute make_WRF_compil (WRF is long to compile 
         (~1h or more) and takes a lot of memory...)

* Pre-processing
  --------------

- WW3:
        - set and run script_make_CFSR_wind_for_ww3.sh
        - use GRIDGEN tool made for WW3 : 
          ftp://ftp.ifremer.fr/ifremer/ww3/COURS/WAVES_SHORT_COURSE/TOOLS/GRIDGEN/
          or to create an equivalent grid to CROCO grid use : 
          make_ww3_grd_input_files_from_croco_grd.m available in croco_in after 
          having run CROCO pre-processing

- CROCO: 
        - If you are using CFSR online interpolation, prepare your surface forcing 
          with Process_CFSR_files_for_CROCO.sh. Link your files to $CROCO_FILES_DIR
        - Matlab pre-processing for grid, forcing, ini, boundaries, etc: 
            First set crocotools_param.m and start.m, then in matlab : 
            start
            make_grid
            make_ini
            make_forcing
            make_bry
            make_ww3_grd_input_files_from_croco_grd

- WRF: Follow readme_wps. Note: you will need to have WPS compiled before.

- OASIS:
        - eventually (otherwise they can be created in the run_cpl script) create 
          restart files: from calm or pre-exising conditions
        - eventually (otherwise they can be created in the run_cpl script) create 
          grids for wrf

* Running
  -------

** FORCED modes
   ------------
        - CROCO: edit run_frc_croco and submit a job that launch: 
         ./run_frc_croco NB_PROCS (where NB_PROCS is the number of CPUs as in param.h)
        
        - WW3: edit run_frc_ww3 and submit a job that launch:
         ./run_frc_croco NB_PROCS (where NB_PROCS is the number of CPUs you choose)


** COUPLED modes
   -------------
        - edit run_cpl and submit a job that launch:
         ./run_frc NB_PROCS1 NB_PROCS2 NB_PROCS3 (where NB_PROCS1 = nb of CPUs for WRF
                                                        NB_PROCS2 = nb of CPUs for WW3
                                                        NB_PROCS3 = nb of CPUs for CROCO
                                        NB_PROCX can be set to 0 is the model is unused)  

