 _     ____     _           _       ____         ___
| |   |  _ \   | |_ __ ___ | |     | ___|       / _ \
| |   | |_) |  | | '_ ` _ \| |     |___ \      | (_) |
| |___|  __/ |_| | | | | | | |___   ___) |  _   \__, |
|_____|_|   \___/|_| |_| |_|_____| |____/  (_)    /_/  Version 5.9.18

Installation of LPJmL C Version 5.9.18

Installation for Unix-like systems
----------------------------------

This text describes how to install LPJmL on your system.
The code has been tested for AIX, Linux, Mac OS X, and Windows
with cygwin and Microsoft C compiler (sequential version without MPI).
Cygwin is a Linux-like environment for Windows (http://www.cygwin.com)
On Mac OS the xcode suite must be installed in order to have the gcc
compiler.

create LPJmL directory:

mkdir lpjml

Go to LPJmL directory:

cd lpjml

untar source files

gzip -cd lpjml-5.9.18.tar.gz|tar -xf -

Directory tree after extracting the tar files:

--lpjml
  |
  +-config        : OS- and compiler-specific Makefiles
  |
  +-bin           : Directory for executables and shell scripts
  |
  +-man           : Manual pages root directory
  | |
  | +-man1        : Manual pages for programs and scripts
  | |
  | +-man3        : Manual pages for functions
  | |
  | +-man5        : Manual pages for file formats
  |
  +--par          : Parameter files for PFTs and soils
  |
  +--output       : output data (created by make test)
  |
  +--restart      : restart data (created by make test)
  |
  +--lib          : library files (created by make)
  |
  +--include      : include files
  |
  +--src          : source tree
     |
     +--numeric   : source for numerical routines
     |
     +--tools     : source for tools
     |
     +--pnet      : source for library for distributed networks
     |
     +--climate   : climate code
     |
     +--lpj       : lpj functions
     |
     +--soil      : soil code
     |
     +--grass     : grass PFT code
     |
     +--tree      : tree PFT code
     |
     +--crop      : crop PFT code
     |
     +--landuse   : land use code
     |
     +--spitfire  : spitfire fire code
     |
     +--reservoir : reservoir code
     |
     +--socket    : socket communication library
     |
     +--coupler   : coupler code
     |
     +--image     : coupler to IMAGE model
     |
     +--netcdf    : NetCDF input/output functions
     |
     +--utils     : utility programs
     |
     +--test      : unit tests for src code using ceedling

Configure for your operating system

On the HLRS2015 cluster at PIK you have to set several modules for compilation and linking:

module load gmp/4.3.2
module load mpi/intel/5.1.1
module load compiler/intel/15.0.3
module load compiler/gnu/4.9.2
module load netcdf-c/4.3.3.1/serial udunits/2.2.19 expat/2.1.0

Or you can simply load the lpjml module:

module load lpjml

On the new PIK cluster you have to set the following modules:

module use /p/system/modulefiles/compiler \
               /p/system/modulefiles/gpu \
               /p/system/modulefiles/libraries \
               /p/system/modulefiles/parallel \
               /p/system/modulefiles/tools

module load intel/oneAPI/2024.0.0
module load udunits/2.2.28
module load json-c/0.17
module load openssl/3.0.12
module load netcdf-c
module load curl/8.4.0
module load expat/2.5.0

On a Linux system you have to install the NetCDF, Udunits2, and JSON-C library:

sudo apt-get install libnetcdf-dev
sudo apt-get install libudunits2-dev
sudo apt-get install libjson-c-dev
and optionally the MPICH parallel library:
sudo apt-get install mpich

Then LPJmL can be configured via

./configure.sh

If configure script exits with message "Unsupported operating system",
Makefile.$osname is created from Makefile.gcc and probably has to be
modified for your operating system/compiler.
If the configure script finds a MPI environment a parallel version of lpjml is built.
The configure script creates a copy of the following OS-specific makefiles from
directory config:

Makefile.aix         - IBM AIX settings (xlc compiler)
Makefile.aix_mpi     - IBM AIX and MPI environment
Makefile.gcc         - GNU C-compiler settings
Makefile.darwin_gcc  - GNU C-compiler settings for MacOS X
Makefile.intel       - Intel C-compiler (icc) settings
Makefile.icx         - New Intel C-compiler (icx) settings
Makefile.cluster2015 - Intel C-compiler and Intel MPI on HLRS2015 PIK cluster
Makefile.hpc2024     - New Intel C-compiler and Intel MPI on the new PIK cluster
Makefile.mpich       - GNU C-Compiler and MPI Chameleon settings
Makefile.win32       - Windows settings (used by configure.bat)

Compilation flags

Compilation of LPJmL is customized by definition of macros in the LPJFLAGS
section of Makefile.inc:

LPJFLAGS= -Dflag1 ...

Flag                Description
------------------- ------------------------------------------------------------
CHECK_BOUNDARY      check array boundaries of output
COUPLING_WITH_FMS   enable coupling to FMS
DAILY_ESTABLISHMENT enable daily establishment
DEBUG               diagnostic output is generated for debugging purposes
DEBUG_COUPLER       diagnostic output is generated for the coupler
DEBUG_IMAGE         diagnostic output is generated for the IMAGE coupler
DEBUG_N             diagnostic output is generated for the nitrogen cycle
IMAGE               include coupler to IMAGE model
LINEAR_DECAY        use linearized functions for litter decay
MICRO_HEATING       enable microbial heating
NO_FAIL_BALANCE     lpjml does not terminate on balance errors
PERMUTE             random permutation of PFT list
SAFE                code is compiled with additional checks
STRICT_JSON         strict JSON checking
USE_MPI             compile parallel version of LPJmL
USE_NETCDF          enable NetCDF input/output
USE_RAND48          use drand48() random number generator
USE_UDUNITS         enable unit conversion in NetCDF files
USE_TIMING          enable timing for socket I/O
WITH_FIRE_MOISTURE  enable moisture dependent fire emissions
WITH_FPE            floating point exceptions are enabled for debugging purposes
------------------- ------------------------------------------------------------

Create executables:

make

One executable in directory bin is created:

lpjml    - LPJmL simulation code

make lpjcheck

creates lpjcheck utility to check JSON configuration files

All utility programs are compiled by

make utils

The following programs are created in the bin directory:

adddrain     - add river basin to coordinate file
addheader    - add CLM header to binary file
arr2clm      - convert 2-D float arrays into CLM file
asc2clm      - convert ARC/Info ASCII files to CLM data files for LPJmL
backtrace    - print backtrace from core
bin2cdf      - convert binary output files into NetCDF files
binsum       - aggregate daily and monthly binary output
cat2bsq      - concatenate output files from distributed LPJmL simulations
catclm       - concatenate LPJmL climate data files
cdf2bin      - convert NetCDF files into raw binary data
cdf2clm      - convert NetCDF files into CLM files
cdf2coord    - extract CLM grid file from NetCDF file
cdf2grid     - convert grid NetCDF file into grid CLM or raw file
cdf2soil     - convert NetCDF file into binary file
cft2clm      - convert binary landuse data files to CLM data files for LPJmL
clm2cdf      - convert CLM files into NetCDF files
cmpbin       - compare two binary output files
country2cdf  - convert country/region code file into NetCDF file
cru2clm      - convert CRU data into file format suitable for LPJmL.
               CRU data files have to be in the format specified in
               http://www.cru.uea.ac.uk/~timm/grid/CRU_TS_2_1.html.
cutclm       - cut LPJmL climate data files
cvrtclm      - convert version of CLM file to version 3 or 4
drainage2cdf - convert CLM drainage file into NetCDF file
getcellindex - get cell index from latitude, longitude values
grid2clm     - convert grid data file to CLM data files for LPJmL
headersize   - print header size of CLM files
lpjcat       - concatenate restart files from distributed LPJmL simulations.
lpjcheck     - check syntax of LPJmL configuration files
lpjfiles     - print list of input/output files of LPJmL
lpjprint     - print contents of restart file of LPJmL model runs
printclm     - print contents of CLM files for LPJmL
output_bsq   - concatenate all output files from distributed LPJmL simulations
manage2js    - convert management  *.par files to JSON files
mathclm      - perform mathematical operations with CLM files
mergeclm     - merge LPJmL climate data files
printglobal  - print global values for raw binary output files
printharvest - print global harvested carbon for crops and managed grasslands
regridclm    - regrid CLM file to new grid
regridlpj    - regrid input data set for selected countries or specified grid file
regridsoil   - regrid soil file to new grid
regridsoil   - regrid binary soil file to different grid file
setclm       - set value in header of CLM files for LPJmL
soil2cdf     - convert binary files into NetCDF files
statclm      - print statistics of clm files
txt2grid     - convert text files to CLM grid data files for LPJmL
txt2clm      - convert text files to CLM data files for LPJmL

The binaries can be copied in an installation directory by invoking

make install

The necessary parameter and include files will be copied, too.
The target directory must be specified by setting the environment variable LPJROOT:

export LPJROOT=<installation directory>


Environment variables

The following environment variables are used by LPJmL:

LPJPREP        - defines preprocessor command for LPJmL configuration  file, default
                 is "cpp"
LPJROOT        - defines the root directory for LPJmL. This directory is added to the
                 include directories of the preprocessor
LPJIMAGE       - sets host where IMAGE is running
LPJCOUPLEDHOST - sets host where the coupled model is running
LPJWAIT        - sets time to wait for connection to IMAGE/coupled model
LPJINPATH      - Path append to the input filenames. Only done for filenames
                 without absolute path.
LPJRESTARTPATH - Path append to the restart filenames. Only done for filenames
                 without absolute path.
LPJOPTIONS     - cpp runtime options for LPJmL
LPJOUTPATH     - Path appended to the output filenames. Only done for filenames
                 without absolute path.

Create output and restart directory:

make test

On the compute cluster at PIK it is better to create symbolic links of the
input, output, and restart directory to the parallel GPFS filesystem /scratch.
This improves performance in particular for the parallel code.

Man pages
---------

Manual pages for the man command are located in the $LPJROOT/man directory. Setting the
environment variable MANPATH to:

export MANPATH=$MANPATH:$LPJROOT/man

will enable help.

Invoking

apropos lpj

yields list of man pages related to LPJmL.

Running the code
----------------

Sequential version can be started by invoking:

./bin/lpjml lpjml_config.cjson

If you run lpjml outside the root directory of LPJmL, the following environment
variable should be set:

export LPJROOT=$HOME/lpjml

Then all includes are found. You can include the bin directory in the search path:

export PATH=$PATH:$LPJROOT/bin

Script lpj_paths.sh is defined in bin setting all environment variables. Put

. <Path to LPJmL root dir>/lpj_paths.sh

in your profile. Then LPJROOT, PATH and MANPATH are set accordingly.

Runtime options of lpjml

The following runtime options are defined:

-Iincludepath      - Adds include path for LPJmL configuration file
-Dmacro[=value]    - Defines macro
-h                 - print usage of lpjml
-l                 - print license
-v                 - print compiler used and LPJmL flags set
-vv                - verbosely print the actual values during reading of the
                     configuration files
-pedantic          - stops on warnings\n"
-ofiles            - list only all available output variables
-param             - print LPJmL parameter for soils and PFTs
-nopp              - disable preprocessing
-pp cmd            - set preprocessor program. Default is 'cpp'
-fpe               - enable floating point exceptions
-image host        - set host where IMAGE model is running. Option is only
                    available for the IMAGE version
-couple host[:port]- set host and port where the coupled model is running
-wait time         - set time to wait for connection to IMAGE/coupled model
-inpath dir        - input directory path
-outpath dir       - output directory path
-restartpath dir   - restart directory path

The first two options will be sent to the preprocessor.

For the parallel version SLURM and LoadLeveler files for Linux and AIX are provided.
A job can be submitted by invoking the bin/lpjsubmit script:

bin/lpjsubmit ntasks LPJmL_args...

Depending on your MPI version installed the program can be started interactively:

mpirun -np 32 ./bin/lpjml lpjml_config.cjson

Error codes
-----------

If lpjml fails an error message is displayed in the following format

ERRORxxx: message

where xxx is the error code returned. The following error codes are defined:

Error code Description                                   Error type
---------- --------------------------------------------- ----------
         1 Error reading configuration                   External
         2 Error initializing input data                 External
         3 Error initializing grid                       External
         4 Invalid carbon balance                        Internal
         5 Invalid water balance                         Internal
         6 Negative discharge                            Internal
         7 Negative fire probability                     Internal
         8 Negative soil moisture                        Internal
         9 Error allocating memory                       External
        10 Negative stand fraction                       Internal
        11 Stand fraction sum error                      Internal
        12 List is empty in dellistitem                  Internal
        13 Index out of range in dellistitem             Internal
        15 Invalid year in getco2()                      External
        16 Crop fraction >1                              Internal
        17 No natural stand for deforest                 Internal
        18 Wrong cultivation type                        Internal
        19 Floating point error                          Internal
        21 PFT list is not empty in setaside             Internal
        22 Negative establishment rate                   Internal
        23 Output channel is broken in socket connection External
        24 Sending data to IMAGE model failed            External
        25 Opening connection to IMAGE model failed      External
        26 Not enough setaside stand created             Internal
        27 Forest left after deforestation               Internal
        28 Outflow reservoir error                       Internal
        29 Permafrost error                              Internal
        30 Global waterbalance error                     Internal
        31 Cannot store climate data                     External
        32 NO FMS coupler                                External
        33 Cannot initialize soil temperature            External
        34 Invalid radiation module                      Internal
        35 Negative soil NO3                             Internal
        36 Negative soil NH4                             Internal
        37 Invalid nitrogen balance                      Internal
        38 Invalid climate data                          External
        39 Invalid FPC value                             Internal
        40 Boundary check error                          Internal
        41 Invalid time step in soiltemp()               Internal
        42 Cannot read external flow                     External
        43 Opening connection to coupled model failed    External
        44 Invalid uniform temp sign value               Internal
        45 Error initializing output data                External
        46 Invalid crop PHU option                       Internal
        47 Invalid fire danger index                     Internal
---------- --------------------------------------------- ----------

External errors are caused by invalid or missing input files while internal
errors are caused by problems inside the LPJ code. The latter will cause a core
dump and have to be fixed by program changes. Some errors will only be generated
if -DSAFE is set in the compile options of Makefile.inc.

Installation for Windows XP without cygwin
------------------------------------------

The code without MPI has been tested on Windows XP using the free Microsoft
C++ compiler and nmake. The Microsoft SDK has to be installed, too.  All necessary
programs can be downloaded from the Microsoft webpage.

Unzip the source and data files

configure makefile:

configure.bat

If the compiler does not find the system libraries, the library path defined in
Makefile.inc has to be modified.

create lpjml executable:

nmake
