 _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/
 _/                                                        _/
 _/                   N A S A / N C A R                    _/
 _/                         Joint                          _/
 _/                     Finite  Volume                     _/
 _/                General Circulation Model               _/
 _/                                                        _/
 _/                     Version 1.3                        _/
 _/                                                        _/
 _/                     1 February  2002                   _/
 _/                                                        _/
 _/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/_/

 0. Preface:

     This is version 1.3 of the joint NASA/NCAR finite-volume
     General Circulation Model. The dynamical core formulation is
     based on the work of Lin and Rood (1996 MWR and 1997 QJ)
     and Lin (1997 QJ). All physical parameterizations (including
     the land surface model) are derived from the NCAR CCM 
     version 3 with DAO modifications.
 
     Currently, parallelization is done by shared-memory multitasking
     (using OpenMP) and distributed memory multitasking (using MPI1
     or MPI2). Multitasking using UNIX shared memory is also implemented.
     The model has been extensively tested on the SGI Origin-2000 
     system and Linux PC. 

     Starting from release 1.0, FVGCM can only be compiled with
     Fortran 90.  FVCORE now supports resolution independence which
     in turn requires dynamic allocation.  All dynamic allocation
     (with exception of some in the Land Surface Model (lsm/) is
     done with Fortran 90 primitives.

    

 1. Installation:

    o getting source codes 
 
      Source codes can be obtained by the following two ways:

      (a) getting a tar file via email. Contact info could be found
          in the end of this file.

      (b) getting codes vis CVS (Concurrent Version System) repository
         (i) gain an account on hera.gsfc.nasa.gov
        (ii) setenv CVSROOT \
             :ext:${LOGNAME}@hera.nas.nasa.gov:/CM/baseline/GEOS_DAS
       (iii) setenv CVS_RSH ssh
        (iv) cvs co fvgcm


    o Installing data files

      The generic filenames of restart datasets needed for 
      a typical simulation are:

                d_rst - dynamics restart file 
                p_rst - physics restart file
              lsm_rst - lsm restart file 
         lsm.rpointer - lsm auxiliary (pseudo pointer) file 

      We provide spun-up datasets (b55) for you to test the model,

                  b55 - 2x2.5 degree, 55 layers

      which could be found in /share/fvccm/FVGCM/TEST_DATA/b55 on NAS
      machines.

      Starting from version 0.9.8, installation of datasets can 
      be done as part of the experiment installation (explained
      below in the 'make install' section).

 2. Building the Model

    o configure

      The script configure is created in version 1.2.0 to
      help choose a right Make.macros and generate fvgcm.h automatically.


    o Make and GNUmake

      We chose GNUmake (known as 'gmake' on DAO systems) to build
      the model executable in previous versions since its behavior 
      is "guaranteed" to be the same across platforms. Later we 
      identified a bottleneck in the .F.a dependency rule which 
      requires an 'ar' command for each source file. This restriction
      not only made the compiling longer than necessary but also
      prevented users from using parallel make on systems with 
      multiple CPUs. 

      We addressed this problem by replacing the .F.a rule to .F.o
      rule; this alone reduced the total compiling time significantly.
      Furthermore, parallel make is now possible under the appropriate 
      environment. The job script 'make.j' can be used interactively
      or as a batch script for compiling. In make.j:

      N_CPUS  sets the maximum number of processes that can be used
              to 'make' simultaneously. It's equal to be 8 by default.

    o Use debug queue for compiling
      
      If the front-end system (currently dixon0) is not identical to the
      production systems in terms of operating system, compiler version
      and patch levels, we recommend the users to use the debug queue on 
      jimpf0 or jimpf1 for compiling.

      - Change directory to the "top" level of fvgcm source tree
 
          cd ~/fvgcm-1.3

      - Run configure

          configure

          (Check the file 'fvgcm.h' for model resolution, FVGCM_LEV 
           should be set to 55 for b55 case.)

      - Submit the compiling job

          qsub make.j 

      - You can safely ignore these three types of warning messages
        on the Origin 2000; it's due to annoying features of the 
        compiler, not the source code.

        . WARNING 84: /usr/lib64/mips4/r10000/libfastm.so is not used for
                      resolving any symbol.
        . WARNING 85: definition of __sqrtf in /usr/lib64/mips4/r10000/libfastm.so
                      preempts that definition in /usr/lib64/mips4/libm.so.
        . WARNING 134: weak definition of log in /usr/lib64/mips4/r10000/libfastm.so 

      - The default loader options for Origin 2000 (LDFLAGS in Make.macros)
        actually suppressed these warnings; you'll need to use the other 
        LDFLAGS (currently commented out) in order to see these warnings.
     
    o Make Targets

      The default 'make' will make the 'All' target. In addition to 
      'All', there are a few other targets that you can make:

        All          -  pseudo target for fvgcm.x, rst_date and diagpp
        Dycore       -  pseudo target for building the dynamical core
        Init_dyn     -  pseudo target for running in NWP spinup mode
        fvgcm.x      -  the model executable
        tar          -  a gzipped tar file for the source tree
        tar-makefile -  a gzipped tar file for Makefiles and macros
        depend       -  update source dependency in all Makefiles
        clean        -  clean a bit
        clean-sgi    -  remove the rii_files directories
        cleaner      -  make it cleaner
        nascar.f     -  single file contains all Fortran source files
        nascar-check -  flint check for the whole model
        dycore-check -  flint check for the dynamical core

    o Directory tree:

        The source code is stored in a directory tree as shown below: 

                                  Top level
                                      |
       +-----+--------+--------+------+----+------+-- ---+------+------+
       |     |        |        |      |    |      |      |      |      |
       cf   drvs   ecmfft   fvcore   lsm  misc pilgrim  phys   util   CASE(*)


        cf           -  configuration files
        drvs         -  drivers
        ecmfft       -  fast Fourier transform routines
        fvcore       -  Lin/Rood finite volume dynamical core
        lsm          -  NCAR land surface model version 1.0
        misc         -  subroutines that don't fit in any of the above
        phys         -  NCAR CCM3.3.8 physics package
        pilgrim      -  communication routines 
        util         -  utility and postprocessing programs
      * CASE         -  subdirectory for specific experiment named 'CASE'
                        (Job scripts should be submitted from this subdirectory. 
                        It is created by the installation procedure, 'make 
                        install', and it is not part of the source code 
                        distribution.)

 3. Running the Model

    o System tour
  
      In order to set the system parameters in the job scripts 
      (fvgcm.j, monthly.j.tmpl and weekly.j.tmpl), it is worthwhile
      to take a look at the related DAO/NAS system configuration 
      illustrated below.

               bjerknes                             
               jimpf0                   
               jimpf1                  
               kalnay                           
         +------------------+                                        
         |    Production    |                                         ooooooooooooooo
         |      Systems     |                                         o             o
         |                  +=============== HiPPI ===================+   helios1   o
         |   $SCRATCH[12]   |               channel                   o             o
         |   $Silo          |                                         ooooooooooooooo
         +------------------+         

         The job scripts use the $SCRATCH[12] directory, which is set by
         the system and is highly volatile, as the working directory ($WorkDir). 
         When the simulation is done, all the files are moved to a less 
         volatile location ($Silo), and then to MSS on helios1. The $Silo location 
         is carefully selected to be on the same filesystem as $WorkDir,
         avoiding overhead of moving files across filesystems.


         The modified routines now allow the flexibility to use
         any frequency of sea surface temperature in the boundary
         condition file.  When the model namelist parameter SSTCYC
         is true, the code no longer requires "pssttim" (generally
         set to 12) data records in the sst boundary condition file 
         or that the first record be in January.

    o System configuration 

      Basic parameters: (for general users)

         N_SMP     - number of CPUs for running OpenMP threads
         N_MPI     - number of CPUs for running MPI processes 
         VER       - source directory, model version is typically embedded
         CASE      - case id, which is used to create a case depending working
                     directory under the "top" level directory.
         SourceDir - where the model installed
         FVData    - directory for input files to FVGCM
         WorkDir   - temporary directory to serve as working space
         PostProc  - postprocessing (vertical interpolation)
         TimeMean  - postprocessing (time average)
         SaveToMSS - save the model output to the Mass Storage System
         MSSHost   - hostname of the mass storage system
         SpinUp    - spinup the model, implying that the latest restart files
                     will be kept on $Silo (see below) with an option ($SaveToMSS)
                     to archive them
         CleanSilo - default to true to clean up the $Silo directory after data 
                     has been transferred to MSS, will be set to false by script
                     if any of the rcp calls failed
         MaxCount  - number of simulation "units" in the same job submission,
                     a simulation unit is typically a month or a week of model
                     simulation
         NUMBER_MLP_PROCESSES - number of CPUs for UNIX forked processes.
         NUMBER_CPUS_PER_MLP_PROCESS - number of CPUs for running OpenMP threads
                     

      Advanced parameter: (designed for model developers)

         Benchmark - benchmark the model, implying that the restart files won't
                     be updated in the source directory. Never run benchmark with
                     automatic resubmitting, the result is an infinity loop!!

      Derived/Hidden parameters:

         Silo      - temporary directory to serve as on-line staging storage,
                     it's the directory $LOGNAME/$VER/$CASE under the same 
                     filesystem as $WorkDir
                      
    * NOTE: By Unix convention, 1 means TRUE and 0 means FALSE. Starting from
            version 0.9.8, we adopt this convention for setting the above
            parameters which take true (yes) or false (no).


    o Model configuration 

      We tried to make the model configuration similar to current GEOS as much as
      possible. For the test cases (b55) included in this release, the only 
      parameters that users may want to change are NYMDE, NHMSE and NDAY in the  
      namelist embedded in each job script. NYMDE and NHMSE together specify the
      end of simulation in YYYYMMDD and HHMMSS format, respectively. NDAY specifies
      the simulation period starting from current restart files. If, at the end of
      simulation, NYMDE/NHMSE is not reached, the jobs script will continue to
      resubmit itself. The current namelist items are

      JOB      : a short job title 

      NCPATH   : absolute path in which netCDF files reside

      SSTDATA  : SST dataset (netCDF)

      SSTCYC   : .true.  - climatological SST is used
                 .false. - real SST is used

      OZNDATA  : ozone dataset (netCDF, zonal mean)

      H2ODATA  : water vapor dataset (direct access, zonal mean)

      NYMDB    : starting date of restart file (YYYYMMDD)

      NHMSB    : starting time of restart file (HHMMSS)

      NYMDE    : ending date of restart file (YYYYMMDD)

      NHMSE    : ending time of restart file (HHMMSS) 

      NDAY     : number of days to simulate in this run

      PDT      : frequency (in seconds) to call physics

      MDT      : frequency (in seconds) to perform vertical remapping

      NDOUT    : frequency (HHMMSS) to output reference state fields (instantaneous)

      NGOUT    : frequency (HHMMSS) to output diagnostic fields (time averaged)

        nsplit : number of small time steps within each large time step

        zstat  : .true.  - output zonal mean of reference state fields
                 .false. - don't output zonal mean of reference state fields 

        ccm3   : .true.  - call CCM3 physics
                 .false. - don't call CCM3 physics

        rayf   : .true.  - apply flow dependent Rayleigh friction
                 .false. - don't apply Rayleigh friction

        iuhs   : unit logic number for surface geopotential and standard deviation

        iuic   : unit logic number for dynamics initial condition

        iout   : unit logic number for reference state fields

        izou   : unit logic number for zonal mean reference state fields

        nsrest : running mode

                 0  - initialization run
                 1  - branch run (not implemented yet)
                 2  - normal restart run

        diag   : .true.  - output diagnostic fields
                 .false. - don't output diagnostic fields

                 The diagnostic fields are selected from the text file 
                 'diag.tbl' at the top level. The first column of the file
                 the name of the fields, the second column is either .T. for
                 selecting the field for output or .F. for not.  The rest of
                 the line is a short description about the field with its 
                 unit included in a square bracket. Exclamation can be used
                 for comments, but it has to be the leftmost non-blank
                 character in the line. The order of the fields in this file 
                 can be arbitrary.

    o Filename convention

      The model works with 'generic' filenames, it is the responsibility of the
      job script to rename them properly.

      On the input side, the restart files are copied from source directory to
      work directory

         d_rst   -->    fort.81
         p_rst   -->    fort.83
      
      then the real lsm restart and history filenames are found from the pseudo
      pointer file 'lsm.rpointer', 

         lsm_rst -->   filename specified at the 1st line of lsm.rpointer
         lsm_hst -->   filename specified at the 2nd line of lsm.rpointer

    * Note that if the 2nd line of lsm.rpointer says 'history_file_not_open',
      it means the lsm history data file for the previous month is not 
      available. It is currently hardwired in the code that lsm should 
      output monthly history tape. This can be modified in the future. 
      It is worth mentioning that the lsm history tape, if exists, should
      contain the monthly history data ending at the time indicated by the
      lsm restart filename.

      An utility program 'rst_date' is built and copied to the CASE directory.
      This program works only on dynamics restart file, either it's by the
      generic name 'd_rst' or by other names (see below).

      On the output side, in addition to copying the new restart files back
      to the source directory with the generic names, all output are renamed
      before they were sent to the MSS. The filename convention is in general
      (string in square parentheses are optional)

        CASE_TYPE_[PP_]YYYYMMDD.HHMMSS[-YYYYMMDD.HHMMSS][.EXT]

      where CASE is the experiment name of the simulation, TYPE specifies various
      output streams (d, p, rout, diag, and zavg), YYYYMMDD.HHMMSS is the time
      stamp (Y2K compliant) of the data in terms of model time. For instantaneous 
      data files, e.g., restart files, there is only one time stamp since the data 
      stores a snapshot of the model state. Diagnostic output files contain two
      time stamps separated by a hyphen '-', which specify that a time sequence of
      data (starting at the 1st stamp and ending at the 2nd) is stored in the
      dataset or it may be a postprocessed dataset (e.g., time average) over this
      period of time. PP is either prs (data is interpolated to pressure 
      coordinates), or tm (data is time averaged over the spcified period), or
      prs_tm (both prs and tm). EXT is file extension, currently it is only used
      for the GrADS control files generated by the model or the postprocessor.
      Since this current release, diagnostic output files could also have HDF format.
      
      Therefore, running an experiment with case name 'test' from 00Z 1 Jan, 1996
      for one month (i.e., simulation ended at 00Z 1 Feb, 1996), the following
      files will be archived to the MSS after the simulation is completed:

        - restart file for dynamics

            fort.81  -->  test_d19960201.000000

        - restart file for physics

            fort.83  -->  test_p19960201.000000
      
        - diagnostic files

            diag.bin -->  test_diag_19960101.120000-19960131.120000
            diag.ctl -->  test_diag_19960101.120000-19960131.120000.ctl

          ==> generated by diagpp
            test_diag_prs_tm_19960101.120000-19960131.120000
            test_diag_prs_tm_19960101.120000-19960131.120000.ctl
      
        - reference state

            fort.82  -->  test_rout_19960101.060000-19960201.000000

        - zonal mean statistics

            fort.99  -->  test_zavg_19960101.060000-19960201.000000

        - restart file for lsm

            lsmr_19960201_00000
      
        - history file for lsm

            lsmh_1996-01.nc
      
      It is worth mentioning that lsm related files were named by lsm itself from 
      the model run, the job script does not rename them, instead it simply moves
      them around.


 4. Notes to developers

    o Software architecture

       - Version control: The source code of this model is maintained by the 
                          public domain software 'cvs'.

       - Make utility:    The compiling of the source code is managed by
                          recursive 'make'. In other words, there is a Makefile
                          in each of the source subdirectories. The top level 
                          Makefile invokes 'make' in each of the subdirectories
                          to update the library files (libgcm.a and/or libdycore.a)
                          at the top level. Then the model is built by linking 
                          to the updated library file.

                          Macro definitions are 'include'd into each Makefile
                          from the top level, therefore allowing partial
                          'make' in each subdirectory. The include file 
                          'Make.macros' contains system dependent macros such
                          as compiler options. The include file 'Make.rules'
                          contains general dependency rules, such as .F.o, and
                          in most cases should be system independent.

                          There are certain conventions we followed for the 
                          Makefiles:

                          o  macros are all in upper case
                          o  real targets are all in lower case
                          o  pseudo targets are lower case with the first letter
                             capitalized when used together with real targets
                             (All, Dycore, Init_dyn), otherwise they are all
                             in lower case (clean, tar)
        
                          In order to utilize parallel make whenever possible, 
                          the price to pay is that the libraries libgcm.a and 
                          libdyn.a are forced to be rebuilt by an 'ar' command
                          for all the object files in each subdirectories. Therefore
                          the executables fvgcm.x, dycore and init_dyn need to be
                          rebuilt regardless of the dependency. However, from the
                          performance analysis for compiling, this minor cost is
                          well justified.


       - Installation:    Current installation ('make install') serves two 
                          purposes. First, gather user information from an
                          interactive interface to create proper job scripts.
                          Three job scripts, fvgcm.j, monthly.j.tmpl and
                          weekly.j.tmpl, are generated from an m4 (cf/script.m4)
                          source. The functionality of the script generation can
                          be further improved in the future. However, local 
                          stack setting for shared memory parallel execution has
                          been taken care of for SGI O2000, Linux and Dec Alpha.

                          The second purpose of the installation is to copy well 
                          spunup initial conditions for the simulation.

       - Postprocessing:  The only postprocessing capability currently available
                          is to interpolate the diagnostic output from eta to 
                          pressure coordinates with an option to perform time 
                          average. The vertical interpolation from eta to pressure
                          coordinates is computed with a finite volume based 
                          algorithm identical to the model's. Postprocessing
                          is submitted to a 1-CPU queue along with the archiving
                          of the data to the mass storage system.
                          

    o Source dependency

       - You should always update the Makefile in the same directory when you
         add or delete a subroutine. Except the Makefiles at the top level
         and the one in the 'util' subdirectory, each Makefile contains macros 

           DYNSRCS - list of subroutines used in both pure dycore and gcm

           PHYSRCS - list of subroutines used only in gcm
      
         which should be revised for the added/deleted subroutines.

       - To create new Makefiles when upgrading to new physics routines from
         NCAR, it's straightforward to use the old one as a template. When
         the new routines are in place, run the following command 

           ls -1 *.F | sort | awk '{printf "                  " $1 " \\\n"}' -

         and save the output from this command in a temporary file. Then
         split the list in this file into DYNSRCS and PHYSRCS. Note that
         DYNSRCS and PHYSRCS are mutually exclusive but their union is
         exactly the content of this temporary file. Replace the definition
         of DYNSRCS and PHYSRCS in the old Makefile by the new ones. The
         final step is to re-generate proper dependency by running 'make 
         depend' from the top level directory.

       - For software management purpose, listing these subroutines in
         alphabetic order is strongly recommended.

       - Run 'make depend' from the top level to update the source dependency.
         The original Makefile will be saved as Makefile.bak. 

    o Diagnostic output

       We adopt a similar idea to NCAR CCM3 in managing diagnostic fields. 
       A huge array for the diagnostic fields is dynamically allocated with known 
       size in the model. This array is filled up by the 'outfld' subroutine
       calls from parallel regions. Our first attempt to keep the interface of
       'outfld' identical to that of CCM3's was not successful. The current 
       interface replaces the field name (1st argument) with an array of 
       "field attributes" which consist 5 elements (see the code for details). 
       
       The procedure below describes how to add a new field into the diagnostic
       array.

       - Add the new entry in misc/diag.h

         First, declare the new field's identification as an integer, the 
         convention being the field name (capitalized) prepended with a lower 
         case 'i', e.g., iSLP for sea level pressure. Then insert this new id
         into the appropriate location of the 2-D/3-D parameter lists. Consider
         the parameter list an alphabetically sorted linked list like

               +---+         +---+         +---+         +---+
               | A |---------| B |---------| D |---------| E |
               +---+         +---+         +---+         +---+

         Standard linked list insertion for a new entry 'C' involves cutting
         the current link between 'B' and 'D', then create two new links 
         from 'B' to 'C' and from 'C' to 'D', respectively.

               +---+         +---+         +---+         +---+
               | A |---------| B |         | D |---------| E |
               +---+         +---+         +---+         +---+
                               |             |
                               |    +---+    |
                               +----| C |----+
                                    +---+

         For example, after inserting iPBLH to the following partial list

            parameter (iORO      = iLWSH    + 1)
            parameter (iPRECC    = iORO     + 1)
            parameter (iPRECL    = iPRECC   + 1)

         the new partial list becomes

            parameter (iORO      = iLWSH    + 1)
            parameter (iPBLH     = iORO     + 1)
            parameter (iPRECC    = iPBLH    + 1)
            parameter (iPRECL    = iPRECC   + 1)

         A quick check is that the id on the LHS of the equal sign should be
         the same as the one on the RHS in the line right below it, except the
         first and the last entries.

         Note that there are lists for 2-D and 3-D, respectively. They should
         be in alphabetic order by themselves. In case you have different lexical
         sense than the computer does, there is a paranoia check on the ordering
         of these lists. The model will stop if it fails to pass this test.

       - Add the new entry in misc/diaginit.F

         You will need to find the appropriate location in diaginit.F to specify
         the name (character*8), short description (character*80) and unit 
         (character*16) of the new fields.

       - Add the new entry in diag.tbl

    o Namelist

       There are logical unit numbers for various output streams (80, 81, 82, 99)
       specified in the namelist. It seems that the users can re-assign these unit
       numbers arbitrarily to something else. However, it won't work with the
       current version. The script needs to be modified for other choices; these
       numbers are currently hardwired in the script. Indeed, a better logical unit
       number management should be planned and maybe the Fortran OPEN statement
       should be used for all I/O.
         

 5. Bug report, questions and suggestions

    o Please contact the modeling group if you find a bug, have questions
      or difficulty in running the test case. Your suggestions to the 
      modeling system are heartily welcomed. The mail alias for the modeling
      group is 

         fvccm@dao.gsfc.nasa.gov

      For general information about the model, please contact

         Shian-Jiann Lin   (301)614-6161      lin@dao.gsfc.nasa.gov

    o For now, happy computing!

