// This file defines what appears on the Main Page of the documentation
// generated by doxygen. The file contains no code, and does not appear
// in any cpp include statement.

/**
\mainpage FV3GFS Workflow Manager

\anchor mainpage-top

\section intro Introduction

With the FV3GFS system, a Rocoto driven workflow is being used. This workflow builds on the experiences and work of Kate.Howard and Terry.McGuinness for the GSM, but strips down a lot of complications arising from the use of a cumbersome <b>\c para_config</b>. The <b>\c para_config</b> no longer exists in the workflow and instead a number of <b> \c config </b> files are added, one each for an individual task e.g. <b>\p config.anal</b> contains the analysis specific information. A base config called as <b>\c config.base</b> contains
information related to the machine, super-structure, etc. The idea behind splitting the <b>\p para_config</b> into individual, smaller and managable configs is to provide a means to run any chosen task without the overhead of the full cycling framework. All the configs are located under <b>\c fv3gfs/config</b>

Additionally, the structure of the <b>\c COMROT</b> directory is now modified to look like operations. This enables the use of the workflow much closer to the operational environment, with the exception of the workflow manager.

This is a very much a work in progress and any issues should be reported back and are greatly appreciated.

\section expt_setup Setting up an experiment

To setup an experiment, a python script <b>\c setup_expt.py</b> (located in <b>\c fv3gfs/ush</b>) can be used:

    $> setup_expt.py -h
    usage: setup_expt.py [-h] [--machine {HERA,WCOSS_C}] --pslot PSLOT
                     [--configdir CONFIGDIR] [--idate IDATE] [--icsdir ICSDIR]
                     [--resdet RESDET] [--resens RESENS] [--comrot COMROT]
                     [--expdir EXPDIR] [--nens NENS] [--cdump CDUMP]

    Setup files and directories to start a GFS parallel. Create EXPDIR, copy
    config files Create COMROT experiment directory structure, link initial
    condition files from $ICSDIR to $COMROT

    optional arguments:
        -h, --help   show this help message and exit
        --machine    machine name
                        (default: WCOSS_C)
        --pslot      parallel experiment name [REQUIRED]
                        (default: None)
        --configdir  full path to directory containing the config files
                        (default: None)
        --idate      date of initial conditions
                        (default: 2016100100)
        --icsdir     full path to initial condition directory
                        (default: /scratch4/NCEPDEV/da/noscrub/Rahul.Mahajan/ICS)
        --resdet     resolution of the deterministic model forecast
                        (default: 384)
        --resens     resolution of the ensemble model forecast
                        (default: 192)
        --comrot     full path to COMROT
                        (default: None)
        --expdir     full path to EXPDIR
                        (default: None)
        --nens       number of ensemble members
                        (default: 80)
        --cdump      CDUMP to start the experiment
                        (default: gdas)

The above script creates directories <b>\c EXPDIR</b> and <b>\c COMROT</b>. It will make links for initial conditions from a location provided via the <b>\c --icsdir</b> argument for a chosen resolution for the control <b>\c --resdet</b> and the ensemble <b>\c --resens</b>. Experiment name is controlled by the input argument <b>\c --pslot</b>. The script will ask user input in case any of the directories already exist. It will copy experiment configuration files into the <b>\c EXPDIR</b> from <b>\c CONFIGDIR</b>.

Sample initial conditions for a few resolutions are available at:<br>
<b>Theia:</b> /scratch4/NCEPDEV/da/noscrub/Rahul.Mahajan/ICS<br>
<b>WCOSS Cray:</b> /gpfs/hps/emc/da/noscrub/Rahul.Mahajan/ICS

Next step is for the user to go through the individual config files (atleast <b>\c config.base</b>) and customize the experiment configuration based on user needs. A stock configuration will be provided at a later stage, but it is imperative that the user understand the steps involved in the system.

\section dependencies Main components and Caveats

The workflow touches almost all parts of the system and listed below are the branches of the components that will work with the FV3GFS workflow.
The workflow has not been integrated into the super-structure completely as yet.

<table>
<caption id="multi_row">FV3GFS Components and versions</caption>
<tr><th>Component                     <th>Build from
<tr><td>Workflow                      <td><c>fv3gfs/trunk</c>
<tr><td>NEMS                          <td><c>nems/trunk</c>
<tr><td>FV3                           <td><c>fv3/trunk</c>
<tr><td>GSI                           <td><c>gsi/branches/EXP-fv3gsi</c>
<tr><td>ObsProc                       <td><c>obsprocs/releases/obsproc_prep_RB-4.0.0</c>
<tr><td>Post                          <td><c>fv3gfs/tags/post4fv3</c>
<tr><td>Verification                  <td>UNSUPPORTED
</table>

\b Caveats:<br>
Post-processing involves converting the FV3 grid to a Gaussian grid for analysis and a LatLon grid for use with <c>ncep_post</c>.<br>
Verification is not enabled in this version of the workflow, but can be exercised offline.<br>
Archive step only archives what is necessary to HPSS for e.g. model restarts. It also creates an online archive of the pressure grib files necessary for offline verification.<br>

\section workflow_setup Setting up the workflow

As mentioned earlier, a \b Rocoto based workflow (as opposed to the Legacy PSUB/PBEG/PEND) is being used for cycling a FV3GFS system. The workflow is contained within an \c XML file which will depend on the configuration of your experiment as described within the config files. A python script <b>\c setup_workflow.py</b> (located in <b>\c fv3gfs/ush</b>) is provided to setup the XML and will parse information contained within the config files.

    $> setup_workflow.py --help
    usage: setup_workflow.py [-h] [--expdir EXPDIR]

    Setup XML workflow and CRONTAB for a GFS parallel.

    optional arguments:
      -h, --help       show this help message and exit
      --expdir EXPDIR  full path to experiment directory containing config files
                            (default: current working directory)


This utility creates a workflow XML file \c PSLOT.xml and crontab \c PSLOT.crontab

\b Rocoto uses the XML file <b>\c PSLOT.xml</b> to submit and control the workflow independent of the users knowledge of the batch scheduling system.

\section rocoto_cheatsheet Rocoto Cheatsheet

For full documentation on \b Rocoto see <a href=http://christopherwharrop.github.io/rocoto>here</a>

To execute the workflow:

    rocotorun -d PSLOT.db -w PSLOT.xml

<b>\c PSLOT.db</b> is a SQL database that \b Rocoto uses to keep a track of the workflow state and the user only interacts with it via Rocoto commands. To enable \b Rocoto to monitor the progress of the tasks, <b>\c rocotorun</b> command needs to be repeated frequently and is achieved by setting up a \b cron. <b>\c PSLOT.crontab</b> is the crontab created for the machine when <b>\c setup_workflow.py</b> was executed. See \b crontab documentation for setting up a \b cron task and be mindful of its existence after your
experiment is concluded.

Check the status of the workflow:

    rocotostat -d PSLOT.db -w PSLOT.xml -c all

Check the status of a task in the workflow:

    rocotocheck -d PSLOT.db -w PSLOT.xml -t taskname -c YYYYMMDDHHSS

Kick-off a task of the workflow irrespective of its dependency status:

    rocotoboot -d PSLOT.db -w PSLOT.xml -t taskname -c YYYYMMDDHHSS

Rewind a task in the workflow:

    rocotorewind -d PSLOT.db -w PSLOT.xml -t taskname -c YYYYMMDDHHSS

Terry.McGuinness has an utility <b>\c rocoto_viewer.py</b> that takes the \b Rocoto command line interface out of the picture. See GSM documentation for details. It is still helpful to interact with \b Rocoto using native rocoto commands.

*/