#PBS -N hwrf%STORMNUM%_gsi_d02_%CYC%
#PBS -j oe
#PBS -S /bin/bash
#PBS -q %QUEUE%
#PBS -A %PROJ%-%PROJENVIR%
#PBS -l walltime=00:30:00
#PBS -l select=3:mpiprocs=16:ncpus=16
#PBS -l place=vscatter:excl
#PBS -l debug=true

export NODES=3
export TOTAL_TASKS=48

model=hwrf
export cyc="%CYC%"
%include <head.h>
%include <envir-p1.h>

export cyc="%CYC%"
export storm_num="%STORMNUM%"

# versions file for hwrf sets $model_ver and $code_ver

module use ${HOMEhwrf}/modulefiles/
module load HWRF/rungsi
module list

export GSI_DOMAIN="d02"
${HOMEhwrf}/jobs/JHWRF_GSI

%include <tail.h>

%manual

TASK gsi_d02

NOTE: This task is optional.  Most reasonable failures of this task
will not be reported to ecFlow.  Instead, they are reported via an
email to the operator, and the system will fall back to using the
init/relocate_gfs output to initialize the vortex.

PURPOSE: Runs Gridpoint Statistical Interpolation (GSI) data
assimilation on domain 02.  

If available, uses the previous HWRF cycle ensemble data assimilation
(ENSDA) from the same storm.  That is storm as in 19L, not storm as in
storm3; storms can change their stormlabel (storm1 vs. storm3) from
cycle to cycle, but HWRF is smart enough to find them.  Usually the
ENSDA is not run, in which case the GDAS ENKF is used in its place, if
available.  If the GDAS ENKF is unavailable, the scripts run with no
ensemble.

Upon successful completion of both gsi_d0* jobs, the merge job will
merge the two into the atmospheric fields as input to the forecast
job.

Events:

  canceled - If the scripts underlying this job fail, the job will be
    marked as complete, and the canceled flag is set.  The forecast
    will use the relocate_gfs job output in place of the GSI.

TROUBLESHOOTING

This job requires an enormous amount of observational data, the entire
GFS ENKF ensemble, GDAS and a number of other inputs.  Hence, there
are many ways that failures elsewhere in of the production suite can
cause the GSI to fail.  Debugging this requires expert assistance, so
please call on-call staff unless the faiure is immediately obvious,
such as a downed node or filesystem crash.

Due to that very issue, the GSI jobs are set up to automatically
disable data assimilation if any reasonable failure happens.  They
signal this through their "canceled" event, which causes the forecast
to start without them.  When they cancel themselves, the
init/relocate_gfs vortex is used in their place.

%end