TITLE:
wave_glwu.v1.0.0: Upgrade of the wave_glw system to unstructured spatial grids

SUMMARY:
- Replace regular grid with unstructured grid,
- Add 20 hourly cycles interspersed with existing 4 daily cycles,
- Wave-physics tune-up for nearshore applications,
- Massive changes to codes and scripts,
- Added netcdf pre and post-processing procedures,
- Change of underlying wave code to v5.16.0,
- Porting to Cray X40
- Reconciling land-sea mask inconsistencies in ice preprocessing.

CHANGE REQUEST CLASS:
Major system upgrade

STRUCTURE AND RESOURCE DESCRIPTION
Four major steps PREP, FORECAST, POST and PGEN
- PREP
 - No changes in node count relative to P1 (5 nodes, but now 120 procs)
 - Runs about 5 times faster (~ 1 min)
- FORECAST
 - Requires now 32 nodes (768 cpus), from 24 nodes (384 cpus)
 - Runs about 3 times slower due to higher res grid (~ 20-25 min)
 - Runs 20 new short range cycles in about ~10 min
- POST
  - Requires 13 nodes (312 cpus), from 2 nodes (32 cpus)
  - Added new output types and files as per description below
  - I recommend test alternative node count in NCO prod envir if need
    to reduce footprint
- PGEN
  - Serial job, no changes relative to P1
- All steps are run at hourly cycles [00,01,02,...,21,22,23]Z
 - Cycles 02, 08, 14 and 20Z are long-range (147h forecast range)
 - All other cycles are short-range (48h) 

CHANGES TO SCRIPTS AND CODES:
0) All scripts
- Removal of switch between glwn and glwn late runs (all is now glwu)
- PGEN step has not been tested. Scripts were modified to comply with new output
  file names, but not tested yet. This is usually perfomed by NCO after code 
  delivery.
- modules have now been specified with versions for both runtime and compilation
- model_ver and code_ver were replaced by specific wave_glwu_ver and 
  wave_code_ver tags
1) J-Jobs
- Root-level variables were added to all j-jobs to define NCO paths (need
  verification at NCO's end)
- Input and output COM are now clearly labeled
- COMINwave and COMOUTwave are used to assign base directories without a date, as
  these are used to specify input or output locations for dates different than
  current run,
- Modified to comply with current NCO standards (eg, removal of variables now set 
in prod_util and prod_envir),
- Removal of all dev blocks,
- JWAVE_GLWU_RUN_CHK needs to be modified to work on Cray (DST_check.pl needs to
  be added to Cray utility scripts etc). ATTENTION: this script may be removed
  altogether in case we can run the GLWU system without EST/EDT dependency
- JWAVE_GLWU_PGEN has been tweaked and runs on the dev end: requires adding 
  module use /gpfs/hps/nco/ops/nwpara/modulefiles and 
  module load util_shared/$util_shared_ver to ecflow script
2) Execution scripts
- exwave_glwu_prep.sh.ecf
 - Uses bottom of the hour files: search for NDFD files now looks at files 
   +- 20 minutes from HH:30:00
 - wavendfd_glwu.sh now uses grib2 -> netcdf conversion and netcdf preprocessing
   to obtain wave model wind forcing file,
 - Addition of procedures for generating prep for hourly short-range runs (48h)
 - Changes to allow processing of unstructured grid (eg, wavemod_def.sh)
 - Reduction of look back ice files to a single daily analysis
   loop "while [ "$ymdh" -le "$ymdh_end_ice" ]" effectively now processes a single
   time slice.
- exwave_glwu_forecast.sh.ecf
  - Minimal cleanup, no significant changes

- exwave_glwu_post.sh.ecf
 - Addition of procedures for generating native unstructured grid netcdf gridded
   output file (eg, multiwavefldn prefix files and actions)
 - preceding step to generate master 500m resolution interpolated grid for feeding
   all outher gridded file generation steps -> slows a little POST
 - Added procedures for new grids grlc_2p5km_sr, glwu and grlr_500m
 - Now generates time series output as per multiwavespec_ts.sh, for rip current 
   application
 - Generates on the fly rip current data transects as per if [ "$ripin_OK" = 'yes' ]
   call
 - Addition of 55 new output point locations (eg fix/wave_glwu.buoys)
 - Requires now 13 nodes due to additional gridded and point outputs

- exwave_gwlu_pgen.sh.ecf
 - Removed publishing of grl grib2 files (5km regular grid) from the product list,
 - Added short-range glrc_2p5km_sr files for all 24 daily cycles.
 - Needs tweaking of perl scripts formbul.pl and formbul_config in util_shared/1.0.5
   paths to formbul_config need to be hardwired or properly added in formbul.pl
   in formbul_config, paths to PCOM, deckpath and jlog_file need to be properly set 

3) utility scripts
- Eliminated ice climatology scripts and programs (eg, imsice_gl.sh)
- multiwavestart_glwn.sh becomes wavestart_glwu.sh
- multiwavemod_def.sh becomes wavemod_def.sh and now deals with unstructured grids
- multiwavendfd_inc.sh becomes wavendfd_glwu.sh, now uses netcdf preprocessing and
  eliminated time slice search for NDFD files, now using single NDFD file, speeding 
  PREP in 3-4 min
- Added multiwavespec_ts.sh to generate time series file output for rip current 
  application

4) Executables
- Eliminated convert_ice2asc.fd and imsice_gl.Cd, 
- Removed steepness files from output: eliminated dependency on multiwavesteepgrb2
- Change: convert_ndfd.fd becomes convert_ndfd_glwu.fd, 
- Added inpaint_nic_glwu.fd.

5) Output files
- All files change prefix from glwn to glwu
- Will require adjustment of PGEN step to reflect this change (already initialized
  by developer, will need scrutiny and final tweaking by NCO)
- Files that are renamed
  glwn.grlc_2p5km.tCYCz.grib2 -> glwu.grlc_2p5km.tCYCz.grib2
  glwn.grlc_2p5km.tCYCz.grib2.idx -> glwu.grlc_2p5km.tCYCz.grib2.idx
  glwn.grlc_2p5km.tCYCz.ice -> glwu.glwu.tCYCz.ice
  glwn.grlc_2p5km.tCYCz.log -> glwu.glwu.tCYCz.log
  glwn.grlc_2p5km.tCYCz.outgrd -> glwu.grlc_2p5km.tCYCz.outgrd
  glwn.grlc_2p5km.tCYCz.restart -> glwu.glwu.tCYCz.restart
  glwn.grlc_2p5km.tCYCz.wind -> glwu.glwu.tCYCz.wind
  glwn.grlr.tCYCz.outgrd -> glwu.grlr.tCYCz.outgrd
  glwn.grl.tCYCz.outgrd -> glwu.grl.tCYCz.outgrd
  glwn.tCYCz.bull_tar -> glwu.tCYCz.bull_tar
  glwn.tCYCz.cbull_tar -> glwu.tCYCz.cbull_tar
  glwn.tCYCz.csbull_tar -> glwu.tCYCz.csbull_tar
  glwn.tCYCz.log glwu.tCYCz.log
  glwn.tCYCz.outpnt -> glwu.glwu.tCYCz.outpnt
  glwn.tCYCz.spec_tar.gz -> glwu.tCYCz.spec_tar.gz
  whatglwice.tCYCz -> whatglwice.tCYCz
  what_waveglwn_used.tCYCz -> what_waveglwu_used.tCYCz
  where_wave_glwn_ran.tCYCz -> where_wave_glwu_ran.tCYCz
- Files that are added
  glwu.glwu.tCYCz.nc
  glwu.grlc_2p5km_sr.tCYCz.grib2
  glwu.grlc_2p5km_sr.tCYCz.grib2.idx
  glwu.glwu.tCYCz.outgrd
  glwu.grlr_500m.tCYCz.grib2
  glwu.grlr_500m.tCYCz.grib2.idx
  glwu.tCYCz.ripin_tar
  glwu.tCYCz.ts_tar.gz
- Files that are removed
  glwn.wstp.grlc_2p5km.tCYCz.grib2
  glwn.wstp.grlc_2p5km.tCYCz.grib2.idx
  glwn.grl.tCYCz.grib2 
  glwn.grl.tCYCz.grib2.idx
- Removed point DBLN6 from output list as it falls outside new unstructured grid

DISSEMINATION INFO
1) All output files should be sent to FTPPRD and NOMADS
- New files need to be added to dataflow as per list above
- Changed and removed files also need to be updated in DATAFLOW
2) New AWIPS product glwu.grlc_2p5km_sr.tCYCz.grib2 needs to be
   added to AWIPS work flow 
3) Both glwu.grlc_2p5km_sr.tCYCz.grib2 (all 24 daily cycles)
   and glwu.grlc_2p5km.tCYCz.grib2 (four longer-range cycles)
   need to be disseminated through NOAAPORT

BUGFIXES
- Error output from child-scripts called by exwave_glwu_prep.sh.ecf (eg, 
  ush/waveice_glw.sh and ush/multiwavendfd_inc.sh) now are saved in stdout
  files no longer removed,
- Dynamically allcated arrays in multiwavesteepgrb2.f are no longer a problem as
  this code was removed from the GLWU system,
- BUGZILLA 504: This was fixed in $USH/waveice_glw.sh by relocating if blocks and 
  re-setting the stag variable relative to PDYCE parameter,
- BUGZILLA 509: fixed by adding section in exwave_glwu_prep.sh.ecf that lists all 
  NDFD files found in given time interval, retaining only three most recent for 
  each input paramter (WIND, WDIR and TMP), in parameters [wind|wdir|tmp]latest,
- BUGZILLA 218: GOTO items, some were replaced, most however belong to wave code
  or to legacy scripts that are duplicated in other wave systems. Agreed to 
  impose a broad replacement of legacy codes with GOTO statemetns in future 
  upgrades,
- BUGZILLA 222: Hard-wired FORTRAN file names, also systematic issue affecting 
  jointly several wave systems, will be addressed in forthcoming upgrade in the 
  near future,
- WAVEWATCH III code GOTOs are being eliminated gradually through a joint effort
  by the developers community. This change will take longer than desired, but
  will eventually be completed.
- Too long scripts: separation into child scripts of long sections will be 
  performed in the next upgrade,

BENEFIT OF CHANGE:
Improved skill in nearshore with new unstructured grid and physics tune-up. 
Hourly cycles will benefit small craft advisory.

USER IMPACT STATEMENT:
Addition of new files will benefit short-range, higher resolution forecasts.
AWIPS delivery times are expected to remain unchanged.
There are 55 new point output locations that will benefit several users and
applications.

TECHNICAL IMPACT STATEMENT:
PREP will have minimal impact, FORECAST will require increase from 384 cpus 
(24 nodes) to 768 cpus (32 nodes), POST will require increase from 2 to 13 nodes
(this should be tested in the prod environment to determined optimal node count
that does not increase runtime).

TESTING
Dev testing completed. Pre-evaluation by stakeholders started and ongoing.

RISKS:
N/A

PROPOSED IMPLEMENTATION
DATE:
TIME: