U _g @szdZddgZddlZddlZddlZddlZddlZddlZddl Zddl Zddl Zddl Z ddlZddlZddlZddlZddlZddlZddlmZmZmZmZmZddlmZmZmZddl mZddlmZmZdd l m Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l m(Z(m)Z)m*Z*m+Z+dd lm,Z,d d ddddZ-Gdddej.j/Z0ddddddddddd Z1Gdddej.j/Z2dS)z8This module handles WW3 related scripts for HAFS system.WW3InitWW3PostN) FileProductRUNNING COMPLETEDFAILED UpstreamFile) make_symlink deliver_filewait_for_files)DBNAlert)NamedDirTempDir)mpimpirunrunrunstrcheckrunexebigexealias) to_datetimeto_datetime_rel to_fraction to_timedelta) WW3InputError) ./mod_def.ww3{intercom}/ww3/mod_def.ww3) ./ww3_mesh.ncz{intercom}/ww3/ww3_mesh.nc)z ./wind.ww3z{intercom}/ww3/wind.ww3)z ./current.ww3z{intercom}/ww3/current.ww3)z ./restart.ww3z{intercom}/ww3/restart_init.ww3)Zmod_defZww3_meshwindcurrentZrestartcspeZdZdfdd ZddZddd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ZS)rN~`Tc  sTtt|j|||fd|i| |t||_t||_t||_t||_ dS)aCreates a WW3Init dstore - the produtil.datastore.Datastore to use conf - the HAFSConfig to use section - the section name for this task taskname - the task name. Default: section fcstlen - the forecast length in hours outstep - the output step in seconds pntstep - the pntout step in seconds rststep - the restart output step in seconds Other keyword arguments are passed to the superclass constructor.tasknameN) superr__init___make_productsfloatfcstlenintoutsteppntsteprststep) selfdstoreconfsectionr#r(r*r+r,kwargs __class__5/lfs/h1/ops/para/packages/hafs.v2.0.7/ush/hafs/ww3.pyr%!s    zWW3Init.__init__c Cst|_tj|jj}|dd}|dkr4dtd<|j Zt D]J\}}|\}}t j |j ||j}|||||_||d<||f|j|<qHW5QRXdS)zCreates FileProduct objects for all output files. The outdir is the directory to which the WW3 package output its final files.ww3_bdynoyes)z ./nest.ww3z{intercom}/ww3/nest.ww3nest localpathN)dict _productstcutilnumericsrr/cycleconfstr prodnamesr. transactionitemsprodutil datastorerr#timestrlocation)r-atimer6prodname filepathsr:compathprodr4r4r5r&3s   zWW3Init._make_productscks8|jD](\}}|\}}|dks,||jkr |Vq dSzIterate over all products.Nr<rCrI)r-namer1rIstuffrLr:r4r4r5productsEszWW3Init.productsc cs`t|jj}td|j|}t|dd}|dd}|dd}t|d}t||}|}||krt |j ||||dVt||}q^|d d } | d krt|jj}td |} |d d}|dd}| }t |j ||||ddV|dd } t d| | d ks| dkr\t|jj}td |} |dd}|dd}| }t |j ||||ddVdS)N input_stepr" gfs_datasetgfsgfs_item datasetitemftimerHr6r7r8頫gfswave_datasetgfswave ww3bdy_item ww3bdy_ibpT)rYrZr[rHoptionalww3_rstz ww3_rst=%salwaysgdaswave_datasetgdaswave ww3rst_itemww3rst_gnh_10m) rr/r?rr(rconfintr@rr;taskvarsprint) r-rHetimeintervalrYrZepsilonendewhenr6wtimerbr4r4r5 inputiterLs8                 zWW3Init.inputiterccsB|}t|jj}t|jd|}t|dd}|dd}|dd}|dd}t j |j||}t |d } t| |} |} d } |d d } |d d}|dd}|dd}| | kr>|j f||| |d|j}t|g|| |||d}|s d|| f}||td|V| |d} t|| } qdS)NrRrSr"rTrUrVcataloghafsdatarWrZ max_grib_waitiZgrib_sleep_timeZ min_grib_sizeZ min_grib_agerX)maxwait sleeptimemin_size min_mtime_agez@FATAL ERROR: %s: did not exist or was too small after %d seconds)logrr/r?rr(rrhr@hafsinput DataCatalogrlocaterir errorsysexit)r-loggerrHrkrlrYrZZhddcrmrnrofhourrwrxryrzthefileZwaitedmsgr4r4r5 gfsgrib2iterlsJ             zWW3Init.gfsgrib2itercCs8|}|jD] \}}|\}}|j|d|dqdS)NT)frominfokeepr)r|r<rCdeliver)r-rrIrPrLr:r4r4r5deliver_productsszWW3Init.deliver_productsc s4|d}|dd}|dkr&d}n&|dkr4d}nd|fd}d}|dd}|dd}zzt|_|d d}t|j|j dd @}fd d }t | d ddd|| dd|dkr|| d| dn|| dd|| dd|| ddt d} |r,| dk} t | dt | dd|dkrd} tjj| dtt| dd} |D]n} d| d} t| | ddD]} td | r| | 7} q| | d!d"d#| f| >}t |dqtj| rd}nd}tjjd$| f|rnt | d%d&dd|| d'd't d(} |r`| d)k} t | dnHt | d*d+dd|| d,d,t d-} |r| d.k} t | d|r6t | d/d&ddtd0d1 }td2D]}|d3qW5QRX|| d'd't d(} |r(| d4k} t | dn d5d}tj d6d!dkrhd7nd8}| d9}tj|rJd:|ft"j#$}|%|| d;}|&dn||'d.linkz {grid_inp}z ww3_grid.inprrz {grid_bot}.z {grid_msk2}z./ww3_grid_{vit[basin1lc]}.mskz {grid_msk}z {grid_obr}Zww3_gridz ./ww3_grid ww3_grid.logrz {grid_mesh}rzgfs.uvgrd10m.ncwgrib2zExtracting wind at 10 m from %sz:[UV]GRD:10 m above ground:z-iz-appendz-netcdfzEWARNING: ww3init: will use dummy wind because %s is missing or empty.z {wind_inp}z ww3_prep.inpZww3_prepz ./ww3_prepww3_prep_wind.logz{prnc_inp_gfswind}z ww3_prnc.inpZww3_prncz ./ww3_prnczww3_prnc_wind.logz {curr_inp}z./curdummy.datwz 0. 0. 0. ww3_prep_curr.logz'WARNING: Capability not implemented yetZww3_force_cold_startzCWARNING: ww3_force_cold_start is yes and will generate restart.ww3.z (unknown)z${oldcom}/{old_out_prefix}.{RUN}.confz%s: prior cycle exists.z0{oldcom}/{old_out_prefix}.{RUN}.ww3.restart.f006configrun_wavezLrestart.ww3: will generate restart.ww3 because prior cycle did not run wave. wave_modelww3zKrestart.ww3: will generate restart.ww3 because prior cycle did not run WW3.z restart.ww3z8%s: warm start from prior cycle 6-h output restart file.z@FATAL ERROR: exiting because piror cycle %s is missing or empty.zdFATAL ERROR: if desired, set force_cold_start or ww3_force_cold_start = yes can bypass this failure.r{zJrestart.ww3: will generate restart.ww3 because prior cycle does not exist.rcZww3gintz?ww3_grid: generating mod_def.ww3 for gnh_10m grid from gdaswave../mod_def.ww3zmod_def.hafs_ww3rz{grid_gnh_10m_inp}rz./mod_def.gnh_10mzSww3_gint: generating restart.ww3 by using ww3_gint with restart files from gdaswaveZww3_gintz ./ww3_gintz ww3_gint.logz./restart.hafs_ww3z../restart.ww3zTWARNING: restart.ww3: will generate dummy because ww3_gint did not run successfully.exc_infoNz+restart.ww3: generating dummy with ww3_strtz {strt_inp}z ww3_strt.inpZww3_strtz ./ww3_strt ww3_strt.logz,ww3_bound: generating ww3 boundary conditionz {bound_inp}z ww3_bound.inpZ ww3_boundz ./ww3_bound ww3_bound.logr9ziWARNING: ww3_bound: will run without input boundary condition because ww3_bound did not run successfully.z FATAL ERROR: WW3 init failed: %s)7r|r@warningrstateconfboolr workdirZscrubr icstrgetexerrrDfileop remove_filerrrinfor splitlinesresearch isnonemptyjloggeropenrangewriteosenvirongetlowerr}launcher HAFSLauncherreadgetboolgetstrcriticalrrr get_ww3rst_inputs make_gint_inp Exceptionget_ww3bdy_inputsr<poprArrstrr)r-Z dummycurrrZ dummywindr6rbrdrcmdZncfilefsubsetlineZrunmeofxZ have_restartZoldrstZ oldconffileZoldconfnameddireeer4rr5rsB                                 3 z WW3Init.runcCs@|}dD].}|jd|d}tj|r t||d|dq dS)N)rrrrww3_untarbdy.logrz!{com}/{out_prefix}.{RUN}.{lf}.ww3)lfTr)r|rrpathexistsr )r-rrZcomlocr4r4r5 _copy_logys  zWW3Init._copy_logcCsf|}|dd}t|jj}td|}|dd}tj |j||}|dd}|dd }|} t d D]} td| |} |j ||| |d } | s| d | d fd} tj| \} }|dkr|d| fd} |jdkr|d| dfd} | sqld| dd}t| |d|d|d}tdd|d|f}|rP|dk}t||ddSdS)z!Obtains WW3 input boundary condition data, links or copies to ww3init dir. WW3 input boundary data comes from previous cycle's gfswave.rTr\rrrsr]r^r_r`rHr.%s: cannot decide data location for this time.%Y%m%d%HN%s: does not existF'$%s: too small (should be >=%d bytes)zgfs.tz%Hz z.ibp_tarrz./gfswave.HWRF{vit[basin1lc]}*tarz-xvfz --wildcardsrr)r|rrr/r?rr@r}r~rrrrstrftimerDr lstat_statrst_sizer rrr)r-rrrHrp ww3catalogww3dcrYrZroZitryZ ww3bdyfileokLSZ ww3bdyspectarZ ww3bdyfbaserr4r4r5rsJ          zWW3Init.get_ww3bdy_inputsc Cs|}t|jj}td|}|dd}tj|j||}|dd}|dd}|}|j ||||d} | s| d | d fd } t j | \} } | d kr| d | fd} | jdkr| d| dfd} | s| d| d ft| dd |dd S)z!Obtains global gdaswave restart file, links or copies to ww3init dir. WW3 input restart file comes from current cycle's gdaswave.r\rrrsrdrerfrgrrrTNrFrrzrr/r?r.rBww3postprodnamesrCrrDrErr#rFrG)r-rHrIrJr:rKrLr4r4r5r&s zWW3Post._make_productsNccs8|jD](\}}|\}}|dks,||jkr |Vq dSrMrN)r-rOrIrPrLr:r4r4r5rQszWW3Post.productscCs6|j}|}tt|dddd||fdk|ddS)Nz-Oz-4z-L6z /dev/nullr) ncks_pathr|rr)r-rtargetignorencksrr4r4r5Z __copy_nckss zWW3Post.__copy_nckscCsV|jdkrP|dd}|js,tjjddd}|dksJt|trF|dksJt||_|jS)zReturns the path to ncks. Returns None if ncks cannot be found. This function will only search for ncks once, and will cache the result. Set self._ncks_path=False to force a recheck.Frr) raise_missingN)rrrDrfind_exe isinstancerAssertionError)r-rr4r4r5r  s  zWW3Post.ncks_pathc3 Cs |}|dd}t|_tjddddg}|d} zt |j d|dd}|d }t |d d|d |j d kr|d }t |dd|d |j d\}} |j| |j|dd|jd kr|d} t | dd|d |j d\}} |j| |j|dd|jdddd} | dkr |j d kr t|ddd|d ||td} |rX| dk} t| |dd} |d}|dtt|d | k|dtd!| |j d"\}} |j| |j|dd||j|d#d$|j d%\}} |j| |j|dd||j|d&d$|jd'ddd}|jd(ddd}|jd krt|d)d)d|d ||td*} | d+k} t| |dd+}t|}|}W5QRXd,d-t|D}||d d.|d.d/}|dkrg}g}g}g}t}t|D]V\}}|d.}| d }| d.}| d/} |d0|||| ft d1|fd|d2}!|!||td3d d|d td4dd|d t|d)d)d|d |d5}"|d6}#|d7}$|d8}%d9|d:}&|"|"|"|#|"|%|"|&d;|!j#d<d=|&d<d>|"d?d>|#d?d>|$d@|%d<dA} |"| W5QRXqdB}'t|'dC}(|($dD%|W5QRXtj&dE})|dF|)tj&'dGdH}*|*dHkr|dI}*t(t)|*dJ|'fddK}+t|+tdLdM|} t| |dtdLdN|} t| |dtdLdO|} t| |dtdP|dQ?} t| |d|j dR\}} |j| |j|dd||j|dSd$|j dT\}} |j| |j|dd||j|dSd$|j dU\}} |j| |j|dd||j|dSd$|dkrg},g}t}|j*j+,dV}-t|D]\}}|d.}| d }| d.}| d/} |dW|||| ft dX|fd|d2}!|-||td3d d|d td4dd|d t|d)d)d|d dY|-dZ}.|dZ}/d[|d:}&|,"|/|"|&d;|!j#d<d=|&d<d>|.d@|/d<dA} |"| W5QRXqd\}'t|'dC}(|($dD%|W5QRXtj&dE})|d]|)tj&'dGdH}*|*dHkr|dI}*t(t)|*dJ|'fddK}+t|+tdLd^|,} t| |dtdP|d_?} t| |d|j d`\}} |j| |j|dd||j|dSd$|jdaddd}0|0dkr|j d krt|dbdbd|d |.|tdc} |rj| ddk} t| |d|j de\}} |df| |jf|j| |j||j/d|jdgddd}1|1dk r:|jd k r:t|dhdhd|d |0|tdi} | r | djk} t| |d|j dk\}} |j| |j||j/dW5QRXt1|_WnNt2k r}2z.t3|_|j4dlt5|2fddmt67d/W5d}2~2XYnXdS)nzRun the WW3 post.rTZMODELz{type}z{job}z {location}z{RUN}rrz mod_def.ww3rrz{WORKhafs}/forecast/out_grd.ww3z out_grd.ww3rN)rrGrcopierz{WORKhafs}/forecast/out_pnt.ww3z out_pnt.ww3r ww3_grib_postr8Zww3post)r0Zww3_gribz ./ww3_gribz ww3_grib.logrz gribfile.idxrz.ww3post: Generating grib idx file for gribfile)z-sZgribfilez chmod -x rZ_WW3GB2)rGtyperZ _WW3GB2_WIDXww3_outp_bull_postww3_outp_spec_postZww3_outpz ./ww3_outpzww3_outp_info.logcSsg|]\}}d|kr|qS)z ----------r4).0ielemr4r4r5 SszWW3Post.run..rur{z&ww3_outp_bull for buoy: %i, %s, %s, %szww3outpbull.%srrz../out_pnt.ww3z.bullz.cbullz.csvz.csbullZww3_outp_bull_z.logzcd z && z./ww3_outp > ../zmv z ../ && z ../zcd ../zcommand.file.ww3outpbullr Z TOTAL_TASKSz ww3_outp_bull total threads: %s MPISERIALz *MISSING* mpiserialz-m)Zallranksr)-cvfz ww3_bull.tar)rz ww3_cbull.tar)rzww3_csbull.tarcatzww3_outp_bull.logrZ_WW3TARrrz%y%m%d%Hz&ww3_outp_spec for buoy: %i, %s, %s, %szww3outpspec.%szww3.z.spcZww3_outp_spec_zcommand.file.ww3outpspecz ww3_outp_spec total threads: %s )rz ww3_spec.tarzww3_outp_spec.logr ww3_ounf_postZww3_ounfz ./ww3_ounfz ww3_ounf.logrz Delivering ww3ounf from %s to %sww3_ounp_spec_postZww3_ounpz ./ww3_ounpz ww3_ounp.logrz FATAL ERROR: WW3 post failed: %sr)8r|rrrrDdbnalertr rupperr rr r*r<rrGr+r@r r make_grib_inprrrrrsystemmake_outp_info_inpr readlines enumeratelistsplitmake_outp_bull_inpappenddirnamerjoinrrrrr/r?rmake_outp_spec_inp make_ounf_inp_WW3Post__copy_ncksmake_ounp_spec_inprrrrrrr)3r-rrZalerterZmodelrunrZ ww3moddefZww3outrLr:Zww3pntrrZ indexfilerrrfnamerZ ww3_outp_infoindicesZbuoysZfilebullZ filecbullZ filecsbullZfilelogcommandsrZbuoyipntZbuoyidZbuoylonZbuoylatrZbuoybullZ buoycbullZbuoycsvZ buoycsbullZbuoylogZcmdfnameZcfpfthreadsZmpiserial_pathZcmd2fileoutZww3tstrZbuoyspcZbuoyoutr!r"rr4r4r5rs                                                         z WW3Post.runc Cstj|j|j}|dd}|s,|d}t|jj}t }|j | dt |j | ddt|dB}tdd,}||j|f||d |jjd |W5QRXW5QRXdS) Ngrib_inprz{PARMww3}/ww3_grib.inp_tmplr)FLD_BEGFLD_DTrrz ww3_grib.inprTrr}rrr/r0r@rrr?r;rrr)r*rrr)r-rrr:rHrrrr4r4r5r%s$      zWW3Post.make_grib_inpc Cstj|j|j}|dd}|s,|d}t|jj}t }|j | dt |j dt|dB}tdd,}||j|f||d |jjd |W5QRXW5QRXdS) Nounf_inprz{PARMww3}/ww3_ounf.inp_tmplr)r;r<rz ww3_ounf.inprTrr=)r-rrr>rHrrrr4r4r5r1s"      zWW3Post.make_ounf_inpc Cstj|j|j}|dd}|s,|d}t|jj}t }|j | dt |j dt|dB}tdd,}||j|f||d |jjd |W5QRXW5QRXdS) N ounp_spec_inprz {PARMww3}/ww3_ounp_spec.inp_tmplrPNT_BEGPNT_DTrz ww3_ounp.inprTrr}rrr/r0r@rrr?r;rrr)r+rrr)r-rrr?rHrrrr4r4r5r3s"      zWW3Post.make_ounp_spec_inpc Cstj|j|j}|dd}|s,|d}t|jj}t }|j | dt |j dt|dB}tdd,}||j|f||d |jjd |W5QRXW5QRXdS) N outp_info_inprz {PARMww3}/ww3_outp_info.inp_tmplrr@r ww3_outp.inprTrrC)r-rrrDrHrrrr4r4r5r' s"      zWW3Post.make_outp_info_inpc Cstj|j|j}|dd}|s,|d}t|jj}t }|j | dt |j t || ddt|dB}tdd,}||j|f||d |jjd |W5QRXW5QRXdS) N outp_bull_inprz {PARMww3}/ww3_outp_bull.inp_tmplrrArBZPNT_NUMrrrErTrrC) r-r7rrrFrHrrrr4r4r5r,s&      zWW3Post.make_outp_bull_inpc Cstj|j|j}|dd}|s,|d}t|jj}t }|j | dt |j t || ddt|dB}tdd,}||j|f||d |jjd |W5QRXW5QRXdS) N outp_spec_inprz {PARMww3}/ww3_outp_spec.inp_tmplrrGrrErTrrC) r-r7rrrHrHrrrr4r4r5r0*s&      zWW3Post.make_outp_spec_inp)r!rr)N)rrr__doc__r%r&rQr2propertyr rr%r1r3r'r,r0rr4r4r2r5rs  I)3rI__all__rrrprodutil.datastorerDprodutil.fileop produtil.cd produtil.run produtil.logprodutil.dbnalerttcutil.numericsr=Z hafs.hafstaskr}hafs.exceptionsZ hafs.namelistZ hafs.input hafs.launcher hafs.configrrrrrr r r r r rrrrrrrrrrrrrrrAZhafstaskZHAFSTaskrr rr4r4r4r5sJ( (  ;