U  gm@sldZddgZddlZddlZddlZddlZddlZddlZddl Zddl Zddl Zddl Z ddlZ ddlZ ddlZ ddlZ ddlZddlZddl mZmZmZddlmZmZmZmZmZddlmZmZmZddl m Z dd l m!Z!m"Z"dd lm#Z#m$Z$m%Z%dd lm&Z&dd lm'Z'd dddddddZ(Gddde j)j*Z+Gddde j,j-Z.Gddde j,j/Z0dS)aj!Runs the POM initialization and POM-WRF coupled forecast. This module handles the POM-coupled WRF simulation. It contains two critical pieces: * POMInit -- an HWRFTask that is a wrapper around the Python pom package. * WRFCoupledPOM - a subclass of hwrf.fcsttask.WRFAtmos that runs the WRF-POM two-way coupled system based on the output of the POMInit.POMInit WRFCoupledPOMN) setrlimitrusage getrlimit) UpstreamFilewait_for_products COMPLETEDRUNNINGFAILED) isnonempty make_symlink deliver_file)NamedDir)mpirunmpi) to_datetimeto_datetime_rel to_fraction)OceanInitFailed) POMInputError)z#{oceandir}/{vit[stormname]}.grid.ncz!{outdir}/{out_prefix}.pom.grid.nc)z){oceandir}/{vit[stormname]}.ts_initial.ncz'{outdir}/{out_prefix}.pom.ts_initial.nc)z&{oceandir}/{vit[stormname]}.ts_clim.ncz${outdir}/{out_prefix}.pom.ts_clim.nc)z){oceandir}/{vit[stormname]}.uv_initial.ncz'{outdir}/{out_prefix}.pom.uv_initial.nc)z){oceandir}/{vit[stormname]}.el_initial.ncz'{outdir}/{out_prefix}.pom.el_initial.nc)z{oceandir}/restart.phase2.ncz*{outdir}/{out_prefix}.pom.restart.phse2.nc)z{nmldir}/pom.nmlz{outdir}/{out_prefix}.pom.nml)gridZ ts_initialZts_climZ uv_initialZ el_initialzrestart.phase2zpom.nmlcseZdZdZd fdd ZddZdd Zd!d d Zd dZd"ddZ e ddZ e ddZ d#ddZ ddZddZddZddZZS)$rz!A wrapper around the pom package that runs the POM initialization. This HWRFTask subclass is a wrapper around the pom package. It runs the POM initialization, and generates the POM namelist for the forecast run.NQc  stt|j|||f||dd||d|_|d|d<|jddksVtt|d| dd|_ t|d| dd|_ t|d | d d |_ t|d | d d |_ t|d | d d|_t|d| dd|_t|d| dd|_t|j|_||_t||_|jdkr:d|_|dkrN|d}||_d|kr~t|dtjjr~|d|_n,t|d| dd} tj|| |j|_t|_|j } t!j"#| d} |$| dS)aX!Creates a POMInit. @param dstore the produtil.datastore.Datastore to use @param conf the HWRFConfig to use @param section the section name for this task @param taskname the task name. Default: section @param vitfile the vitals file with tcvitals for all times this storm has existed. Default: self.icstr('{WORKhwrf}/{stormlabel}.vitals') @param fcstlen The forecast length in hours. @param outstep The output timestep in seconds. @param kwargs Other keyword arguments are passed to the superclass constructor.com)tasknameoutdirrintercomrZ sfc_datasethwrfdataZ loop_datasetZ sfcanl_item gfs_sfcanlZ sanl_itemZgfs_sanlZ loop_itemZ gfdl_loopZ wc_ring_itemZ gfdl_wc_ringrtofs_init_basinsErNz{WORKhwrf}/{stormlabel}.vitalscatalogoutput)%superr__init__getdirrfindAssertionErrorstrgetconfstr _sfc_dataset _loop_dataset _sfcanl_item _sanl_item _loop_item _wc_ring_item_rtofs_init_basinsrcycle_atime_POMInit__fcstlenint_POMInit__outstepicstr_vitfile isinstancehwrfinput DataCatalog_catalogdict _productsworkdirospathjoin_make_products) selfdstoreconfsectionrvitfilefcstlenoutstepkwargsZincatrundir outputdir __class__:/lfs/h1/ops/prod/packages/hmon.v3.2.7/ush/hwrf/mpipomtc.pyr%?sr                       zPOMInit.__init__c CsRzt|_|}|j}|sttj|d}tj|d}tj|rPt |t |d|dd}t |d|d}| W5QRXt |d|d(}| |||tj|d|W5QRXW5QRXt|_Wn|tjjk r }z|dt|_W5d}~XYnDtk rL}z$|jdt|fdd t|_W5d}~XYnXdS) zl!Runs the POM initialization and copies the results to their destinations within the HWRF work area.r<r#TkeeploggerOCEANzBasin is unsupported.Nz%Unhandled exception in ocean init: %s)exc_info)r statelogrAr(rBrCrDexistsshutilrmtreer get_inputsrun_initdeliver_productsr pom exceptionsZPOMUnsupportedBasininfo Exceptionerrorr)r )rFrVrNinputdirrOdZueerRrRrSrunxs<     z POMInit.runc Cs|j}tj|d}|jtD]\}}|\}}tj |j||j }|j ddks`t|j||||j d|_|j ddkst|j|||||d} | |d<|| f|j|<q(W5QRXdS)z!Creates FileProduct objects for all output files. @param outdir The directory to which the pom package output its final files.rWrr)r)oceandirnmldir localpathN)r4rBrCrDrG transaction prodnamesitemsprodutil datastore FileProductrrr'r(timestrlocationr@) rFratimerjprodnameZ filepathsrlZcompathprodZ slocalpathrRrRrSrEs*   zPOMInit._make_productsTc Cs:|js t|}d}t}|j}tjj|j|d|j D]\}} t | t sTt| \} } | j r|s| d| jt| j t| jfq>tj| s|| dd}|| dq>n(t| s|| dd|| d | j| d|d q>t|d kr6d d |} || |s6|d| t| dS)al!Delivers files to their final destination Copies results to their destinations within the HWRF work areas @param oceandir the OCEAN directory created by the pom package in run() @param nmldir the directory in which the forecast namelist was made @param redeliver if True, products are re-copied even if available=TrueTrVzV%s: already available and redeliver=False, so skipping this (available=%s location=%s)z&: expected output file does not exist.Fz (missing)z!: is empty. Will deliver anyway.z Beware of impending failures.z (empty))frominforUrVrz*Some ocean outputs were empty or missing: z, zOcean init failed: N)r@r(rZlistr4rpfileopmakedirsrror:tuple availablercdidreprrtrBrCr[warningappendr deliverlenrDcriticalr) rFrjrkZ redeliverrVgoodZbaddiesrurvstuffrwrlmsgrRrRrSr`sL     zPOMInit.deliver_productsc Cs|jj}|d}|d}tj|dd}|}|}|} |} |d} |jd} |jj } |jj }t | |j d}|dd }|r||d <|d d kst| }t|d ddt|dtjj|| | |||||| | | f |j|jd|dS)ag!Internal function that passes control to the pom package This internal implemenentation function passes control to the pom package. This is part of the implementation of run(). Do not call directly except for debugging. @param inputdir the ocean input data directory @param outputdir the ocean data output directoryEXEChwrfPARMhwrfFIXhwrfzhwrf-pomr%Y%m%d%H)rVrHmethodZ input_methodz pom/outputrg ZAT)rVstackignorerx)rKrLN) storminfocenterupperr&rBrCrDr4strftimestormid3 stormnamer?rZrHr+r'r(rrramasterr_r5r7)rFrfrOZCENTERIDrrrZVITDIRZGFSDIRZLCDIRZCSTREAMZCOMINZ STARTDATESTORMIDZ STORMNAMErMrrVrRrRrSr_sB          zPOMInit.run_initccsB|dkr$|jD]}|dVqn||jkr>|j|dVdS)aq!Iterates over products Iterates over Product objects for all of the files that need to be copied to the forecast directory to run POM. The products will all have a "localname" metadata value telling the local filename they should have in the forecast directory. @param name If given, only the product with this name is yieldedNr)r@values)rFnameprRrRrSproductss  zPOMInit.productscCs|jS)z`!Return a string containing the list of one-letter basins that use RTOFS initialization.)r2rFrRrRrSrszPOMInit.rtofs_init_basinscCs|jj}t|j|j|jS)z9!Get the analysis time of the parent ocean model (RTOFS).)rHr3datetimeyearmonthday)rFcycrRrRrSparent_ocean_atimeszPOMInit.parent_ocean_atimercCs|j}|dkr$tt|dd|}t|jj|}|dd}|dkrpt|dkrb|dd}q|d d }n&t|d kr|dd}n |d d }|d d }|||fS)aY!Get the atime, dataset and item of the parent model input, for hwrf.input.DataCatalog @return a tuple (oceanatime,dataset,item) suitable for hwrf.input.DataCatalog.get() @param idays A positive integer; the number of days to subtract from the RTOFS analysis time. Negative numbers are ignored.gGz?ii rt00zonlyrrZ ocean_nowZ rtofs_nowZ ocean_fcstZ rtofs_fcsti,Z ocean_datasetZrtofs) rrroundrrHr3confintabsr+)rFZidays oceanatimedtr ocean_itemoceandsrRrRrSparent_ocean_adi s      zPOMInit.parent_ocean_adiccs|jj}|d}|dd}|dkrn|jj}|\}}}t||||ddVt||||ddVt|j|j |j dVt|j|j |j dV|d krt|j |j |j dVt|j |j|j dVd S) z%!Iterates over all needed input data.ini_datagdemRTOFa)datasetitemruftimeabb)rrruLN)rrrr+rHr3rr?r,r/r4r.r-r0r1)rFrBASIN init_data hwrfatimerrrrRrRrS inputiter)s<      zPOMInit.inputiterc Cs|}|jj}|d}||dd}|dkrH|d|jjjf}| d|j |ft |dB}t |j d*}|D]}| d d krq| |qW5QRXW5QRXd S) zt!Obtains input data using get_rtofs_inputs and/or get_gdem_inputs as appropriate, and pre-processes tcvitalsrrrrzsyndat_tcvitals.%04dzCopy vitals %s to %swtrtINVESTrN)rZrrrget_gdem_inputsr+get_rtofs_inputswhenrrcr9openr'write) rFrVrrrZvitdestoutfinflinerRrRrSr^Ls   zPOMInit.get_inputsc Cs|}|jj}|j}|dd}tdD]`}||\}}}|dkrv|jj||||dd} |jj||||dd} n0|jj|||||dd} |jj|||||dd} | r| s| d| d fd } | d g| d gfD]b\} } t j | \}}|d kr| d| fd} q<|j| kr| d| | fd} q=%d bytes)zrtofs_glo.t00z.f%02d.archv.%czrtofs_glo.t00z.n%02d.archv.%cforcerV)rZrHr3hourrrangerr>locatercrrpr{ lstat_statst_sizer )rFrVrfhrritryrrrZafilebfileokZxfileZxsizerSprefixrRrRrSrbsf        zPOMInit.get_rtofs_inputsc s|}|j}|jv}|jj|j|j||d}|jj|j|j||d}t |jd|j dd}||_ d|_ t |jd|j dd}||_ d|_ W5QRX||j dd|jd ||j d d|jd ifd d }d d} t||g||| td|dd} d} t| D]} | d} t| |}|d}|jj|j|j||d}|jj|j|j||d}d} t|sxd} |dt||ft|sd} |dt||f| sqq| s|j dd|jd }t|||d|j dd|jd }t|||dn|d| |dfdS)a!Obtains input data, links or copies to places expected by POM when using GDEM climatology initialization. Copies all inputs to locations expected by the pom package when it uses GDEM climatology initialization. Copies the GFS sanl and sfcanl, waiting for them if needed. Makes a new tcvitals file by parsing the old one, and generating a new one, discarding lines containing "INVEST".)rurVZ input_sanli0u)minsizeFZ input_sfcanlzgfs.t{aHH}z.sanlr)ruzgfs.t{aHH}z.sfcanlcs|S)NrR)rrVrnamesrRrSnamerz&POMInit.get_gdem_inputs..namercWst|j|d|ddS)NTr)r rt)rrrVrrRrRrSactorsz&POMInit.get_gdem_inputs..actorZloop_current_max_days_backr!Tg8@rz.%s (loop at time %s): is empty or non-existantz6%s (loop wc_ring at time %s): is empty or non-existantz&hwrf_gfdl_loop_current_rmy5.dat.{aYMD}rxz.hwrf_gfdl_loop_current_wc_ring_rmy5.dat.{aYMD}zENo loop current available. Checked %d day(s) for loop current for %sz%Y%m%dN)rZr4rGrmr>rr,r/r.rrrtr~rsrmaxrrrrr-r0r1r rr)r r)rFrVrutZsanlxZsfcanlxZsanlZsfcanlrrZmaxbackbadZideltaZhdeltaZlooptimestimeloopZwc_ringZlooplocZ wc_ringlocrRrrSrs              zPOMInit.get_gdem_inputs)NNNr)T)N)r)__name__ __module__ __qualname____doc__r%rirEr`r_rpropertyrrrrr^rr __classcell__rRrRrPrSr9s&9 (    #1c@s(eZdZdZddZddZddZdS) POMIniterz!This is an internal implementation class that should never be used directly. It instructs the hwrf.coupling.CoupledWRF to call the WRFCoupledPOM.copy_pom_inputs to check or link POM input data.cCs ||_dS)ziCreates a POMIniter that will pass control to the given WRFCoupledPOM object, stored as self.wcp.N)wcp)rFrrRrRrSr%szPOMIniter.__init__cCs|jjddS)z=Calls the WRFCoupledPOM.copy_pom_inputs with just_check=True.T) just_check)rcopy_pom_inputs)rFrVrRrRrScheck_coupled_inputsszPOMIniter.check_coupled_inputscCs|jt|S)z;Calls the WRFCoupledPOM.copy_pom_inputs passing just_check.)rrbool)rFrrVrRrRrSlink_coupled_inputsszPOMIniter.link_coupled_inputsN)rrrrr%rrrRrRrRrSrsrcsHeZdZdZdfdd ZddZed d Zd d ZdddZ Z S)ra!Runs a WRF-POM coupled simulation. Most of the work of this class is done by the superclass, WRFAtmos. This class adds code to copy the inputs needed by POM and the coupler. There are three critical new config section values: * wm3c_ranks = number of coupler ranks. Default: 1 * pom_ranks = number of POM ranks. Default: 9 * wrf_ranks = nubmer of WRF ranks. No default. This one is mandatory.Tauxhist1Nc st|ts$tdt|jt|ftt|j||||||f|||_ t |} | dddd| | dd} | dd d | | ||d d d ddddgdS)a&!WRFCoupledPOM constructor. @param dstore the produtil.datastore.Datastore to use @param conf the hwrf.config.HWRFConfig that provides configuration ifnormation @param section the config section in conf @param wrf the hwrf.wrf.WRFSimulation object that is being run @param keeprun if True, the simulation temporary files are retained @param wrfdiag_stream the stream that generates wrfdiag files @param pominit The POMInit object. @param kwargs passed to hwrf.fcsttask.WRFAtmos.__init__z]The pominit argument to WRFCoupledPOM.__init__ must be a POMInit object. You passed a %s %s.ZcouplerZ hwrf_wm3cZ wm3c_ranksr ocean_ranks rahwrf_ocean_fcstZ pom_ranksoceanrriiiFiN)r:r TypeErrortyperrr$rr%_pominitrcouple _add_waver couplewrfadd_coupled_stream) rFrGrHrIwrfkeeprunwrfdiag_streampominitrMZ pominiterrrPrRrSr%s4  zWRFCoupledPOM.__init__cCs|ddS)zl!Removes the ocean component from coupling. @post Any call to run() will not include ocean coupling.raN)uncouplerrRrRrS remove_oceanszWRFCoupledPOM.remove_oceancCs |djS)z6!Returns the POMInit object for this coupled forecast.ra) componentiniterrrRrRrSrszWRFCoupledPOM.pominitcCsdS)zp!Internal function for adding wave coupling. This must be implemented by a subclass. @protectedNrRrrRrRrSr szWRFCoupledPOM._add_waveFc Cs|}|d|jjfd}|jD]8}t|tjjsBt |j sT|j |d| dd}|j }|j }|sd|jt|t|f}|r||n||t||sd|jt|t|t|f}|r||dS||t||s>d |jt|t|t|f}|r,||dS||t||s\t|tj|d |d |d 7}q*|d krd }|r||dS||t||d|fd S)aT!Copies or checks for the inputs required by the POM model. This is an internal function used by the PomIniter class. Do not call directly. @param just_check If just_check=True, the inputs are not copied; instead, the routine just checks for them. Do not use just_check: call check_inputs instead.z'Copying POM inputs from POMInit task %srrxrlrz;POM product %s (available=%s location=%s) has no localname.zHPOM product %s (available=%s location=%s localname=%s) is not available.FzGPOM product %s (available=%s location=%s localname=%s) has no location.TrTrzQNo outputs reported by POM initialization. Did you forget to run the ocean init?z&Copied %d POM inputs. Returning True.)rZrcrrrr:rprqProductr(r~checkmetartrrrrerrrBrCbasename) rFrrVZn_copiedrw localnameavaillocrrRrRrSr&sh            zWRFCoupledPOM.copy_pom_inputs)TrN)F) rrrrr%rrrrrrrRrRrPrSrs  )1r__all__rBr\mathrprodutil.datastorerpprodutil.fileop produtil.cd produtil.runprodutil.rusage hwrf.hwrftaskr; hwrf.numericshwrf.exceptions hwrf.fcsttask hwrf.couplingZ pom.masterraZpom.exceptionsrrrrrr r r r r rrrrrrrrrrnhwrftaskHWRFTaskrcouplingComponentIniterr CoupledWRFrrRrRrRrSs:       %