U $gǑ@sHdZdddddgZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZ ddlZddlZddlZddlZddlmZdd lmZmZdd l mZmZmZmZmZdd l mZdd l Tdd l Tdd lmZdd lTddZdddZe Z!Gdddej"jZ#GdddeZ$GdddeZ%dS)zu!Runs the Unified Post Processor on outputs from the WRF-NMM, producing E grid GRIB files as EGRIB1Product objects. PostOneWRF PostManyWRF EGRIB1Product check_post link_post_fixN) PostFailed) GRIB1File UpstreamGRIB1) FileProduct COMPLETEDFAILEDRUNNING UNSTARTED)TempDir)*)HWRFTaskc Cs|dkr|d||ftdd}|dtj|}td|}|| tj||}d|dkr|d|||f| }|d d D]}|d |fqW5QRd SW5QRXtj drtdd }W5QRXd } d } d} t j ds4|d|fttddttddd Stdd }|dD]} zFtd| } | r| \}}|dkrt|} |dkrt|} WnJttttfk r}z"| sd} |jd|fddW5d }~XYnXqJW5QRX| d kr|d|f| d kr"|d|f| d k o2| d k }tdD](}|d|| | |f|| | |fS|d|fttddttddd S) a!Did the post run successfully in the current working directory? Checks the current working directory and the specified return value retval from the post to determine if the post succeeded. Returns a four-element tuple (ok,cenla,cenlo,filename) where "ok" is True if the post succeded, (cenla,cenlo) is the domain center and filename is the name of the post output file. @param retval the post return value @param what for log messages: what was posted? @param logger the logging.Logger for log messagesrz%%s:Non-zero exit status %d from post. vpost.logrbizALL GRIDS PROCESSEDzj%s: Did not find "ALL GRIDS PROCESSED" in last %d bytes of vpost.log (file size %d). Post probably failediNz vpost.log: %s)FNNNrtFthis-domain-center.ksh.inczG%s: this-domain-center.ksh.inc is empty or does not exist. WRF failed.Zlsz-ltrtail)z-20rz (clat|clon)\s*=\s*([+-]?[.0-9]+)ZclatZclonTz6%s: Exception while parsing this-domain-center.ksh.incexc_infoz5%s: Could not get clat from this-domain-center.ksh.inz5%s: Could not get clon from this-domain-center.ksh.inzWRFPRS*z'%s: success: cenla=%s cenlo=%s file=%s.z%s: No WRFPRS* files found.)warningopenseekosSEEK_ENDtellminreadfind splitlinespathexistsprodutilfileop isnonemptyerrorrunexe readlinesresearchgroupsfloatKeyError ValueError TypeErrorAttributeErrorglobinfo)retvalwhatloggerffilesizereadsizeZvpostdatZlinsZlincenlacenloZsentexclinemZvarrwhereeokfilenamerE7/lfs/h1/ops/prod/packages/hwrf.v13.2.9/ush/hwrf/post.pyrs            &   FcCs8|dk r&|dttt|fdD]*}tj|dd}tjj |||dddq*|r4tj|d }g}d D](}| d ||f| d ||fqrd D]}| d|||fqd}| d|||f| d|||f| d|||f| d|||f| d|||ft |d|d|ddS)z!Links or copies all fix files for the post to the current working directory. @param fixd the fix directory @param needcrtm flag: is the CRTM data needed? @param logger the logging.Logger for log messagesNz*%s: link post fix files here. Needcrtm=%s)zeta_micro_lookup.datznam_micro_lookup.datzhires_micro_lookup.datzhwrf-wrfzETAMPNEW_DATA.expanded_rainTF)r9keeppreserve_permszhwrf-crtm-2.2.3)Z amsre_aquaZimgr_g11Zimgr_g12Zimgr_g13Zimgr_g15Z imgr_mt1rZimgr_mt2Z seviri_m10Zssmi_f13Zssmi_f14Zssmi_f15Z ssmis_f16Z ssmis_f17Z ssmis_f18Z ssmis_f19Z ssmis_f20Ztmi_trmmz v.seviri_m10Z imgr_insat3dZ ahi_himawari8z&%s/SpcCoeff/Big_Endian/%s.SpcCoeff.binz+%s/TauCoeff/ODPS/Big_Endian/%s.TauCoeff.bin)ZAerosolZCloudz!%s/%sCoeff/Big_Endian/%sCoeff.binZEmiszB%s/%sCoeff/IR_Land/SEcategory/Big_Endian/NPOESS.IRland.%sCoeff.binzB%s/%sCoeff/IR_Snow/SEcategory/Big_Endian/NPOESS.IRsnow.%sCoeff.binz@%s/%sCoeff/IR_Ice/SEcategory/Big_Endian/NPOESS.IRice.%sCoeff.binz8%s/%sCoeff/IR_Water/Big_Endian/Nalli.IRwater.%sCoeff.binz:%s/%sCoeff/MW_Water/Big_Endian/FASTEM6.MWwater.%sCoeff.bin.)r9forcecopy) r6strrgetcwdreprr$joinr&r' deliver_fileappendmake_symlinks_in)fixdneedcrtmr9rKtgtsrcZcrtmdZlinksrErErFrms6  c@sVeZdZdZdddZddZddZd d Zd d Ze eddd Z e edddZ dS)rz!Represents an E grid WRF-NMM GRIB1 file, and stores two metadata values: CENLA and CENLO which contain the domain center location.Nc Cs|dk s ttj|}tj|s.t|d}tj|rntj|}t}||}|dkrn|dkrnd}|rt|d|d|d|d} |d } | dk st| dk st|j B} | |d<| |d <| |d| d|kstd |kstW5QRX|j |d dS) at!Copies the file to its destination, and sets the CENLA and CENLO metadata values to the domain center. @param location the destination location @param fileinfo a dict containing fromloc, the location; CENLA, the domain center latitude; and CENLO, the domain center longitude. @param logger a logging.Logger for log messagesNTxFfromlocrGr9CENLACENLOr9)AssertionErrorrr$dirnamer%makedirsgetmtimetimerPdstore transaction set_loc_availupdatecall_callbacks) selflocationZfileinfor9r_Z need_to_copyZfile_modified_timeZtime_nowZ time_diffr=r>trErErFdelivers2           zEGRIB1Product.delivercCstjtj||j|jS)zy!Decides a filename which is of the format "outdir/category/prodname". @param outdir the output directory)rr$rOcategoryprodname)rhoutdirrErErF make_locationszEGRIB1Product.make_locationcOsL|j}|dtj|\}}tjj|||jdt |dd|d|dS)z!Delivers the file. @returns a hwrf.regrib.GRIB1File for the file. @param regrib the hwrf.regrib.Regrib with input information @param args,kwargs ignoredzprod.r]Nr[r\) riZgribtemprr$basenamer&r'rPr9r)rhregribargskwargslocrDindexrErErFmakeszEGRIB1Product.makecCs|dS)z!The domain center latitude.r[rErhrErErF getnscenterszEGRIB1Product.getnscentercCs|dS)z!The domain center longitude.r\rErwrErErF getewcenterszEGRIB1Product.getewcenterz7Returns None or the center latitude of this GRIB1 file.z8Returns None or the center longitude of this GRIB1 file.)N) __name__ __module__ __qualname____doc__rkrorvrxrypropertyZnscenterZewcenterrErErErFrs ! cseZdZdZd#fdd Zdd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ddZddZddZd$d!d"ZZS)%rz}!This is an HWRFTask that post-processes output data for a single WRF stream, from several WRF domains at a a given time.historyTNc  s@tt|j|j|j|fd| i| ||_||_t|| |_ ||_ ||_ i|_ i|_|dk|_|snt|dkrzdnd} |dkrtdd} |j D]}d}|dk st|dk st|j|g||j d D]b}d} |std}||j |<|d |jt|t|f||}t|j|j|d |j|<qq| s %s)rlrm)superr__init__rcconf_PostOneWRF__needcrtm_PostOneWRF__wrfto_datetime_relwrfsimstart_PostOneWRF__time_PostOneWRF__domainsZ_PostOneWRF__stream_PostOneWRF__wrfproducts_PostOneWRF__myproducts_PostOneWRF__grib2r^NotImplementedErrorproductslogdebugrrN product_namer)rhrrrsectionrbrrTgribZfaketimerrsZprodextZaddeddomainfirstproductrm __class__rErFrsN         zPostOneWRF.__init__cCsJ|jr dnd}|dkr&d|j|f}nd|j|j|f}tdd|}|S)z!Returns a human readable string representation of the product name for the given domain. This is used for filenames and product ids. @param domain the domain of interesrrNz%s.%sz%s-%s.%sz[^a-zA-Z0-9_.-]_)rrnamer-sub)rhrextresultrErErFrs zPostOneWRF.product_namecCs |jS)z"!Returns the WRFSimulation object.)rrrwrErErFr!szPostOneWRF.wrfcCs|jS)z=!Returns the Task that represents the running WRF simulation.)rrwrErErFwrftask$szPostOneWRF.wrftaskcos|d|kr.|dD]}||jkr|j|VqnJd|kr`|jD] }|j|djkr<|j|VqrDmsgrBrErErFr*s>                              zPostOneWRF.run)rTrNN)FF)rzr{r|r}rrrrrrrrrrrrrr* __classcell__rErErrFrs(/  cseZdZdZedddgdddffdd Zdd Zd d Zd d ZddZ ddZ ddZ ddZ ddZ ddZddZddZddZd d!Zd"d#Zd$d%Zd(d&d'ZZS))rz>!A wrapper around PostOneWRF that posts many WRF output times.NrTrc  sFttj|j|j|fd| i| | _|_g_t_ |_ |d | d\}}}t }| D]f}zH|D]4} | |\}}}||kr|}||krx|}qx||Wqntjjk rYqnXqn|} |dkr|n|}|dkr|n|}|}t||}t||}t|}|_fdd|D_| ||dS)a0!PostManyWRF constructor @param wrf the hwrf.fcsttask.WRFTaskBase or subclass, whose data is being posted @param domains list of domains to post, as hwrf.wrf.WRFDomain objects @param conf the hwrf.config.HWRFConfig that provides configuration ifnormation @param section the config section in conf @param step time step between post input times @param postclass should be PostOneWRF @param start the first time to post @param end the last time to post @param streams the streams to consider posting @param needcrtm do we need CRTM fix files? @param grib GRIB version: 1 or 2 @param taskname the task name in the database @param kwargs additional keyword arguments passed to PostOneWRF.__init__ rrNcsg|]}|qSrE)r).0rrwrErF Nsz(PostManyWRF.__init__..)rrrrcr _needcrtm_PostManyWRF__wrf _subtasksrZ_PostManyWRF__done _postclassrget_output_rangelistrQhwrf exceptionsOutputStreamDisabledr to_timedelta_step_PostManyWRF__mydomains _add_subtasks)rhrrrrstepZ postclassstartendstreamsrTrrrsistartZiendZ iintervalZ keepstreamsrrjstartZjendZ jintervalintervalrrwrFr sN     zPostManyWRF.__init__cCs|j}|}t|d}t}|}t||} || krt} |D]} |d| d} |D]} | | f}| j| ||d| |<| |dkrq`|dt| t| t|t| |f| | }t t||dd}||kr`|d t|t|t|t|f| }d} qq`| rBq&qB| r|d | |t| t|f|j ||j|d d |D|j|j|||j| |d f| }n|d|df||7}q.|dtt|j dS)ao!Generate subtasks Fills the self._subtasks array. It figures out which times have data for all domains and creates a PostOneWRF object for each of those times. The subtasks are created in temporal order. @param streams the list of streams to consider @param start,end start and end of the range of times to considerrz check stream %sFr]Nz' domain %s stream %s time %s result %sTnegokz dt=%s-%s=%s > epsilon=%sz%Adding post %s with stream %s time %scSsg|]}|qSrErE)rxrErErFr|sz-PostManyWRF._add_subtasks..)rrTrz>%s: ignoring duplicate output time due to WRF output frequencyz%Y-%m-%d %H:%M:%Szlen(self._subtasks)=%s)r to_fractionrrrrr get_outputrN validtimeabs taskname_forrrQrr_conf_sectionrrr)rhrrrrwhenepsilonlastr9ZendethisrrCrkeyZattimedtZokstreamrErErFrRsv        zPostManyWRF._add_subtasksccs|jD] }|VqdS)z%!Iterates over all WRFDomain objects.N)rrhrrErErFrs zPostManyWRF.domainsccs8tt|jD]$}||j|d|j|dfVqdS)a4!Iterate over all subtasks Iterator that loops over all subtasks yielding a tuple: @code (itask,rtime,subtask) @endcode Where: * itask = task index from 0 to ntasks-1 * rtime = output time this task processes * subtask = the Task objectrrN)rangerrrrErErFsubtaskss zPostManyWRF.subtaskscCs"||D] }|qdS)z1!Calls uncomplete, and then deletes all products.N) uncompleter undeliver)rhrrErErFunruns zPostManyWRF.unruncCs$t|_|D]\}}}t|_qdS)z!Marks this task and all subtasks as incomplete so that all post-processing will be rerun. Does not undeliver any delivered products.N)rrr)rhitaskrtimesubtaskrErErFrszPostManyWRF.uncompletecCs&t||\}}d|j||fS)zo!Returns a human-readable taskname for the given subtask time. @param time the time of interestz%s-f%02dh%02dm) wrf_hr_minrrr)rhrbihoursiminutesrErErFr szPostManyWRF.taskname_forcCs t|jS)z !Returns the number of subtasks.)rrrwrErErF subtask_countszPostManyWRF.subtask_countcCs|jddS)z'!Returns the first time to be processedrr)rrwrErErF starttimeszPostManyWRF.starttimecCs|jt|jddS)z&!Returns the last time to be processedr)rrrwrErErFendtimeszPostManyWRF.endtimecCs |jS)z$!Returns the WRF object being posted)rrrwrErErFrszPostManyWRF.wrfcCs|jS)z!Returns the Task that ran WRF)rrwrErErFrszPostManyWRF.wrftaskcCs@d}|D].\}}}|jr6|dkr(|}||kr:|}q qs.8     P 2B*