U ����� g������������������������@���sH��d�Z�dddddgZddlZddlZddlZddlZddlZddlZddlZddl Z ddlZ ddlZ ddl Z ddlZ ddlZddlZddlZddlZddlmZ�dd lmZmZ�dd l mZmZmZmZmZ�ddl mZ�ddlT�ddlT�dd lmZ�ddlT�dd��Zddd�Ze Z!G�dd��dej"j�Z#G�dd��de�Z$G�dd��de�Z%dS�)zu!Runs the Unified Post Processor on outputs from the WRF-NMM, producing E grid GRIB files as EGRIB1Product objects. � PostOneWRF�PostManyWRF� EGRIB1Product� check_post� link_post_fix�����N)� PostFailed)� GRIB1File� UpstreamGRIB1)�FileProduct� COMPLETED�FAILED�RUNNING� UNSTARTED)�TempDir)�*)�HWRFTaskc�������������� ���C���s���|�dkr�|��d||�f���tdd���}|�dtj��|���}td|�}|�|�tj��|�|�}d|�d�kr�|��d|||f���|� ��}|d d ��D�]}|��d|f���q�W�5�Q�R���dS�W�5�Q�R�X�tj �d�r�tdd ��}W�5�Q�R�X�d } d } d}tj �d��s4|�d|f���ttd�d���ttd�d���dS�tdd ���}|�d�D�]�}zFt�d|�} | �r�| ���\}}|dk�r�t|�} |dk�r�t|�} W�nJ�ttttfk �r��}�z"|�s�d}|j�d|f�dd��W�5�d }~X�Y�nX��qJW�5�Q�R�X�| d k�r|��d|f���| d k�r"|��d|f���| d k �o2| d k }t�d�D�](}|�d|| | |f���|| | |f��S�|��d|f���ttd�d���ttd�d���dS�) a��!Did the post run successfully in the current working directory? Checks the current working directory and the specified return value retval from the post to determine if the post succeeded. Returns a four-element tuple (ok,cenla,cenlo,filename) where "ok" is True if the post succeded, (cenla,cenlo) is the domain center and filename is the name of the post output file. @param retval the post return value @param what for log messages: what was posted? @param logger the logging.Logger for log messagesr���z%%s:Non-zero exit status %d from post.� vpost.log�rbi��������zALL GRIDS PROCESSEDzj%s: Did not find "ALL GRIDS PROCESSED" in last %d bytes of vpost.log (file size %d). Post probably failedi����Nz vpost.log: %s)FNNN�rtF�this-domain-center.ksh.inczG%s: this-domain-center.ksh.inc is empty or does not exist. WRF failed.Zlsz-ltr�tail)z-20r���z (clat|clon)\s*=\s*([+-]?[.0-9]+)ZclatZclonTz6%s: Exception while parsing this-domain-center.ksh.inc��exc_infoz5%s: Could not get clat from this-domain-center.ksh.inz5%s: Could not get clon from this-domain-center.ksh.inzWRFPRS*z'%s: success: cenla=%s cenlo=%s file=%s.z%s: No WRFPRS* files found.)�warning�open�seek�os�SEEK_END�tell�min�read�find� splitlines�path�exists�produtil�fileop� isnonempty�error�run�exe� readlines�re�search�groups�float�KeyError� ValueError� TypeError�AttributeError�glob�info)�retval�what�logger�fZfilesize�readsizeZvpostdatZlinsZlin�cenla�cenloZsentexc�line�mZvarr�where�e�ok�filename��rD����6/lfs/h1/ops/prod/packages/hmon.v3.2.7/ush/hwrf/post.pyr������s����� �� � � ���& � � �Fc�����������������C���s����|dk r&|��dtt����t|�f���dD�]*}tj�|�dd�}tjj |||ddd��q*|r�tj�|�d �}g�}d D�](}|� d||f���|� d||f���qpd D�]}|� d|||f���q�t|d|d|d��dS�)z�!Links or copies all fix files for the post to the current working directory. @param fixd the fix directory @param needcrtm flag: is the CRTM data needed? @param logger the logging.Logger for log messagesNz*%s: link post fix files here. Needcrtm=%s)zeta_micro_lookup.datznam_micro_lookup.datzhires_micro_lookup.datzhwrf-wrfzETAMPNEW_DATA.expanded_rainTF)r9����keep�preserve_permszhwrf-crtm-2.0.6)Z amsre_aquaZimgr_g11Zimgr_g12Zimgr_g13Zimgr_g15Z imgr_mt1rZimgr_mt2Z seviri_m10Zssmi_f13Zssmi_f14Zssmi_f15Z ssmis_f16Z ssmis_f17Z ssmis_f18Z ssmis_f19Z ssmis_f20Ztmi_trmmzv.seviri_m10Zimgr_insat3dZ ahi_himawari8z&%s/SpcCoeff/Big_Endian/%s.SpcCoeff.binz&%s/TauCoeff/Big_Endian/%s.TauCoeff.bin)ZAerosolZEmisZCloudz!%s/%sCoeff/Big_Endian/%sCoeff.bin�.)r9����force�copy)r6����strr����getcwd�reprr$����joinr&���r'����deliver_file�append�make_symlinks_in)�fixd�needcrtmr9���rJ����tgt�srcZcrtmdZlinksrD���rD���rE���r���n���s*���� ��� �c�������������������@���sV���e�Zd�ZdZddd�Zdd��Zdd��Zd d ��Zdd��Ze eddd �Z e eddd�ZdS�)r���z�!Represents an E grid WRF-NMM GRIB1 file, and stores two metadata values: CENLA and CENLO which contain the domain center location.Nc�������������� ���C���s����|dk st��tj�|�}tj�|�s.t�|��t|d�|d|d��|d�}|d�}|dk s^t��|dk sjt��|�j����B}||�d<�||�d<�|�� |d��|�� ���d|�ks�t��d|�ks�t��W�5�Q�R�X�|�j|d��dS�) at��!Copies the file to its destination, and sets the CENLA and CENLO metadata values to the domain center. @param location the destination location @param fileinfo a dict containing fromloc, the location; CENLA, the domain center latitude; and CENLO, the domain center longitude. @param logger a logging.Logger for log messagesN�fromlocF�rF���r9����CENLA�CENLOT�r9���)�AssertionErrorr���r$����dirnamer%����makedirsrO����dstore�transaction� set_loc_avail�update�call_callbacks)�self�locationZfileinfor9���r\���r<���r=����trD���rD���rE����deliver����s"���� zEGRIB1Product.deliverc�����������������C���s���t�j�t�j�||�j�|�j�S�)zy!Decides a filename which is of the format "outdir/category/prodname". @param outdir the output directory)r���r$���rN����category�prodname)rc����outdirrD���rD���rE���� make_location����s����zEGRIB1Product.make_locationc�����������������O���sL���|�j�}|�dtj�|���\}}tjj|||jd��t |dd|�d�|�d��S�)z�!Delivers the file. @returns a hwrf.regrib.GRIB1File for the file. @param regrib the hwrf.regrib.Regrib with input information @param args,kwargs ignoredzprod.rZ���NrX���rY���) rd���Zgribtempr���r$����basenamer&���r'���rO���r9���r���)rc����regrib�args�kwargs�locrC����indexrD���rD���rE����make����s����zEGRIB1Product.makec�����������������C���s���|�d�S�)z!The domain center latitude.rX���rD����rc���rD���rD���rE����getnscenter����s����zEGRIB1Product.getnscenterc�����������������C���s���|�d�S�)z!The domain center longitude.rY���rD���rr���rD���rD���rE����getewcenter����s����zEGRIB1Product.getewcenterz7Returns None or the center latitude of this GRIB1 file.z8Returns None or the center longitude of this GRIB1 file.)N)�__name__� __module__�__qualname__�__doc__rf���rj���rq���rs���rt����propertyZnscenterZewcenterrD���rD���rD���rE���r�������s��� ��c�����������������������s����e�Zd�ZdZd#��fdd� Zdd ��Zd d��Zdd ��Zdd��Zdd��Z dd��Z dd��Zdd��Zdd��Z dd��Zdd��Zdd��Zd$d!d"�Z���ZS�)%r���z}!This is an HWRFTask that post-processes output data for a single WRF stream, from several WRF domains at a a given time.�historyT����Nc�������������� ������s@��t�t|��j|j|j|fd| i|���||�_||�_t||���� ���|�_ ||�_||�_i�|�_ i�|�_|dk|�_|snt�|dkrzdnd}|dkr�td��d} |�jD�]�}d}|dk s�t�|dk s�t�|j|g||�j d �D�]b}d} |s�t�d}||�j |<�|�����d |�jt|�t|�f���|��|�}t|�j|�j|d�|�j|<�q�q�| �s<t�dS�)a���!PostOneWRF constructor @param wrf the hwrf.fcsttask.WRFTaskBase or subclass, whose data is being posted @param domains list of domains to post, as hwrf.wrf.WRFDomain objects @param conf the hwrf.config.HWRFConfig that provides configuration ifnormation @param section the config section in conf @param time the forecast time being posted @param stream the WRF stream to post @param needcrtm do we need CRTM fix files? @param grib GRIB version: 1 or 2 @param faketime unused @param taskname the task name in the database @param kwargs additional keyword arguments passed to hwrf.hwrftask.HWRFTask.__init__�taskname�����egrb2�egrbz.GRIB2 support not yet implemented in hwrf.postFTN)�domains�stream�timez%s added %s => %s)rg���rh���)�superr����__init__r^����conf�_PostOneWRF__needcrtm�_PostOneWRF__wrf�to_datetime_rel�wrf�simstart�_PostOneWRF__time�_PostOneWRF__domainsZ_PostOneWRF__stream�_PostOneWRF__wrfproducts�_PostOneWRF__myproducts�_PostOneWRF__grib2r[����NotImplementedError�products�log�debugr|���rM����product_namer���)rc���r����r����r�����sectionr����r����rS����gribZfaketimer|���rn���ZprodextZadded�domain�first�productrh����� __class__rD���rE���r��������sN������ � � �� �zPostOneWRF.__init__c�����������������C���sJ���|�j�r dnd}|dkr&d|�j|f�}nd|�j|j|f�}t�dd|�}|S�)z�!Returns a human readable string representation of the product name for the given domain. This is used for filenames and product ids. @param domain the domain of interesr~���r���Nz%s.%sz%s-%s.%sz[^a-zA-Z0-9_.-]�_)r����r|����namer-����sub)rc���r�����ext�resultrD���rD���rE���r������s����zPostOneWRF.product_namec�����������������C���s ���|�j����S�)z"!Returns the WRFSimulation object.)r����r����rr���rD���rD���rE���r������s����zPostOneWRF.wrfc�����������������C���s���|�j�S�)z=!Returns the Task that represents the running WRF simulation.)r����rr���rD���rD���rE����wrftask��s����zPostOneWRF.wrftaskc�����������������o���sr���d|kr.|d�D�]}||�j�kr|�j�|�V��qn@d|krV|d�|�j�krV|�j�|d��V��n|�j�D�]}|�j�|�V��q\dS�)z�!Iterates over products. If the domain keyword is in kwargs, then only the specified domain is iterated. If the domains keyword is in kwargs, then that list of domains is iterated. @param args,kwargs argument listr����r����N)r����)rc���rm���rn���r����rD���rD���rE���r������s���� zPostOneWRF.productsc�����������������C���sx���d}|�����}|��d|f�|�}||kr0|��d�}|������d|f���tj�d�r`tjj d|d��tjj |dd|d ��d S�)a���!Makes the post control file for the specified stream as the file fort.14 in the local directory Figure out what control file to use from the conf section for this task. We look for a stream-specific one first (ie.: "auxhist2_control"). If that is missing we use a default control file from the "control" variable. The control file is then copied to fort.14. @bug this function should use deliver_file with loggingz**MISSING**z %s_control�controlzUse control file %sz ./fort.14zfort.14rZ���TrW���N)r�����confget�confstrr����r���r$���r%���r&���r'����remove_filerO���)rc���r����ZMissingr9���r����rD���rD���rE����make_control*��s���� � zPostOneWRF.make_controlc�����������������C���s6���|���d�}|�j}|��dd�}t|||����|�d��dS�)z)!Calls link_post_fix() to link fix files.�FIXhwrfZlink_wrf_fixT)r9���rJ���N)�getdirr�����confboolr���r����)rc���rR���rS���Z link_filesrD���rD���rE����link_fix=��s���� zPostOneWRF.link_fixc�����������������C���s���|�j�S�)z3!Returns the forecast time that is being processed.)r����rr���rD���rD���rE����requested_timeD��s����zPostOneWRF.requested_timec�����������������c���s���|�j�D�] }|V��qdS�)z2!Iterates over all domains that will be processed.N)r�����rc���r����rD���rD���rE���r����G��s���� �zPostOneWRF.domainsc�����������������c���s(���|�j�D�]}||�j|�|�j|�fV��qdS�)a��!Iterates over wrf domains and products Iterates over tuples (domain,wrfproduct,myproduct) where domain is the WRF domain, wrfproduct is the WRF output Product from that domain for the chosen time and myproduct is the output EGRIB1Product for that time.N)r����r����r����r����rD���rD���rE����domprodJ��s���� zPostOneWRF.domprodc�����������������C���s4���t���d�D�]}t�|��q tj�d�r0t�d��dS�)zJ!Deletes any post output files from the current working directory.z ./WRFPRS*r���N)r5���r����unlinkr$���r%���)rc���rC���rD���rD���rE����del_post_outputS��s����zPostOneWRF.del_post_outputc�����������������C���s���t�|||�����S�)z�!Calls check_post() to see if the post succeeded @param retval the post return value @param what String description of what the post was run on (for log messages) @returns True if the post succeeded, False otherwise)r���r����)rc���r7���r8���rD���rD���rE���r���Z��s����zPostOneWRF.check_postc�������������� ���C���sd���|�����D�]V\}}}|jrq|js|����|js|�����d|�j|jt|j�t|j�f����dS�qdS�)za!Returns True if the needed inputs are available for the post, and False if they are not.z2%s: cannot run: %s not available (loc=%s avail=%s)FT) r����� available�checkr����r����r|����didrM���rd���)rc���r����ZwrfprodZmyprodrD���rD���rE����can_run`��s ���������zPostOneWRF.can_runFc�����������������C���s��|�����}|����}d}|tks$|tkr(dS�td|��d�f�dd|����d����}|�����dt������t � dt����rvt�|�����t ��}t ��}t��} t|�j�} |�j}t|�t|��| k��r�d }|����D��]�\} }}| j}| |ks�|����r|�| ��|d | jf�7�}n`|����sbtt����}||d�k}t�rZ|�d��ttd �|d��|�d��tt����}|����| |k�rnq�| | k�r�| | �dk�r�|�| ��q�|����r�z| | k�r�| | ��d7��<�nd| | <�|�����t|jdd|d��|��|d���tj �!|j�}t �"d|�}|�r,t|�#���dk�r,|�#��d�}n |�j$�%d�}|�&d|||f���|dd��d�|dd���d�|dd���d�|dd ���d!�|d"d#���d!�|d$d%���}t't(|��)d&��dd'�d(k}|�d)t*|�f���t+d*d+��}|�,d,|f���W�5�Q�R�X�t|�}|��-||�\}}}}|�rh|�d-|f���|j.|�/|�|||d.�|d��|�| ��|d/| jf�7�}nJ|�r�d0|f�}|�&|��t0|��n&|�d0|f���|d1| j| | �f�7�}W�nP�t1k �r�}�z0|j&d2|t2|�f�dd3��|d4| jf�7�}��W�5�d}~X�Y�nX�q�|j�rH|d5t2|j�| jf�7�}|�rpt3d6| j|j4|jf���q�|d7| jf�7�}|r�t3d8| j|j4f���q�|�|��t|�t|��| k��r�|�d9��t�5d��|�d:��|�r�W�5�Q�R���dS�q�t|�dk�r�|�6d;��|��7t��q�|�d<��|��7t��q�W�5�Q�R�X�dS�)=z�!Runs the post for one forecast time and all WRF domains. @param nosleep If True, disable sleep calls. @param raiseall If True, do not catch exceptions.r���Nzpost.%s.rH���z.work)�prefix�dir�suffixr9���zcwd: z\A/tmpzstatus: z[%s: done] ����zCalling sync...�syncrZ���zReturned from sync.����r{���z./INFILET)rI���r9���r����z#(\d\d\d\d.\d\d.\d\d.\d\d.\d\d.\d\d)z%Y-%m-%d_%H:%M:%Sz:%s: %s: cannot get datestamp from this name; will guess %s�����-��������� ���r��������� ����:�����������������post)�allranksr���zPost command: %s�itag�wtzINFILE netcdf %s NMM NEST z%s: deliver)rX���rY���rV���z[%s: just posted] z %s: failedz[%s: post failed %d times] z %s: Exception caught in post: %sr���z[%s: exception] z[%s (%s) unavailable] z;%s: %s: not available (should be at %s but available=False)z[%s: unavailable] z(%s: %s: not available, and location=NonezSleep 30...zDone sleeping.zstate=FAILEDzstate=COMPLETED)8�getstater����r���r���r���r����r6���r���rL���r-����matchr[���r�����set�dict�lenr����ri���r����r�����is_available�add�intr����Zshould_syncr*����batchexer����r�����make_symlinkrd���r����r$���rk���r.���r/���r�����strftimer����mpirun�mpi�getexerM���r����writer���rf���rj���r���� ExceptionrK����PostHasNoInputr�����sleep�critical�setstate)rc����nosleepZraiseall�stater9���Zlastsync�tempdir�doneZgaveupZtries�countri����messager����Z wrfproductZ myproductr8����nowZ shoudsyncZwrf_baser����� datestamp�cmdr�����retrB���r<���r=���rC����msgrA���rD���rD���rE���r*���n��s>����� ������� �� �� �� ��������� ������������� � zPostOneWRF.run)rz���Tr{���NN)FF)ru���rv���rw���rx���r����r����r����r����r����r����r����r����r����r����r����r���r����r*���� __classcell__rD���rD���r����rE���r�������s(������������/ c�����������������������s����e�Zd�ZdZedddgdddf��fdd� Zdd ��Zd d��Zdd ��Zdd��Z dd��Z dd��Zdd��Zdd��Z dd��Zdd��Zdd��Zdd��Zd d!��Zd"d#��Zd$d%��Zd(d&d'�Z���ZS�))r���z>!A wrapper around PostOneWRF that posts many WRF output times.Nrz���Tr{���c �������������� ������sF��t�t���j|j|j|fd|i| ���| ��_|��_g���_t����_ |��_ �����|d���| d��\}}}t ��}| D�]f}zH|D�]4}�����|��|�\}}}||k�r�|}||krx|}qx|�|��W�qn�tjjk r����Y�qnX�qn|} |dkr�|n|}|dkr�|n|}|}t||�}t||�}t|�}|��_��fdd�|D����_���| ||��dS�)a0��!PostManyWRF constructor @param wrf the hwrf.fcsttask.WRFTaskBase or subclass, whose data is being posted @param domains list of domains to post, as hwrf.wrf.WRFDomain objects @param conf the hwrf.config.HWRFConfig that provides configuration ifnormation @param section the config section in conf @param step time step between post input times @param postclass should be PostOneWRF @param start the first time to post @param end the last time to post @param streams the streams to consider posting @param needcrtm do we need CRTM fix files? @param grib GRIB version: 1 or 2 @param taskname the task name in the database @param kwargs additional keyword arguments passed to PostOneWRF.__init__ r|���r���Nc��������������������s���g�|�]}������|��qS�rD���)r����)�.0r����rr���rD���rE���� <listcomp>=��s�����z(PostManyWRF.__init__.<locals>.<listcomp>)r����r���r����r^���r����� _needcrtm�_PostManyWRF__wrf� _subtasksr����Z_PostManyWRF__done� _postclassr�����get_output_range�listrP����hwrf� exceptions�OutputStreamDisabledr�����to_timedelta�_step�_PostManyWRF__mydomains� _add_subtasks)rc���r����r����r����r�����stepZ postclass�start�end�streamsrS���r����r|���rn����istartZiendZ iintervalZkeepstreamsr����r�����jstartZjendZ jinterval�intervalr����rr���rE���r�������sN�������� � �� zPostManyWRF.__init__c�����������������C���s���|�j�}|}t|�d�}t��}|����}t||�} || k��r�t��} |D�]�}|�d|���d}|����D�]�} | |f}| j|||����d�| |<�| |�dkr�q`|�dt| �t|�t|�t| |��f���| |�� ��}t t||�dd��}||k�r`|�d t|�t|�t|�t|�f���|}d}��qq`|rB��q&qB|�r�|�d |��|�t|�t|�f���|�j� ||�j|����dd��|����D��|�j|�j|||�j|��|�d �f��| }n|�d|�d�f���||7�}q.|�����dtt|�j�����dS�)ao��!Generate subtasks Fills the self._subtasks array. It figures out which times have data for all domains and creates a PostOneWRF object for each of those times. The subtasks are created in temporal order. @param streams the list of streams to consider @param start,end start and end of the range of times to considerr����z check stream %sFrZ���Nz' domain %s stream %s time %s result %sT��negokz dt=%s-%s=%s > epsilon=%sz%Adding post %s with stream %s time %sc�����������������S���s���g�|�]}|�qS�rD���rD���)r�����xrD���rD���rE���r����k��s����z-PostManyWRF._add_subtasks.<locals>.<listcomp>)r����rS���r|���z>%s: ignoring duplicate output time due to WRF output frequencyz%Y-%m-%d %H:%M:%Szlen(self._subtasks)=%s)r�����to_fractionr����r����r����r����r����� get_outputrM���� validtime�abs�taskname_forr����rP���r����r�����_conf�_sectionr����r����r����)rc���r����r����r����r�����when�epsilon�lastr9���Zende�thisr����rB���r�����keyZattime�dtZokstreamrD���rD���rE���r����A��sv���� ��������������� zPostManyWRF._add_subtasksc�����������������c���s���|�j�D�] }|V��qdS�)z%!Iterates over all WRFDomain objects.N)r�����rc���r��rD���rD���rE���r����w��s���� zPostManyWRF.domainsc�����������������c���s8���t�t|�j��D�]$}||�j|�d�|�j|�d�fV��qdS�)a4��!Iterate over all subtasks Iterator that loops over all subtasks yielding a tuple: @code (itask,rtime,subtask) @endcode Where: * itask = task index from 0 to ntasks-1 * rtime = output time this task processes * subtask = the Task objectr���r{���N)�ranger����r����r��rD���rD���rE����subtasks{��s���� zPostManyWRF.subtasksc�����������������C���s"���|������|����D�]}|����qdS�)z1!Calls uncomplete, and then deletes all products.N)� uncompleter����� undeliver)rc���r����rD���rD���rE����unrun���s�����zPostManyWRF.unrunc�����������������C���s$���t�|�_|����D�]\}}}t�|_qdS�)z�!Marks this task and all subtasks as incomplete so that all post-processing will be rerun. Does not undeliver any delivered products.N)r���r����r��)rc����itask�rtime�subtaskrD���rD���rE���r�����s����zPostManyWRF.uncompletec�����������������C���s&���t�||��������\}}d|�j||f�S�)zo!Returns a human-readable taskname for the given subtask time. @param time the time of interestz%s-f%02dh%02dm)� wrf_hr_minr����r����r|���)rc���r�����ihours�iminutesrD���rD���rE���r�����s����zPostManyWRF.taskname_forc�����������������C���s ���t�|�j�S�)z !Returns the number of subtasks.)r����r����rr���rD���rD���rE���� subtask_count���s����zPostManyWRF.subtask_countc�����������������C���s���|�j�d�d�S�)z'!Returns the first time to be processedr���r{���)r����rr���rD���rD���rE���� starttime���s����zPostManyWRF.starttimec�����������������C���s���|�j�t|�j��d��d�S�)z&!Returns the last time to be processedr{���)r����r����rr���rD���rD���rE����endtime���s����zPostManyWRF.endtimec�����������������C���s ���|�j����S�)z$!Returns the WRF object being posted)r����r����rr���rD���rD���rE���r�������s����zPostManyWRF.wrfc�����������������C���s���|�j�S�)z!Returns the Task that ran WRF)r����rr���rD���rD���rE���r�������s����zPostManyWRF.wrftaskc�����������������C���s@���d�}|�����D�].\}}}|jr6|d�kr(|}||kr:|}q�q<q|S�)N)r��� completed)rc���r��r��r��r��rD���rD���rE���� last_complete���s������zPostManyWRF.last_completec�����������������C���s���|�����}|����tkrdS�d}d}|����}d}tj�|��d�|�j�}t j �|��|�sN||�|k��rNd}d}d}d} |����D��]�\} }}|���} | tkr�|d7�}qznN| t kr�|d7�}qzn:|���s�|d7�}|�d|jf���|| krz|r�d}��qqztj�|d |�d �| f��}t jj|dd�}zX|�J�|�d|jf���|jdd ��|����rh|�rhd}W�5�Q�R���W���qW�5�Q�R�X�W�qz�t jjk �r��}�z|�d|jf���W�5�d}~X�Y�qz�tk �r�}�zN|��dd��s�|jd|jt|�f�dd����n|jd|jt|�f�dd��W�5�d}~X�Y�qzX�qz|sN||�|k�rN|�d��t�d��|�d��qN||k�rv|��t��|��d||f���n||k�r�|��t ��|� d��dS�)z�!Internal implementation function: this implements run and runone. Do not call directly. @param one True = runone(), False=run()Nr���F�lockdirr����r{���zsubstask %s cannot run yet.Tz %s.task%dz %Y%m%d.%H%M%S)rC���� max_trieszrun subtask %s)r����z!subtask %s: lock held, moving on.� ignore_errorsz,aborting: %s raised unexpected exception: %sr���z"%s raised unexpected exception: %szSleep 20...����zdone sleeping.z All %d of %d subtasks completed.zMULTI-TASK WORKSTREAM FAILED.)!r����r����r���r��r���r$���rN���r����r|���r&���r'���r]���r��r���r����r����r�����locking�LockFiler6���r*����is_completed�LockHeldr����r����r)���rK���r���r����r����r�����postmsgr����)rc���Zoner9���r���failedr����Zbreak_outerr!��Zn_unrunableZ max_unrunabler��r��r��r�����lockfileZlockerZlhrA���rD���rD���rE����_run_helper���s������� ��� ����� zPostManyWRF._run_helperc�����������������C���s���|���d��dS�)z!Post all inputs.FN�r,��rr���rD���rD���rE���r*������s����zPostManyWRF.runc�����������������C���s���|���d��dS�)z !Post one input time and return.TNr-��rr���rD���rD���rE����runpart���s����zPostManyWRF.runpartc����������� ������k���s����|dkr6|�����D�]"\}}}|jf�|�D�] }|V��q&qntt||��������}t|�jd��}|�����D�]J\}}}tt||�dd��} ||kr^| |k�r^|jf�|�D�] }|V��q��q�q^dS�)aT��!Iterate over products. @param time Only iterate over this time's products. The time that is actually iterated is the last time that is not before the requested time. @param kwargs passed to the subtask to further limit the products iterated over. Typically, that is PostOneWRF.products()Nr����Tr���)r��r����r����r����r����r��r����r��) rc���r����rn���r��Zxtimer��r����Zreltimer��r��rD���rD���rE���r������s���� zPostManyWRF.products)N)ru���rv���rw���rx���r���r����r����r����r��r��r��r��r��r��r��r����r����r ��r,��r*���r.��r����r����rD���rD���r����rE���r������s2��������E6 H)NF)&rx����__all__r5���r����os.path�statr�����mathr-����produtil.cdr&����produtil.fileop�produtil.run�produtil.datastore�produtil.locking� hwrf.hwrftaskr����� hwrf.numerics�hwrf.regrib�hwrf.exceptionsr���r���r ���r ���r���r���r ���r���r���r���r���r����fcst_hr_minr��rl���r���r���r���rD���rD���rD���rE����<module>���s.����8 P ,:��&