U  g|@sdZdgZddlZddlZddlZddlZddlZddl Zddl Zddl Zddl Z ddlm Z ddlmZmZmZmZddl mZmZmZmZddl mZmZmZmZddlmZdd lmZGd d d eZGd d d eZGdddeZ GdddeZ!dS)aThis module implements FcstTask classes for the hwrf multistorm basin scale multistorm final merge task. FinalMergeTask - Create a multistorm final merge domain 01 product, from the required domain 01 information from storm in a multistorm hwrf run. FinalMergeTaskN)NamedDir)realcwd make_symlink deliver_filemake_symlinks_in) FileProduct COMPLETEDRUNNINGFAILED)aliasbigexeexecheckrun)FcstTask)WRFInputMissingc@s,eZdZdZddZd ddZd dd ZdS) Input2FinalMergezThis is the abstract base class of anything that gets, or creates, input files for the Final Merge without running another Task. For example, something that copies the wrfinput output would be a subclass of Input2FinalMergecCs ||_dSN)src)selfrr@/lfs/h1/ops/prod/packages/hmon.v3.2.7/ush/hwrf/finalmergetask.py__init__5zInput2FinalMerge.__init__FcKsdS)zpIn subclasses, this function copies or links the input files. This default implementation does nothing.Tr)r just_checkkwargsrrr get_inputs6szInput2FinalMerge.get_inputsNc Cs|dk s|rtt|tjjs"t|j|j}}|r|r|dkrLtj |}|r|t |r\dS|dk rxd|f}| |dSt |||dddSdt |jt|t|f}|dk r| ||rdS||dS)zLinks the file from the given product to the target location (basename(product.location) if no target is provided). If the product is not yet available or has no location, then the given exception class excclass is raised.NTz!%s: file is empty or non-existentF)loggerforcez*%s: unavailable (available=%s location=%s))AssertionError isinstanceprodutil datastoreProductlocation availableospathbasename isnonemptywarningrstrdidrepr) rproductexcclassrtargetrLAmsgrrr link_product:s2    zInput2FinalMerge.link_product)F)NF)__name__ __module__ __qualname____doc__rrr4rrrrr0s  rcs6eZdZdZfddZeddZd ddZZS) WRFInput2FinalMergezLinks real_nmm gfsinit/realinit/wrfinput_d01 file or returns the FileProduct, regardless of the time and domain, in the current directory for a final merge.cstt||||_dSr)superr9rstormidrr;r __class__rrr\szWRFInput2FinalMerge.__init__cCsdS)N wrfinput_d01rrrrrtgtname_szWRFInput2FinalMerge.tgtnameFcKs>|jdk r:|j|j}|j}|r6|j|t|d|dSdSdS)Nr?r0rF)r get_wrfinputr;rAr4r)rrrrptgtrrrrcs  zWRFInput2FinalMerge.get_inputs)F r5r6r7r8rpropertyrAr __classcell__rrr=rr9Xs   r9cs6eZdZdZfddZeddZd dd ZZS) Merge2FinalMergezLinks gdas_merge/wrfinput_d01 file or returns the FileProduct, regardless of the time and domain, in the current directory for a final merge.cstt||||_dSr)r:rIrr;r<r=rrrqszMerge2FinalMerge.__init__cCsd|jdS)N wrfinput_d01_ gdas_merge.nc)r;r@rrrrAtszMerge2FinalMerge.tgtnameFNcKsp|dk r2||jkr2|dk r.|d||jfdS|jdk rl|j}|j|j}|rh|j|t|||dSdSdS)Nz7Wrong stormid requested: %s instead of %s, cannot link.FrB)r;errorrrA get_merger4r)rrrr;rrErDrrrrws"  zMerge2FinalMerge.get_inputs)FNrFrrr=rrIms   rIcseZdZdZd%fdd ZddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ d&ddZddZddZddZd'd d!Zd(d#d$ZZS))raUThis is a FcstTask since it prepares input to a forecast model. The FcstTask class provides some methods required by the final merge task. This task merges all the domain 01 files from the relocation.Merge(RelocationTask) of each storm and produces a single d01 merged file that is comprise of all the storms in a multstorm.Nc stt|j|||fd|i||jrV|dd|_|dd|_|dd|_n t j d| d|_ d|_|*}t||j|jtj|j|jd |_W5QRXdS) Ntasknameconfig fakestormidmultistorm_sidsmultistorm_priority_sidzSMultistorm run: ERROR Can only run finalmerge on the fakestorm of a multistorm run.WORKhwrfr?)r$)r:rr isfakestormgetstr fake_stormidsplit real_stormidspriority_stormidhwrf exceptionsInvalidConfigOptNamegetdirdest_dir fmprodname transactionrrNr&r'joinoutdir_prod_wrfinput_d01_finalmerge)rdstoreconfsectionrNrtr=rrrs"  zFinalMergeTask.__init__cCs t|_dSr)r stater@rrr update_stateszFinalMergeTask.update_statecCs|jSrrc)ratimedomainrrrwrfinput_at_timeszFinalMergeTask.wrfinput_at_timec CsX|}|dttj|j|j}d|_ tj |rr| d|ftj |d|rhtt|t||j |dd}z`t|_|d|jtf||\}}||||||d|j_WnHt k r.}z(t!|_|j"d|jt#|fddW5d }~XYnXW5QRX||d |jfd S) aPerforms all work needed to run the Final Merge program and create the final merged product. Creates the work directory, CD's to it, Creates a domain mapping dictionary for all the storms. ,link_all_inputs, run_exe, and deliver_products.z'Final Merge Task running in directory: Tz!%s: directory exists; will deleterS)keepr keep_on_errorz%s running in directory %sz %s failed: %s)exc_infoNz %s: completed)$logpostmsgr&getcwdr!fileopmakedirsrbworkdirkeeprunr'existsr*samefiler]rshutilrmtreerr rhinforNr copy_inputsconvert_wrfinput2bin final_mergefinalmerge2netcdfdeliver_productsrircr% Exceptionr criticalr+)rrrunhererundirZ gfsrealinit gdas_mergeerrrrunsD      zFinalMergeTask.runcCs~|d|}d}||}|ddd}d|jd}tj|jd}d|jd }t ||d |d |}|} d |j |j j |d |jf} t j|d|d| f} |jr| | k} t jj| |dtt jg} |jD]} tj|jd| d}tj|}d| d} | | d |j |j j |d| f} t j|d|d| f} |jrd| | k} t jj| |dq|| fS)a3Runs the hwrf_diffwrf_3dvar program to convert all the required wrfinput_d01 files from netcdf to binary. returns a tuple (gfsrealinit_bin, gdas_merge_bins) with the the binary filenames that will be used as input for the final merge executable. convert_wrfinput_d01_2bin hwrf_3dvarrSNrJzgfsrealinit.ncr?zgfsrealinit.binTrr%s/logs/%s_%s_%s.logZgfsrealinit2binstorm_relocateflnm3rrKzgdas_merge.binZgdas_merge2bin)rqr|getexer]rYr&r'rarvrr^r>r5r!rrredirectrprintlistingListingrXr(append)rrfprogprogZcommonWORKhwrfZgfsrealinit_nclinknameZgfsrealinit_ncfilegfsrealinit_bin fin_target fout_targetrqcmdZgdas_merge_binsr;Zgdasmerge_nclinknamerrrr~sN     z#FinalMergeTask.convert_wrfinput2bincCs|d|}d}||}t|jddd|dd}|j}d|j|jj|d|jf}t j |d ||f}|j r||k}t j j ||d d S) z]Runs the hwrf_diffwrf_3dvar program to convert the final merge binary file to netcdf.rr.finalmerge.binzfort.56TrrZ finalmerge2nc 3dvar_updaterN)rqr|rrr_r^r>r5rVr!rrrr)rrrrrrrqrrrrrAs" z FinalMergeTask.finalmerge2netcdfc Cs|dd}|}|d}t|j}d}||g}d|i} t|D]\} } | | | d<qJd|jdi} |j||| | dd S) aRuns the hwrf-utilities hwrf_final_merge program with the following inputs and generates the final merge output file, all in this tasks self.workdir . gfsrealinit_bin - the name of the d01 priority storm gfsrealinit binary filename gdasmerge_bins - a list of gdas_merge d01 binary filenames for each real stormrZhwrf_final_merge()8r)echoinputsoutputsN)rqr|rlenrX enumerater_run_ext) rrZgdasmerge_binsrrrZ numstormsZgesfhrevarsinsiZgdas_merge_binousrrrr\s    zFinalMergeTask.final_mergecks |jVdS)z2Iterates over all products generated by this task.Nrj)rrrrrproductsszFinalMergeTask.productscCs|dt||dS)aAdds an input source (via self.add_input) that will provide the wrfinput output file from real_nmm. The given object must have a wrfinput_at_time(atime,domain) function that returns a Product for a given analysis time and WRFDomain object. Returns self.wrfinputN) add_inputr9rrr;rrr add_wrfinputszFinalMergeTask.add_wrfinputcCs|dt||S)zAdds an input source (via self.add_input) that will provide the wrfinput output file from a prior gdas_merge, real_nmm or relocation. This is used for the storm's final output of the per-storm initialization.merge)rrIrrrr add_mergeszFinalMergeTask.add_mergeFc Ks|dkr|}|d|jftjj|j|d|j}|j}t j |}t j |r|d|j |f|jf|||d|n&|d|j |f|dk r|||dS)aDelivers products to intercom via Product.deliver. Any keyword arguments are passed on to Product.deliver. By default, keep=False, which means the local copy of the file may no longer exists. If frominfo is specified, it will be ignored.NzDelivering products for %srz%s: deliver product from ./%s)frominfornrz)%s: ./%s does not exist. Cannot deliver.)rqr*rNr!rtrurbrcr$r&r'r(rxr,deliver) rmissingrrnrrrDlocblocrrrrs  zFinalMergeTask.deliver_productscsD|}|\}fdd}fdd}tj||||dS)z*Copies, or makes, one or more input files.cs|Srr)rDrargs)namesrrnamerrz)FinalMergeTask.copy_inputs..namercs||||f|dSrrrDnamerr)actionrractorrz)FinalMergeTask.copy_inputs..actorN)rq_make_plist_and_namesr!r"wait_for_products)rrplistrrr)rrrr}s   zFinalMergeTask.copy_inputscsdddd}t}|jddj|_d||j<|jdD]N}|j|j}t|tj j sdt |j }| d |j|j|f|||<q@d d |D}tfd d |D}|||fS)aThis is an internal implementation function that should not be called directly. It returns a three-element tuple containing a list of products, and a dict mapping from product to the local filename, and a dict mapping from product to the copy method. This is used to implement copy_inputs, to copy input files to the local directory from remote tasks' Products.cWst|j||dddS)NT)rrn)rr$rrrrcopiersz4FinalMergeTask._make_plist_and_names..copiercWst|j|d|ddS)NTr)rr$rrrrlinkersz4FinalMergeTask._make_plist_and_names..linkerrrr?rz"Product %s location %s going to %scSsg|]}|qSrr).0krrr sz8FinalMergeTask._make_plist_and_names..c3s|]}|fVqdSrr)rnrrr sz7FinalMergeTask._make_plist_and_names..)dictrrget_wrfinput_gfsinit_realinitZ _wrfinput_d01rMr;r r!r"r#rrArqr|r,r$keys)rrrrZ mergeprodrAractionsrrrrs$   z$FinalMergeTask._make_plist_and_namescCs^|}ddl}|d|||tj|j|j}tjd|j}t ||d|ddS)z{Create a link to a phony final merge file for initial dev testing. without having to run the final merge executablerNz-fake_merge_file going to sleep for %s secondsz../finalmerge.tTr) rqtimer|sleepr&r'rarvr_r)rZ sleep_secsrrZ fmlinknameZfmfilerrr_fake_merge_file s zFinalMergeTask._fake_merge_filecCs&t|}|}||} d|j|jj|f} |rd} |D]J} t| trX| d| f7} q:t| trr| d| f7} q:| dt| f7} q:| dt |t | f| d7} t j t j | | >}nt j t j | }|jr|| k}i} |dkr| }|dkr| }|dkr| }ttt|t|t|}|D]}t jjd t||d q>|rvt jj|d |d |rt jj|d d |d |t |t j j||d |r |D]N\}}d t|}tj|rt||d|dn|d||t|fq|rj|D]N\}}d t|}tj|rNt||d|dn|d||t|fq|D]R\}}tj d t|rr| dd t||ft jjd t||d qr|D]R\}}tjd t|r| dd t||ft jjd t||d qdS)aRun an external command linking in fort.X files for input and output. If redirect=True, redirect logs to a separate file. It will use "self.getexe()" on the command to find the external program to execute. If a list is passed in as the echo variable, then /bin/echo will be run with that list and piped to the external command. The inputs dictionary should consist of a fortran file number and the source file, such as: inputs = {11:tcvitals, 12:wrfout_d01} would produce symbolic links: fort.11 -> tcvitals fort.12 -> wrfout_d01 input files can also be copied using incopies: incopies = {11:tcvitals, 12:wrfout_d01} would create files instead of links. The outputs and opt_outputs (optional outputs) should be of the dictionary as the inputs. As in: outputs = {56:new_data_4x, 85:storm_radius} this would mean the "fort.56" file would be renamed to "new_data_4x" and the "fort.85" renamed to "storm_radius". If opt_outputs is given then the fortran file is tested to see if it exists and only if it does is it renamed to the output filename. A log file will be created consisting of the stdout and stderr of the command run. It will be named consisting of the taskname and command. For example, if this is relocation stage 1 and the command is hwrf_pert_ct then the log file is "rel_stage_1_hwrf_pert_ct.log" z%s/logs/%s_%s.logz%g z%d z%s z6Converted %s to %s for stdin input to fortran command. Nzfort.rTr)ronly_log_errorsrF)rnrz)%s: did not make file %s (would mv to %s)z*%s: did not make file %s (would mv to %s).z,%s: deleting input fort file (symlink to %s)z)%s: deleting input fort file (copy of %s))!r+rqrr^r>r5r floatintr|r-r!ropenmpr rr itertoolschainlistitemsrt remove_filefortlinkfortcopyr*rr&r'rxrrLislink)rrrrincopiesr opt_outputscmdnamerrlogfechostrsemptyiofrvffilerrrrs#           zFinalMergeTask.run_extrcCs|dkr|dd}|}|dkr:||}tj|}tjj|||d}|d||ftdd|dsd |f}||t |n|d |fdS) aRuns the executable this task is responsible for running. Determines if the program ran correctly. The exename is the name of the argument in the [exe] section of the HWRFConfig. Options: runner=Runner - pass a produtil.prog.Runner object if desired. This overrides any decision of what to run: the exename will be ignored, and whatever is supplied in runner is simply passed to produtil.run.run. sleeptime - passed to produtil.run.run to determine how often to check the child process. By default, the sleeptime option in this task's config section is used, or if that is absent, 30 seconds.N sleeptime)rrz%s: exit status %dz rsl.out.0000zSUCCESS COMPLETErz0%s: did not see SUCCESS COMPLETE in rsl.out.0000z$%s: SUCCESS COMPLETE in rsl.out.0000) conffloatrqrr!rrr|check_last_linesrL RealNMMError)rexenamerunnerrrrstatr3rrrrun_exes      zFinalMergeTask.run_exe)N)NNFN)NNNNN)rNN)r5r6r7r8rrirmrr~rrrrrrr}rrrrrHrrr=rrs,/@=%  00 y)"r8__all__r&rz hwrf.fcsttaskrZprodutil.fileopr! produtil.cd produtil.runprodutil.datastoreprodutil.listingrrrrrrrr r r r r rrrhwrf.exceptionsrobjectrr9rIrrrrrs    (