U $ùÎg"iã@sždZdgZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZ ddlZ ddlZ ddlZ ddlZddlZddlZddlZddlZddlZddl mZddlmZmZmZmZmZmZmZm Z ddlm!Z!m"Z"m#Z#ddlm$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddlm,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4dd lm5Z5m6Z6m7Z7dd lm8Z8Gd d „d e9ƒZ:Gd d„de:ƒZ;Gdd„de:ƒZGdd„de:ƒZ?Gdd„de:ƒZ@Gdd„de:ƒZAGdd„dejBjCƒZDGdd„deDƒZEGdd„deEƒZFe Gd d!¡ZHGd"d#„d#eEƒZIGd$d%„d%eIƒZJe9ƒZKe9ƒZLGd&d'„d'eIƒZMGd(d)„d)eMƒZNGd*d+„d+eEƒZOGd,d-„d-eJƒZPdS).aÜ!Runs the real_nmm or wrf executables. This module, contrary to its name, implements HWRFTask classes to run the real_nmm and WRF, including tasks to make the the wrfanl and ghost files. The uncoupled forecast is found in the WRFAtmos class. The coupled forecast is not in this module: it is hwrf.mpipomtc instead. All classes you will want to use derive from WRFTaskBase and FcstTask, classes that exist solely to simplify later code. These are the classes you will want to use: * RealNMM - to run the real_nmm program * WRFAnl, WRFAnl4Trak - 1 minute simulation to generate wrfanl files * WRFGhost, WRFGhostForPost - 1 minute simulation to generate wrfghost files * WRFAtmos - full-length uncoupled forecast * AnalysisCycle - short (1-6 hour) forecast that outputs wrfinput and wrfghost files, suitable for input to another AnalysisCycle * hwrf.mpipomtc.WRFCoupledPOM - coupled WRF+POM forecast, in the hwrf.mpipomtc module All of these classes take input from one another, or from other sources. The abstract base class FcstTask keeps track of those through its FcstTask.add_input() function. The WRFTaskBase class provides simple functions to call add_input for you. Those inputs are stored in objects of class Input2Fcst and its subclasses: * Geog2WRF (WRFTaskBase.add_geogrid()) - geogrid data from hwrf.wps.Geogrid * Met2WRF (WRFTaskBase.add_metgrid()) - metgrid data from hwrf.wps.Metgrid * WRFInput2WRF (WRFTaskBase.add_wrfinput()) - wrfinput data from RealNMM, AnalysisCycle, hwrf.relocate.Merge or hwrf.ensda.FromPriorCycle * Fort652WRF (WRFTaskBase.add_fort65()) - coupler fort.65 file from RealNMM * WRFAnl2WRF (WRFTaskBase.add_wrfanl()) - wrf analysis (time 0 restart) files from WRFAnl, WRFAnl4Trak, WRFGhost, WRFGhostForPost, AnalysisCycle, hwrf.relocate.Merge or hwrf.relocate.RelocationTask (and its subclasses) * WRFBdy2WRF (WRFTaskBase.add_wrfbdy()) - wrf boundary (wrfbdy) files from RealNMM * Prep2WRF (WRFTaskBase.add_prep_hybrid()) - prep_hybrid files from hwrf.prep.PrepHybrid ÚFcstTaskéN)ÚNamedDir)ÚrealcwdÚ deliver_fileÚ make_symlinkÚcheck_last_linesÚ isnonemptyÚ remove_fileÚ netcdfverÚfind_exe)Ú FileProductÚ UpstreamFileÚ UNSTARTED)ÚcheckrunÚmpiÚmpirunÚopenmpÚbigexeÚExitStatusExceptionÚrunÚbatchexe) ÚWRFGeogridMissingÚWRFMetgridMissingÚWRFPrepMissingÚWRFInputMissingÚ WRFBdyMissingÚ WRFAnlMissingÚForecastInputErrorÚ SetNestFailedÚ RealNMMError)Ú to_fractionÚwithin_dt_epsilonÚ to_timedelta)ÚExternalWRFTaskc@s,eZdZdZdd„Zd dd„Zd dd „ZdS) Ú Input2Fcsta!abstract base class of wrf/real_nmm input providers This is the abstract base class of anything that gets, or creates, input files for a WRF simulation without running another Task. For example, something that copies the geogrid output would be a subclass of Input2FcstcCs ||_dS)zÍ!creates a new Input2Fcst with the specified src. The src is implementation-dependent and can be interpreted as desired by subclasses. If no "src" is needed, None is acceptable.N)Úsrc)Úselfr%©r'ú;/lfs/h1/ops/prod/packages/hwrf.v13.2.9/ush/hwrf/fcsttask.pyÚ__init__BszInput2Fcst.__init__FcKsdS)a³!copies or links input files. This function is unimplemented: subclasses are expected to replace it. If just_check=True, checks to see if inputs are available, returning True if they are and False otherwise. If just_check=False, then the files are actually copied or linked. This default implementation does nothing and returns True. @return True on success, False otherwiseTr')r&Ú just_checkÚkwargsr'r'r(Ú get_inputsIs zInput2Fcst.get_inputsNc Csú|dkrt d¡}t|tjjƒs$t‚|j|j}}|sL|  ¡|j|j}}|r¸|r¸|st|  dd¡}|stt j   |¡}|r¤t|ƒr„dS|dk r d|f}| |¡dSt|||dddSd t|jƒt|ƒt|ƒf}|dk ræ| |¡|rîdS||ƒ‚dS) a5!helper function that links data If just_check=True, checks to see if data is available, returning True if it is, and False otherwise. If just_check is False, then the file is linked from the given product to the target location (basename(product.location) if no target is provided). If the product is not yet available or has no location, then the given exception class excclass is raised. @return True on success, False otherwise @param product the produtil.datastore.Product to link @param excclass the class of Exception to raise on error @param logger the logging.Logger to use for logging @param just_check if True, just check for data, but link nothing @param target the name of the link Nz hwrf.fcsttaskÚbasenameÚTz!%s: file is empty or non-existentF©ÚloggerÚforcez*%s: unavailable (available=%s location=%s))ÚloggingÚ getLoggerÚ isinstanceÚprodutilÚ datastoreÚProductÚAssertionErrorÚlocationÚ availableÚcheckÚmetaÚosÚpathr-rÚwarningrÚstrÚdidÚrepr) r&ÚproductZexcclassr0Útargetr*ÚLÚAÚmsgr'r'r(Ú link_productUs>     ÿ zInput2Fcst.link_product)F)NF)Ú__name__Ú __module__Ú __qualname__Ú__doc__r)r,rHr'r'r'r(r$;s  ÿr$c@seZdZdZddd„ZdS)ÚGeog2WRFan!Links Geogrid data to the current directory for a wrf or real_nmm run. This class links geogrid data to the current directory for a wrf or real_nmm run, taken from some source (src) object sent to __init__. That src must have a geodat(domain) function that can provide a Product with geogrid output data (such as hwrf.wps.Geogrid.geodat())FcKsJ|jdk rF|dk rF|j |¡}|rB|js0| ¡|j|t||dSdSdS)zù!Links geogrid data if any is available for the specified domain. If just_check=True, checks for geogrid input data. Otherwise, links the data using Input2Fcst.link_product. Raises WRFGeogridMissing if the data is missing.N©r*F)r%Úgeodatr:r;rHr)r&r0Údomainr*r+Úpr'r'r(r,s  ÿzGeog2WRF.get_inputsN)F©rIrJrKrLr,r'r'r'r(rM†srMc@seZdZdZddd„ZdS)ÚMet2WRFaÆ!Links Metgrid data to the current directory for a wrf or real_nmm run. This class links metgrid.exe output data to the current directory for a wrf or real_nmm execution. The output data is taken from some source object, sent to the src argument of __init__. That object must have a met_at_time(ftime) function that can provide a Product for metgrid data at a specific forecast time (such as hwrf.wps.Metgrid.met_at_time())FcKsJ|jdk rF|dk rF|j |¡}|rB|js0| ¡|j|t||dSdSdS)z÷!Links metgrid data if any is available for the specified time. If just_check=True, checks for metgrid input data. Otherwise, links the data using Input2Fcst.link_product. Raises WRFMetgridMissing if the data is missing.NrNF)r%Ú met_at_timer:r;rHr)r&r0Úftimer*r+rQr'r'r(r,¨s  ÿzMet2WRF.get_inputsN)FrRr'r'r'r(rSžs rSc@seZdZdZddd„ZdS)Ú WRFInput2WRFa½!Links real_nmm wrfinput_d01 files the current directory for a wrf run. This class links real_nmm wrfinput_d01 files the current directory for a wrf run. It gets those files from a source object specified in the src argument to __init__. The src must have a wrfinput_at_time(ftime) function that can provide a Product for metgrid data at a specific forecast time (such as hwrf.fcsttask.RealNMM.wrfinput_at_time()).FcKsV|jdk rR|dk rR|dk rR|j ||¡}|rN|js:| ¡|j|t|d|dSdSdS)zÏ!Links wrfinput_d01 files. If just_check=True, checks for wrfinput_d01 data. Otherwise, links the data using Input2Fcst.link_product. Raises WRFInputMissing if the data is missing.NÚ wrfinput_d01©rDr*F)r%Úwrfinput_at_timer:r;rHr©r&r0ÚatimerPr*r+rQr'r'r(r,Ás þzWRFInput2WRF.get_inputsN)FrRr'r'r'r(rV·s rVc@seZdZdZddd„ZdS)Ú Fort652WRFa¦!Links real_nmm fort.65 files the current directory for a wrf run. This class links real_nmm fort.65 files the current directory for a wrf run. The files are taken from some source object, specified in the src argument to __init__. The src must have a fort65_at_time(atime,domain) function that provides a Product for the fort.65 file. See hwrf.fcsttask.RealNMM.fort65_at_time() for an example.FcKsV|jdk rR|dk rR|dk rR|j ||¡}|rN|js:| ¡|j|t|d|dSdSdS)ak!Links coupler input data for the specified domain and time. If just_check=True, checks for a coupler "fort.65" input product. Otherwise, links the data using Input2Fcst.link_product to the filename "fort.65" in the current working directory. Raises WRFInputMissing if the data is missing. @param atime the analysis time @param domain the wrf domain (a number, string name or WRFDomain) @param logger a logging.Logger for logging messages @param just_check if True, just check for data, otherwise link it @param kwargs ignoredNúfort.65rXF)r%Úfort65_at_timer:r;rHrrZr'r'r(r,Ús  þzFort652WRF.get_inputsN)FrRr'r'r'r(r\Ñsr\cs*eZdZdZ‡fdd„Zddd„Z‡ZS) Ú WRFAnl2WRFa[!Links wrfanl or ghost files the current directory for a wrf run. This class links wrfanl or ghost files the current directory for a wrf run. The files come from some source object, sent to the src argument of __init__. That object must have a wrfanl_at_time(atime,domain) function like hwrf.fcsttask.WRFAnl.wrfanl_at_time()cstt|ƒ |¡||_dS)z9!creates a WRFAnl2WRF for the specified source and domainN)Úsuperr_r)rP)r&r%rP©Ú __class__r'r(r)ùszWRFAnl2WRF.__init__ÚwrfanlFc KsŒ|jdk rˆ|dk rˆ|dk rˆ|jd|jjdkrˆ|j ||¡}|r„|jsP| ¡tj t |ƒd|  ¡||j ¡}|j |t |||dSdSdS)a !Links wrfanl files for the specified domain and time. If just_check=True, checks to see if there should be wrfanl files for the specified domain and time. Otherwise, links the data using Input2Fcst.link_product to the filename "fort.65" in the current working directory. Raises WRFAnlMissing if the data is missing. @param atime the analysis time @param domain the wrf domain (a number, string name or WRFDomain) @param logger a logging.Logger for logging messages @param wrfanl the beginning of the link filename. Usually this is "wrfanl", "wrfghost" or "ghost". @param just_check if True, just check for data, otherwise link it @param kwargs ignoredNÚnamez_d_rXF)r%Ú__dict__rPÚwrfanl_at_timer:r;ÚhwrfÚwrfbaseÚparse_wrf_outnamer@Ú get_grid_idÚnocolonsrHr) r&r0r[rPrcr*r+rQZ localnamer'r'r(r,ýs*ÿ þþzWRFAnl2WRF.get_inputs)rcF)rIrJrKrLr)r,Ú __classcell__r'r'rar(r_ñs  ÿr_c@seZdZdZddd„ZdS)Ú WRFBdy2WRFas!Links real_nmm wrfbdy_d01 files the current directory for a wrf run. This class links real_nmm wrfbdy_d01 files the current directory for a wrf run. Those files are taken from some source object specified in the src argument to __init__. The source must have a wrfbdy_at_time(atime,domain) function like hwrf.fcsttask.RealNMM.wrfbdy_at_time() FcKsV|jdk rR|dk rR|dk rR|j ||¡}|rN|js:| ¡|j|t|d|dSdSdS)aP!Links a wrfbdy file for the specified domain and time. If just_check=True, checks for a wrfbdy input product. Otherwise, links the data using Input2Fcst.link_product to the filename "fort.65" in the current working directory. Raises WRFBdyMissing if the data is missing. @param atime the analysis time @param domain the wrf domain (a number, string name or WRFDomain) @param logger a logging.Logger for logging messages @param just_check if True, just check for data, otherwise link it @param kwargs ignorexNÚ wrfbdy_d01rXF)r%Úwrfbdy_at_timer:r;rHrrZr'r'r(r,(s  þzWRFBdy2WRF.get_inputsN)FrRr'r'r'r(rmsrmc@seZdZdZddd„ZdS)ÚPrep2WRFa—!Links prep_hybrid files to the current directory for a real_nmm run. This class links prep_hybrid files to the current directory for a real_nmm run. The files must come from some source object, specified in the src argument to __init__. That object must have a products() function that behaves like hwrf.prep.PrepHybrid.products() when called with a time= and name= argument.Fc Ksì|jdk rÆ|rÆtt|ƒƒD]¨}| dt|ƒ¡||}|dkrDdnd}dd„|jj||dDƒ}|s†|rpd Std || d ¡fƒ‚|d}|jsœ|  ¡|j |t|d |d |ds|rd Sqn|  dt tƒt |ƒf¡|rèd SdS)ah!Links prep_hybrid output files for the specified time index. If just_check=True, checks for the input product's availability. Otherwise, links the data using Input2Fcst.link_product to the filename "fort.65" in the current working directory. Raises WRFPrepMissing if the data is missing. @param times a list of integers, 0 for the initial time, 1 for the first boundary input time, and so on. @param logger a logging.Logger for logging messages @param just_check if True, just check for data, otherwise link it @param kwargs ignoredNzLook for prep at time rÚinitÚbdycSsg|]}|‘qSr'r')Ú.0rQr'r'r(Ú Xsz'Prep2WRF.get_inputs..)ÚtimerdFzNo prep %s data for t=%sz %Y%m%d-%H%M%Sz fort.%03di„rNzIWhen looking for prep data, src is none or times is false:src=%s times=%sT) r%ÚrangeÚlenÚinfor@ÚproductsrÚstrftimer:r;rHr?rB) r&r0Útimesr*r+ÚiÚtÚwhatÚprodr'r'r(r,Gs< ÿ  þ ÿÿzPrep2WRF.get_inputsN)FrRr'r'r'r(rp=s rpcsDeZdZdZd‡fdd„ Zdd„Zdd„Zd d „Zdd d „Z‡Z S)ra$!abstract base class of anything that runs or prepares input for a forecast model Abstract base class of anything that runs a forecast model, or prepares input to a forecast model. This should not be instantiated directly. It exists just to simplify code in subclasses.Nc sJ|dk s t‚|dkr|}tt|ƒj|||f||dœ|—Žt t¡|_dS)z¨!Creates a FcstTask object. Creates a new FcstTask object. All parameters are passed to the superclass constructor, hwrf.hwrftask.HWRFTask.__init__()N)ÚtasknameÚoutdir)r8r`rr)Ú collectionsÚ defaultdictÚlistÚinputs)r&ÚdstoreÚconfÚsectionrr€r+rar'r(r)ss ÿÿzFcstTask.__init__cCs ||jkS)a›!is there any input data of this type to this task? Returns True if there is at least one input source for the specified input type. That is, if someone called add_input(typename,(something)). Returns False otherwise. @param typename the typename value that was sent to add_input @returns True if add_input was called with that typename, or False otherwise.)r…)r&Útypenamer'r'r(Ú has_input~s zFcstTask.has_inputcCs|j| |¡|S)a3!add input of a specified type Adds an input of the given type typename that should be provided by the given object. The object should be a subclass of Input2Fcst. @param typename the type of input @param inobj the Input2Fcst object that will provide the input)r…Úappend)r&r‰Zinobjr'r'r(Ú add_input‰szFcstTask.add_inputcKs|j|fddi|—ŽS)aj!check if input data is available Checks all inputs of the given typenames to make sure link_input would work if called with the same parameters. Returns True if link_input should succeed, and False if it would fail. This is a simple wrapper around link_input with just_check=True. However, subclasses may override this to perform additional checks, such as for a coupled model. @param typenames a list of typenames that were sent to add_input @return True if all inputs are available, and False otherwise @param kwargs passed to link_input()r*T)Ú link_input)r&Ú typenamesr+r'r'r(Ú check_input“s zFcstTask.check_inputFc Ksft|tƒr|f}| ¡}|D]B}| d|t|ƒf¡||jkr|j|}d}d}|D]Ð} | d| |f¡|d7}zd| j|fd|i|—Ždk}|r¼| dt|ƒ|t| ƒf¡Wq*n| d t|ƒ|t| ƒf¡WnBtt fk r} z|jd |t| ƒfdd W5d } ~ XYnX|rXq*qX|sd |f} |rN| | ¡dS|  | ¡t| ƒ‚qdS)aÒ!link or check for inputs Links all inputs of types given in typenames (an iterable) by calling obj.get_inputs on anything sent to self.add_input. If multiple input sources are available for a given input type, then only the first one that has input is used. Do not use the just_check option: it is part of the internal implementation of check_input; if you need to just check the inputs, use check_input instead. If just_check=True, then nothing is linked. Instead, the routine just checks to see if the inputs are available. That is the same as calling check_input. However, subclasses may override check_input to check additional inputs as part of a coupled model. @param typenames a list of typenames to check, as sent to add_inputs() @param kwargs passed to Input2Fcst.get_inputs() @param just_check if True, just check for data, do not link. Do not use this argument - it is part of the internal implementation of this class. If you want to check for inputs, call check_input() instead, as subclasses may override that function to provide additional checks.z(Look for input of type %s with kwargs=%sFrzCheck %s for input of type %sér*Tz(Found input type %s in inputter #%d (%s)z0Could not get input type %s in inputter #%d (%s)z(cannot get %s files due to exception: %s©Úexc_infoNz8%s: could not find input files of this type. Giving up.) r4r@ÚlogrxrBr…r,r?rÚKeyErrorÚerror) r&rŽr*r+r0r‰ÚthelistÚfoundZitryZinputterÚerGr'r'r(r¡sr   ÿ  ÿÿÿÿþÿÿ þÿ ÿþÿ   zFcstTask.link_input)NN)F) rIrJrKrLr)rŠrŒrrrlr'r'rar(rks    cs eZdZdZd?‡fdd„ Zd@dd„Zdd „Zd d „Zd d „Zdd„Z e dd„ƒZ dd„Z dd„Z dd„Zdd„Zdd„Zdd„Zdd„Zd d!„ZdAd#d$„Zd%d&„Zd'd(„Zd)d*„ZdBd,d-„Zd.d/„Zd0d1„Zd2d3„Zd4d5„Zd6d7„ZdCd9d:„Zd;d<„Ze d=d>„ƒZ ‡Z!S)DÚ WRFTaskBasez·!base class of classes that run wrf or real_nmm This is the abstract base class of tasks that run real or WRF. The purpose of this class is simply to reduce code duplication.Tc sVtt|ƒj|||f|Ž| |¡|_| ¡| ¡|_tƒ|_ ||_ |  dd¡|_ dS)a2!constructor Creates a WRFTaskBase for the specified datastore, conf and section. The wrf argument is a WRFSimulation. It will not be used directly: instead the wrf is copied, and the copy is used. If keeprun=True, then the output of the simulation, after running it, will not be scrubbed. Other keyword arguments are passed to the superclass FcstTask. @param dstore the produtil.datastore.Datastore to use @param conf the hwrf.config.HWRFConfig that provides configuration ifnormation @param section the config section in conf @param wrf the hwrf.wrf.WRFSimulation object that is being run @param keeprun if True, the simulation temporary files are not deleted @param kwargs passed unmodified to FcstTask.__init__()Úis_ensda_realnmmÚFalseN) r`r™r)Úmake_wrfÚ_WRFTaskBase__wrfÚ make_productsÚ bdyepsilonÚ dt_epsilonr„Ú_WRFTaskBase__run_exe_callbacksÚkeeprunÚgetrš)r&r†r‡rˆÚwrfr¢r+rar'r(r)ís  zWRFTaskBase.__init__NcCs˜|dkr| ¡}|dk st‚td| dd¡ƒ}| dd¡ ¡ d¡ ¡}d|kr\| d¡}n| ¡}d |¡}d}t|ƒdkr–|D]}|t |ƒ7}q‚ntd| dd¡ƒ}|}|  dd ¡}||dkrt|ƒdkrð|  d |t |ƒt |ƒf¡n|  d ||t |ƒf¡|  |||¡n |  d ¡| d d¡} | dd¡} |j ddd¡ ¡ d¡ ¡} | dd¡ ¡ d¡ ¡} | dd¡ ¡ d¡ ¡} | dd¡ ¡ d¡ ¡}| dkrÞ|dkrÞ| dkrÞt | ƒ}t |ƒ}t | ƒ}nœd| krô|  d¡}n|  ¡}d |¡} t |dƒ}d| kr(|  d¡}n|  ¡}d |¡} t |dƒ}d|kr\| d¡}n| ¡}d |¡}t |dƒ}| dkrŠd}n&d| kr¤t|  d¡ƒ}n t|  ¡ƒ}|dkrÎ|dkrÎ| dksì|dkr2|dkr2|dkr2|dkrú|dkrú|dkrú|j d|dd¡ ¡ d¡ ¡}|j d|dd¡ ¡ d¡ ¡}|dkrŠ|dkrŠ| dt |ƒt |ƒf¡d}d}d}np|} |}d| kr¨|  d¡}n|  ¡}d |¡} t |dƒ}d|krÜ| d¡}n| ¡}d |¡}t |dƒ}|  dt | ƒt | ƒt |ƒf¡|j| | ||||dnb| dkrF| dksZ| dkrz| dkrz|  d| | f¡| | | ¡n|  dt | ƒt | ƒf¡dS)aë!sets nproc_x, nproc_y and I/O server settings This is a protected member function meant to be used by the make_wrf implementation in subclasses. This class does not use it. It sets the WRF nest_pes_x and nest_pes_y, and sets I/O server settings if desired based on config options: @param nest_pes_x,nest_pes_y compute grid dimensions per domain @param nproc_x,nproc_y compute grid dimensions @param nio_groups number of I/O server groups @param nio_tasks_per_group number of servers per group @param poll_servers set to True to poll I/O servers. This generally decreases the number of I/O server groups needed for a given WRF run. NrÚ nio_groupsrÚnio_tasks_per_groupÚ0ú,Ú poll_serversTzlUsing %d nio_groups. With a possible nio_tasks_per_group domain scheme up to %s domains with poll_servers=%sz3Using %d groups of %d io tasks with poll_servers=%szNot setting io_server settings.Únproc_xéüÿÿÚnproc_yÚconfigÚmultistorm_sidsz-999zdm_task_split.comm_startzdm_task_split.nest_pes_xzdm_task_split.nest_pes_yéÿÿÿÿz wrf_%sstormÚ nest_pes_xÚ nest_pes_yz®Trying to set nest_pes x and y for %s storms, but no '[wrf_%sstorm]' section in conf file. Will set dm_task_split values to -1, which will cause wrf to decide the task mesh .zASetting dm_task_split comm_start=%s, nest_pes_x=%s, nest_pes_y=%s)Úcomm_start_d01Únest_pes_x_d01Únest_pes_y_d01z1No dm_task_split so Setting nproc_x=%d nproc_y=%dz;No dm_task_split and Not setting nproc_x or nproc_y (%s,%s))r“r8ÚmaxÚconfintÚconfstrÚstripÚsplitÚjoinrwÚintÚconfboolÚdebugrBÚset_io_serversr‡Úgetstrr•Úset_dm_task_splitÚ set_nprocs)r&r¤r0r¥r¦Ú nio_tpg_splitÚ total_nio_tpgÚnumr©rªr¬Ú multi_sidsÚ comm_startr°r±r³r´r²Zcomm_start_splitZnest_pes_x_splitZnest_pes_y_splitÚstormsZ inest_pes_xZ inest_pes_yr'r'r(Ú_set_wrf_proc_configsò      ÿÿÿÿÿ                  ÿÿÿÿÿÿÿü        ÿ þ(ÿz WRFTaskBase._set_wrf_proc_configcCs| ¡S)aŒ!creates a WRFSimulation object for this class Given the wrf object passed to the constructor, creates a new WRFSimulation that is modified for this object. For example, this class may need to add output of a restart and wrfinput file at the end of the simulation, or enable hourly output of history files. @param wrf the wrf argument to __init__)Úcopy©r&r¤r'r'r(rœ s zWRFTaskBase.make_wrfcCs| dt|ƒ¡S)as!adds a geogrid input source Adds an input source (via self.add_input) that will provide the output of WPS geogrid.exe. The given object must have a geodat member function which takes a WRFDomain as its argument and returns a Product to link. Returns self. @param g the src argument to Geog2WRF.__init__() @returns selfÚgeogrid)rŒrM)r&Úgr'r'r(Ú add_geogridªs zWRFTaskBase.add_geogridcCs| dt|ƒ¡S)ag!adds a metgrid input source Adds an input source (via self.add_input) that will provide the output of WPS metgrid.exe. The given object must have a met_at_time function that returns a Product to link for a specified forecast time. Returns self. @param m the src argument to Met2WRF.__init__() @returns selfÚmetgrid)rŒrS)r&Úmr'r'r(Ú add_metgridµs zWRFTaskBase.add_metgridcCs| dt|ƒ¡S)a¯!adds a prep_hybrid input source Adds an input source (via self.add_input) that will provide the output of the prep_hybrid program. The given object must have a products function that iterates over products for a given name='bdy' or name='init' and a time=F for a given forecast time F. Returns self. @param p the src argument to Prep2WRF.__init__() @returns selfÚprep)rŒrp)r&rQr'r'r(Úadd_prep_hybridÀs zWRFTaskBase.add_prep_hybridcCs | d¡S)zH!returns True if prep_hybrid was requested, and False otherwise.rÑ)rŠ©r&r'r'r(Úuse_prep_hybridÌszWRFTaskBase.use_prep_hybridcCs| dt|ƒ¡S)a!adds a wrfinput_d01 input source Adds an input source (via self.add_input) that will provide the wrfinput output file from real_nmm. The given object must have a wrfinput_at_time(atime,domain) function that returns a Product for a given analysis time and WRFDomain object. Returns self. @param r the src argument to WRFInput2WRF.__init__() @returns selfÚwrfinput)rŒrV©r&Úrr'r'r(Ú add_wrfinputÑs zWRFTaskBase.add_wrfinputcCs| dt|ƒ¡S)a¡!adds a wrfbdy_d01 input source Adds an input source (via self.add_input) that will provide the wrfbdy output of a real_nmm run. The given object must have a wrfbdy_at_time(atime,domain) function that returns a Product to link for a specified analysis time and WRFDomain object. Returns self. @param r the src argument to WRFBdy2WRF.__init__() @returns selfÚwrfbdy)rŒrmrÖr'r'r(Ú add_wrfbdyÝs zWRFTaskBase.add_wrfbdycCs| dt|ƒ¡S)a†!adds a coupler fort.65 input source Adds an input source (via self.add_input) that will provide the fort.65 output from real_nmm. given object must have a fort65_at_time(atime,domain) function that returns a Product to link for a specified analysis time and WRFDomain object. Returns self. @param r the src argument to Fort652WRF.__init__ Úfort65)rŒr\rÖr'r'r(Ú add_fort65és zWRFTaskBase.add_fort65cCs| |¡| |¡| |¡S)am!add a fort.65, wrfinput_d01 and wrfbdy_d01 input source This is a convenience function that simply passes its argument to self.add_fort65, add_wrfinput and add_wrfbdy in that order. Returns self. @param r the src argument to Fort652WRF.__init__(), WRFBdy2WRF.__init__() and WRFInput2WRF.__init__() @return self)rÜrØrÚrÖr'r'r(Úadd_realôs  zWRFTaskBase.add_realcCsd|jf}| |t||ƒ¡S)aì!add a wrfanl input source for a specific domain Adds an input source (via self.add_input) that will provide the wrfanl output file from a prior run of wrf.exe. The given object must have a wrfanl_at_time function that returns a Product to link for a specified analysis time and domain. Returns self. @param r the src argument to WRFAnl2WRF.__init__() @param domain the domain for whom this is the wrfanl product @return selfú wrfanl-%s)rdrŒr_)r&r×rPrdr'r'r(Ú add_wrfanls zWRFTaskBase.add_wrfanlcCsdS)as!called from constructor, creates Products This is called from the WRFTaskBase constructor. Subclasses should re-implement this method to generate internal information about what products this class can provide, so that self.products() and self.deliver_products() will function properly. The default implementation does nothing.Nr'rÓr'r'r(ržszWRFTaskBase.make_productsc Cs¸| dd¡}|rdnd}dD]–}| |d¡}| ¡}|sJ| d|f¡q| d||f¡t |¡D]J}t d dtj   |¡¡}|stj   |¡r¢t |||dd qft |||d qfqd S) zÆ!links or copies fixed data files Links or copies fix files needed by WRF. Will copy if the link_wrf_fix=no in this task's config section. Otherwise, the files are linked.Ú link_wrf_fixTÚlinkrÉ)zfix.eta_lookupz fix.trackz fix.wrf_otherr.z7%s: no conf entry for this fix file or set of fix filesz3Will %s WRF fix files from %s to current directory.z^hwrf_r/©r0N)r¼r·r“r?rxÚglobÚreÚsubr=r>r-Úisdirrr)r&Ú link_filesZactZconfitemÚpatternr0r%Útgtr'r'r(Úlink_fixs(   ÿÿÿzWRFTaskBase.link_fixcCs |jddS)z0!Checks to see if all needed input is available.TrN)Úlink_all_inputsrÓr'r'r(Úcheck_all_inputs/szWRFTaskBase.check_all_inputsFcCsnd|jk}d}|r6|o4|jddd„|j ¡Dƒ|d}|jD]}|oR|jd||d}q<|j ¡D](}|ov|jd||d }| ¡r„|r`qŠq`d |jkr¶|o´|jd |j ¡|j ¡|d }d |jkrâ|oà|jd |j ¡|j ¡|d }d |jkr|o|jd |j ¡|j ¡|d }|j ¡}|jD]F}||kr"d|jf}||jkr"|od|j|||j ¡|d }q"|S)z¹!Links all inputs provided by the various add_* member functions. @param just_check if True, just check to see if data is available. If False, actually copy.rÑTcSsg|]}|‘qSr'r')rsr}r'r'r(rt<sz/WRFTaskBase.link_all_inputs..)r{r*rË)rPr*rÎ)rUr*rÕ)rPr[r*rÙrÛrÞ)r…rrÚbdytimesÚneed_all_metgridÚget_moadÚsimstartrd)r&r*Zuse_prepÚokayrPrUÚMOADrdr'r'r(rë2sn þ ÿÿ  þ þ  þ      þ zWRFTaskBase.link_all_inputscCsdS)zK!Returns True if all metgrid files are needed as input to this TaskFr'rÓr'r'r(rîZszWRFTaskBase.need_all_metgridcCs|j |¡dS)a !Adds a callable object exe=f(self,exe) to the list of calls to make in run_exe() before executing the command. @param callback a function f(task,exe,comp,ranks) that takes a Runner "exe" as input and returns a Runner as output. The resulting Runner will be run. The comp and ranks are a list of coupled components, and a dict mapping from component to the number of ranks for that component. The special value 0 for ranks means "use all ranks". N)r¡r‹)r&Úcallbackr'r'r(Úrun_exe_callback^s zWRFTaskBase.run_exe_callbackcCs–| ¡}| d¡t|tjjƒs$t‚|jD]R}| dt|ƒt|ƒf¡|||||ƒ}t|tjjƒsht‚| dt|ƒf¡q*| dt|ƒf¡|S)a·!Calls the callbacks provided to run_exe_callback, in the order they were provided, giving each the return value of the last. @param exe the Runner to execute @param comp a list of coupled component names @param ranks a dict mapping from component name to the number of ranks. The special value 0 for ranks means "use all ranks" @returns The Runner returned by the last callback function.zCalling run_exe callbacks.zCalling callback %s on %s...zResult of callback is %szFinal result is %s) r“rxr4r5ÚprogÚRunnerr8r¡rB)r&ÚexeÚcompÚranksr0Úcr'r'r(Úcall_run_exe_callbacksjs  z"WRFTaskBase.call_run_exe_callbacksr¤c Cs2|dkr| dd¡}| ¡}tj dd¡}t|ƒ}| d|¡|dkr¨| |¡}t|ƒ}|dkrnt ||d}|r€t t|ƒƒ}nt |d d j |d }|  |d gd d i¡}tj  ¡D]\} } td| t| ƒfƒq²t|||d} | d|| f¡tdd|dsd|f} | | ¡t| ƒ‚n| d|f¡dS)a®!runs the executable Runs the executable this task is responsible for running. Determines if the program ran correctly. The exename is the name of the argument in the [exe] section of the HWRFConfig. Options: @param not_allranks By default (False) all ranks are used to run the executable. Pass not_allranks=True to run on only one rank. @param runner pass a produtil.prog.Runner object if desired. This overrides any decision of what to run: the exename and not_allranks will be ignored, and whatever is supplied in runner is simply passed to produtil.run.run. @param sleeptime passed to produtil.run.run to determine how often to check the child process. By default, the sleeptime option in this task's config section is used, or if that is absent, 30 seconds. @param exename if no runner is specified, self.getexe(exename) is called to get the executable path, and it is run as an MPI program. @note The call_run_exe_callbacks() will NOT be called if the runner argument is specified and non-null. NÚ sleeptimeéZ ATM_THREADSÚ1zATM_THREADS %sr)ÚthreadsT)Úallranks)ZOMP_NUM_THREADSr¤rz$%s=%s)r0rüz%s: exit status %dú rsl.out.0000zSUCCESS COMPLETErâz0%s: did not see SUCCESS COMPLETE in rsl.out.0000z$%s: SUCCESS COMPLETE in rsl.out.0000)Ú conffloatr“r=Úenvironr£r»rxÚgetexerrrÚenvrûÚitemsÚprintrBrrr•r) r&ÚexenameÚ not_allranksÚrunnerrür0rÿr÷ÚcmdÚkÚvÚstatrGr'r'r(Úrun_exe|s4   ÿ   zWRFTaskBase.run_execCsdS)zÌ! last step before running executable Called by self.run() just before calling run_exe. The default implementation does nothing. This is intended to be overridden by subclasses.Nr'rÓr'r'r(Ú final_prerun³szWRFTaskBase.final_preruncCsdS)a!called immediately after cding to a scrub directory Called by self.run() after CDing to the new directory, but before doing anything else. The default implementation does nothing. This is intended to be overridden by subclasses.Nr'rÓr'r'r(Úinitial_prerun¹szWRFTaskBase.initial_prerunc Cs.| ¡}|j}tj |¡rL| d|f¡tj | d¡|¡rBt‚t   |¡|j   ¡}t ||j|ddª}z`| d|jtƒf¡||_| ¡| ¡| ¡| ¡| ¡| ¡| ¡| ¡WnBtk r }z"|jd|jt|ƒfdd‚W5d}~XYnXW5QRX| d |jf¡dS) a!run the wrf or real_nmm Performs all work needed to run the program. Creates the work directory, CD's to it, runs the initial_prerun(), link_fix(), link_all_inputs(), make_namelist(), final_prerun(), run_exe(), postrun() and deliver_products().z!%s: directory exists; will deleteÚWORKhwrfT)Úkeepr0Ú keep_on_errorz%s running in directory %sz %s failed: %sr‘Nz %s: completed)r“Úworkdirr=r>Úexistsr?ÚsamefileÚgetdirr8ÚshutilÚrmtreerrðrr¢rxr€rr9rrêrëÚ make_namelistrrÚpostrunÚdeliver_productsÚ ExceptionÚcriticalr@Úpostmsg)r&r0Zrunherer[Zrundirr˜r'r'r(r¿s>    ÿÿ ÿzWRFTaskBase.runcCsdS)zÙ!called just after run_exe() succeeds. Called by self.run() just after run_exe returns successfully. The default implementation does nothing; this is intended to be overridden in subclasses.Nr'rÓr'r'r(ràszWRFTaskBase.postruncCs tdƒ‚dS)a1!delivers products to their destinations Called by self.run() after postrun(), just before CDing out of the work directory. This should deliver products to their destinations. The default implementation raises NotImplementedError. This MUST be overridden in subclasses.z9A WRFTaskBase subclass did not implement deliver_productsN)ÚNotImplementedErrorrÓr'r'r(ræszWRFTaskBase.deliver_productsúnamelist.inputc Csn|dkr| ¡}|j dd¡}|j dd¡}| ¡ | d¡|j|||¡}t|dƒ}| |¡W5QRXdS)a%!generates the wrf namelist.input file Runs set_ij_start (swcorner_dynamic) to generate the i & j start locations for domain 2, then generates the namelist. @param filename the namelist.input filename @param logger the logging.Logger to log messages (optional)Nr­ÚdomlatÚdomlonÚswcorner_dynamicÚwt) r“r‡Úgetfloatr¤r%rÚ storminfoÚopenÚwrite)r&Úfilenamer0r#r$ÚsZnlinr'r'r(rïsÿ zWRFTaskBase.make_namelistcCs|jS)z4!Returns the WRFSimulation object used by this task.©rrÓr'r'r(r¤ýszWRFTaskBase.wrfcCs|jS)zÔ!Returns the WRFSimulation object used by this task. This property has the same effect as self.wrf(), but this is a property instead. Hence, you can type self.sim instead of self.wrf()r-rÓr'r'r(ÚsimszWRFTaskBase.sim)T)N)F)r¤FNN)r"N)"rIrJrKrLr)rÈrœrÍrÐrÒÚpropertyrÔrØrÚrÜrÝrßržrêrìrërîrôrûrrrrrrrr¤r.rlr'r'rar(r™èsD           ( ÿ 7! r™csšeZdZdZdd„Zdd„Z‡fdd„Zdd „Zd d „Zd d „Z d"‡fdd„ Z dd„Z d#dd„Z d$dd„Z d%dd„Zdd„Zdd„Zd&‡fd d!„ Z‡ZS)'ÚRealNMMz’!a HWRFTask subclass that runs real_nmm This subclass of WRFTaskBase runs the real_nmm to generate inputs for the provided WRFSimulation.csTtddd‰ˆsdS| ¡‰ˆ d|t|ƒf¡t|ƒdk rBdS‡‡fdd„}|S) a-!@brief returns a decompression copier for deliver_file @details Returns an object that has the reverse effect of self.compression_copier. This will uncompress files that compression_copier copier would compress. HDF5 files will all be converted to 64-bit indexing NetCDF 3 files. @param src the source file @returns None if the source file is not HDF5. If it is HDF5, then a copy(s,t,x) function is returned, suitable for passing to the copy argument of produtil.fileop.deliver_file()ÚncksF)Ú raise_missingNz%s file type is %sÚHDF5cs0t|ˆdttˆƒdd||fdkˆddS)Nrâz-6z-Oz /dev/null)r rr)r,r}Úx©r0r1r'r(rÉ,s z*RealNMM.decompression_copier..copy)r r“rxr )r&r%rÉr'r5r(Údecompression_copiers  zRealNMM.decompression_copiercCsŒt|jd|jtj |jd¡d|_t|jd|jtj |jd¡d|_t|jd|jtj |jd¡d|_ t|jd|jtj |jd¡d|_ dS)zÙ!prepares products for deliver_products() and products() Generates produtil.datastore.FileProduct objects for the files that should be delivered in deliver_products() and listed by products().rW©r9rnrr]N) r r†r€r=r>rºrÚ prod_wrfinputÚ prod_wrfbdyÚprod_logÚ prod_fort65rÓr'r'r(rž0sÿ ÿ ÿ ÿzRealNMM.make_productscs&tt|ƒ |¡}|jj ddd¡|S)aC!adds a prep_hybrid data source Modifies the simulation namelist generator so that it specifies use_prep_hybrid=T. Calls WRFTaskBase.add_prep_hybrid(p) to add the specified object as a prep_hybrid data source. @param p the src parameter to Prep2WRF.__init__() @return selfÚdomainsrÔT)r`r0rÒr.ÚnlÚnl_set)r&rQÚretrar'r(rÒ>s zRealNMM.add_prep_hybridcCs| ¡}|j dd|j¡|S)aA!creates a WRFSimulation object for this class Creates a copy of the specified WRFSimulation wrf. The copy has a simulation length of one minute or one physics timestep, whichever is larger. Also, it produces wrfanl files for all domains. @param wrf the wrf argument to __init__r<rÔ)rÉr=r>rÔrÊr'r'r(rœKszRealNMM.make_wrfcCsdS)zK!returns True if all metgrid files are needed as input to this TaskTr'rÓr'r'r(rîVszRealNMM.need_all_metgridc Csb| ¡}dD]P}zt||ƒWq tk rZ}z|jd|t|ƒfddW5d}~XYq Xq dS)z)!deletes the wrfinput_d01 and wrfbdy_d01.)rWrnz8%s: cannot remove, but continuing anyway. Error was: %sTr‘N)r“r ÚEnvironmentErrorr?r@)r&r0Úfr˜r'r'r(rZs ÿÿzRealNMM.initial_prerunÚreal_nmmrýc s6| ¡}| dt|jƒ¡| dd¡}| dt|ƒ¡td|ƒ}| dt|ƒ¡| d|f¡t|ƒD]Â}z2tt |ƒj |d|j k|d | d ¡Wq2Wqnt t tfk r.}zd|d|kr|jd |d|t|ƒfd d t |¡| d¡n|jd|t|ƒfd d ‚W5d}~XYqnXqndS)a¨!runs the real_nmm program Tries several times to run the real_nmm. This is a workaround for problems encountered on Zeus when running the experimental 9:3:1 HWRF. The real_nmm would exit for no apparent reason about 50% of the time. Eventually that was tracked down to a memory error caused by NetCDF, which forced us to use intio for some files, NetCDF for others, and PNetCDF for yet more files. @param exename the name of the real_nmm executable in the configuration file "exe" section. @param sleeptime passed to produtil.run.run to determine how often to check the child process.zreal section is Ú max_triesr«zreal max_tries is rz$after max(1,...), real max_tries is zWill try to run real %d timesrÑ)rüzReal succeeded. Hurrah!zEReal failed %d time(s). Will retry after %d second sleep. Error: %sTr‘z-Returned from sleeping. Will now retry real.z-Real failed %d time(s). Aborting. Error: %sN)r“rxrBrˆr¶rµr@rvr`r0rr…r@rrr?ruÚsleepr•)r&rrür0ZmaxtriesZtrynumr˜rar'r(rcsL   ÿÿ  ÿÿý ÿ ÿþzRealNMM.run_execCs|jS)zC!returns the wrfinput file regardless of the time or domain©r8rÓr'r'r(Ú get_wrfinputŽszRealNMM.get_wrfinputNcCsL|r$|jd| ¡ ¡jdkr$dS|dk rHt|| ¡ ¡|jƒrH|jSdS)a/!returns the wrfinput file for a specified time and domain @param atime the time @param domain the WRF domain (an integer, string name or WRFDomain) @return the wrfinput file for the specified time and domain. Returns None if that time and domain are not valid.rdN)rer¤rïr!rðr r8©r&r[rPr'r'r(rY’s ÿzRealNMM.wrfinput_at_timecCsL|r$|jd| ¡ ¡jdkr$dS|dk rHt|| ¡ ¡|jƒrH|jSdS)a+!returns the wrfbdy file for a specified time and domain @param atime the time @param domain the WRF domain (an integer, string name or WRFDomain) @return the wrfbdy file for the specified time and domain. Returns None if that time and domain are not valid.rdN)rer¤rïr!rðr r9rGr'r'r(roŸs ÿÿzRealNMM.wrfbdy_at_timecCsL|r$|jd| ¡ ¡jdkr$dS|dk rHt|| ¡ ¡|jƒrH|jSdS)a5!returns the coupler fort.65 file for a specified time and domain @param atime the time @param domain the WRF domain (an integer, string name or WRFDomain) @return the fort.65 file for the specified time and domain. Returns None if that time and domain are not valid.rdN)rer¤rïr!rðr r;rGr'r'r(r^¬s ÿÿzRealNMM.fort65_at_timecCsˆtj tj |jj¡¡d}|jr,|  d¡}|jj dd|dd}|jrR|  d¡}|j j dd|d|j j ddd|j j ddddS) z:!delivers products that were identified by make_products()NrWF)ÚfrominforÚcopierrnr)rHrr])r5ÚfileopÚmakedirsr=r>Údirnamer8r9ršr6Údeliverr9r:r;)r&rIr'r'r(r¹s ÿ  zRealNMM.deliver_productsccs$|jV|jV|jV|jVdS)z¡!iterates over all products Iterates over all of this Task's products (produtil.datastore.FileProduct objects) created by make_products()N)r8r9r:r;rÓr'r'r(ryÊszRealNMM.productsr"csn|dkr| ¡}| d¡}d}t d¡D]}|}| ¡ |||¡qHq(|dkrXtdƒ‚tt|ƒ ||¡dS)a°!Writes the namelist for real. Writes the namelist.input file for a call to real_nmm. This does the same as its parent class's implementation, except that the num_metgrid_levels is also overridden to match whatever WPS actually created (which may not match the original namelist). @param filename the name of the namelist.input file @param logger the logging.Logger objectNZhwrf_metgrid_levelszmet_nmm.d01*.nczFCould not find a met_nmm.d01*.nc file for running hwrf_metgrid_levels.) r“rrãr¤Úset_metgrid_levels_fromrr`r0r)r&r+r0ÚexepathZmetfiler4rar'r(rÓs  ÿzRealNMM.make_namelist)rBrý)NN)N)N)r"N)rIrJrKrLr6ržrÒrœrîrrrFrYror^rryrrlr'r'rar(r0s   +  r0ééc@sNeZdZdZdd„Zedd„ƒZdd„Zdd „Zd d „Z d d „Z ddd„Z dS)ÚWRFAnlz5!runs a short WRF simulation to generate wrfanl filescCs| ¡}| |¡| ¡}t|j dd¡ƒ|j}tdtt   |t ¡ƒƒ}|dksVt ‚|  ¡|j|d| ¡| ¡ks~t ‚|jd|d|S)aü!creates the WRFSimulation object (self.wrf()) Creates a copy of the specified WRFSimulation wrf. The copy has a simulation length of one minute or one physics timestep, whichever is larger. Calls the _set_wrf_proc_config to set I/O server and compute grid dimensions based on this HWRFTask's config section. Configures the WRF to produce wrfanl files for all domains. @param wrf the wrf argument to __init__ @returns the new WRFSimulationÚphysicsÚnphsgN@r©ÚendÚhistory)Ústep)rÉrÈrïr r=Únl_getÚdtrµÚfloatÚmathÚfloorÚanl_fudge_factorr8Ú analysis_outÚ set_timingrðÚsimendÚ add_output)r&r¤ròÚendtimeZflooredr'r'r(rœs    zWRFAnl.make_wrfcCs | ¡ ¡S)z1!the time for which analysis files are generated.©r¤rðrÓr'r'r(ÚanltimeszWRFAnl.anltimecCshtƒ|_| ¡ ¡}| ¡D]F}||kr*q| ¡ |¡}tj |j|¡}t |j ||j |d|j|<qdS)z2!creates FileProduct objects for all wrfanl files.r7N) ÚdictÚ _productsr¤rïÚ analysis_namer=r>rºrr r†r€)r&ròrPÚpnameÚlocr'r'r(rž#s  ÿzWRFAnl.make_productscCs:|dkr dS||jkrdS|j|}| ¡r0dS|j|S)z{!returns the wrfanl product for the specified domain @param domain the domain for whom the wrfanl file is requestedN©r.Úis_moadrg©r&rPr'r'r(Ú get_wrfanl-s  zWRFAnl.get_wrfanlcCs@||jkrdS|| ¡krdS| ¡|}| ¡r6dS|j|S)a'!get a wrfanl file for a specified domain and analysis time Returns the wrfanl product for the specified domain and analysis time, if one exists, and otherwise, None. @param domain the domain (an integer, string name or WRFDomain) @param atime the analysis time N)rer¤rlrg)r&r[rPÚdr'r'r(rf5s   zWRFAnl.wrfanl_at_timecCs~| ¡}| d|jf¡d}| ¡D]R}||jkr6q&|j|}|r^d}tj tj   |j ¡¡|j tj   |j ¡|ddq&dS)z!delivers products during a call to run() Delivers the products (FileProduct objects) that were identified by make_products.z%s: make directoryTF©rHr0rN)r“rxrr¤rgr5rJrKr=r>rLr9rMr-)r&r0ÚfirstrPrQr'r'r(rAs   ÿzWRFAnl.deliver_productsNccsD|r||jkr@|j|Vn$| ¡D]}||jkr$|j|Vq$dS)z»!iterates over products Iterates over all Product objects that were identified by make_products. @param domain if provided, only this domain's Products are yieldedN)rgr¤rmr'r'r(ryQs    zWRFAnl.products)N) rIrJrKrLrœr/reržrnrfrryr'r'r'r(rRs   rRcs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚWRFGhostzß!runs a short WRF simulation to generate wrfanl files named "ghost" This is the same as the superclass, WRFAnl, but it renames wrfanl files to "ghost" instead of "wrfanl". Also, history output is disabled.cs0tt|ƒ |¡}| d¡|jddddd|S)z÷!makes the WRFSimulation object for this class. Creates a copy of the WRFSimulation object wrf. This first calls the WRFAnl.make_wrf, and then disables the history stream. @param wrf the wrf argument to __init__úghost_drWi0*i~)rXrVÚstart)r`rrrœÚset_wrfanl_outnamerbrÊrar'r(rœfs zWRFGhost.make_wrfcCs:|dkr dS||jkrdS|j|}| ¡r0dS|j|S)zB!same as get_wrfanl() @param domain the domain of interestNrkrmr'r'r(Ú get_ghostqs  zWRFGhost.get_ghost)rIrJrKrLrœrvrlr'r'rar(rr_s rrcsfeZdZdZedf‡fdd„ Zdd„Zdd„Zd d „Zd d „Z ‡fd d„Z ddd„Z ddd„Z ‡Z S)Ú WRFAnl4Traka!runs wrfanl and generates a track file This subtask of WRFAnl modifies the time of the 1 minute forecast wrfout file from the outer domain to have the analysis time instead so that the tracker can be run to get the initial storm location.ztrankin_dc s*||_||_tt|ƒj||||f|ŽdS)aF!constructor Constructor for WRFAnl4Trak. @param trakdoms what domains should be used for the tracks? either JUST_MOAD or ALL_DOMS or a list of WRFDomain objects @param trakname the track output filename. @param dstore,conf,section,wrf,kwargs passed to superclass constructorN)Ú _trakdomsÚ_trackin_name_patternr`rwr)©r&r†r‡rˆr¤ZtrakdomsZtraknamer+rar'r(r)‘s zWRFAnl4Trak.__init__ccsR|jtkr|j ¡Vn6|jtkr6|jD] }|Vq(n|jD]}|j|Vq". That file is intended to be used to generate the parent domain vortex information.r7rWÚstreamN)rRržr.rfÚ trackin_nameÚ trackin_prodryr}rjrgrhrirðÚ get_nocolonsr=r>rºrr r†r€r9)r&r.Ztrackin_name_patternrPZidomrdrjrr'r'r(rž¨s&   þ zWRFAnl4Trak.make_productscCs| ¡D]}| |¡qdS)a!modifies trakin_* files to say they're at time 0 Calls retime_wrfout for all domains whose trackin_d0X files are requested. This produces the modified 1 minute forecast wrfout files that lie and say they're from the analysis time.N)r}Ú retime_wrfoutrmr'r'r(rÂs zWRFAnl4Trak.postrunc Cs>d}| ¡}| ¡}||}|j|}| dt| ¡ ¡ƒdt| ¡ ¡ƒ¡|j|| ¡ ¡|d}| dt| ¡ƒ¡|  |¡d}|dkr¾|  d||f¡t   | ¡|¡d Sz8t  | ¡|¡tt| d ¡ƒ|| ¡ d ¡fƒWnBtk r8} z"| d | ¡|t| ƒf¡‚W5d } ~ XYnXd S) a !modifies a trakin file's internal timestamp If possible, modifies the stated output time in the one minute forecast trackin_d file to the analysis time. Does this for one domain. For intio files, it does not modify the file, but instead simply renames it. That is done because, at last test, the post does not actually look in an intio wrfout file for the time, so no modification is necessary. @param domain the domain whose trakin file is being modifiedrWz simend = z = râz infile = édrz^%s: renaming instead of running wrfout_newtime since file is (probably) not NetCDF: io_form=%dNZwrfout_newtimez%Y-%m-%d_%H:%M:%Sz6%s (from %s): unable to modify time in NetCDF file: %s)r“r¤rrxrBrar@Ú get_outputr>Ú io_form_forr?r=ÚrenamerÚcopy2rrrrðrzrr•) r&rPr~r0r.ÚdomrdÚinfileÚio_formr˜r'r'r(r‚Ës2  .þ ÿ ÿzWRFAnl4Trak.retime_wrfoutcs@tt|ƒ ¡|j ¡D]"\}}|j|j|| ¡ddqdS)zæ!delivers products Delivers all products (FileProduct objects) created by make_products, including the new trackin_d added by this subclass, and all products added by the superclass WRFAnl.FrpN)r`rwrr€rrMrr“)r&rorQrar'r(rís  ÿzWRFAnl4Trak.deliver_productsNcCsD|dkr|j ¡}|jD]&}|jd|jdkr|j|SqdS)aV!returns a trakin (trackin) Product Returns a trakin (trackin) Product. If a domain is specified, returns the trackin file for that domain. Otherwise, the MOAD (outermost domain) is assumed. @param domain None, or the domain of interest @return a Product for the trakin file for the specified domainNrd)r.rïr€re)r&rPror'r'r(Ú get_trackinùs   zWRFAnl4Trak.get_trackinccs¢|s&|s&|s&tj||dD] }|Vq|r6|dkr6dS|rN|| ¡ ¡krNdS|j ¡D]D\}}|j|}|r„|jdd„|Dƒkr„qX|dk r–||kr–qX|VqXdS)aB!iterates over all products from this task. Iterates over all products from this task. This class adds the trackin_d0* files. @param domain the domain of interest @param domains a list of domains of interest @param time the analysis time @param stream the output stream)rPrWNcSsg|] }|j‘qSr')rd)rsZdmr'r'r(rtsz(WRFAnl4Trak.products..)rRryr¤rðr€rr.rd)r&rPr<rur~rQroZsimdr'r'r(ry s   zWRFAnl4Trak.products)N)NNNN)rIrJrKrLr{r)r}ržrr‚rr‹ryrlr'r'rar(rwŠsÿ   " rwcsBeZdZdZedf‡fdd„ Z‡fdd„Zdd„Zd d „Z‡Z S) ÚWRFGhostForPostaˆ!runs wrf to generate ghost (wrfanl) and wrfout files This class generates wrfghost files, and wrfout files, for each domain. The wrfout files happen at the end of a roughly one minute forecast so that they can be correctly post-processed. However, their internal timestamp has been changed to be for the analysis time. This class derives from WRFAnl4Trak instead of WRFGhost to reuse the wrfout renamer functionality, but it may be used in place of a WRFGhost object. Note that a wrf ghost file is a wrfanl file. There is no difference whatsoever, except in the nomenclature in the HWRF initialization.zghout_dc s"tt|ƒj||||||f|ŽdS)zh!constructor Creates a WRFGhostForPost, passing all arguments to WRFAnl4Trak.__init__()N)r`rŒr)rzrar'r(r)5s ÿÿzWRFGhostForPost.__init__cstt|ƒ |¡}| d¡|S)zS!creates a WRFSimulation that calls its wrfanl files "ghost" files instead.rs)r`rŒrœrurÊrar'r(rœ=s zWRFGhostForPost.make_wrfcCs|dk s t‚| |¡S)z;!returns the ghost wrfout product for the specified domain.N)r8r‹rmr'r'r(Ú get_ghoutDs zWRFGhostForPost.get_ghoutcCs:|dkr dS||jkrdS|j|}| ¡r0dS|j|S)zp!same as get_wrfanl() This works exactly like get_wrfanl() @param domain the domain of interestNrkrmr'r'r(rvIs  zWRFGhostForPost.get_ghost) rIrJrKrLr|r)rœrrvrlr'r'rar(rŒ's ÿ rŒcs‚eZdZdZd‡fdd„ Z‡fdd„Zdd „Z‡fd d „Zdd d„Zdd„Z dd„Z dd„Z d‡fdd„ Z dd„Z dd„Z‡ZS) ÚWRFAtmosaô!The uncoupled HWRF forecast Task. This class runs an atmosphere-only WRF run, using wrfanl files from a prior WRFAnl simulation. It encapsulates an ExternalWRFTask, which performs the actual tracking of products. This allows the post-processors and wrf copy tasks to keep track of the model's output while the model is running. That subtask shows up as ".mon" (for "monitor") relative to this task (so if this task is wrfatmos, then the external task is wrfatmos.mon).TÚauxhist1c sh||_tt|ƒj||||fd|i|—Žtjj|j|j|j | ¡|j d|j |j d|_ |j |j d<dS)a!WRFAtmos constructor The constructor for WRFAtmos. @param dstore the produtil.datastore.Datastore object @param conf the HWRFConfig object for configuration information @param section the section name within that HWRFConfig object @param wrf the hwrf.wrf.WRFSimulation that is to be executed @param keeprun True means the output directory should NOT be deleted @param wrfdiag_stream stream that generates wrfdiag files @param kwargs passed to the parent class constructor.r¢z.mon)r†r‡rˆr¤r€r9rrN) Z_wrfdiag_streamr`rŽr)rgr¤r#r†r‡rˆr€rÚ _WRFAtmos__ex)r&r†r‡rˆr¤r¢Zwrfdiag_streamr+rar'r(r)_s ÿÿýzWRFAtmos.__init__cs0tt|ƒ |¡}| ¡| ¡}| ||¡|S)aˆ!creates a WRFSimulation for an uncoupled forecast run Creates a WRFSimulation for an uncoupled forecast run. Runs the superclass WRFAtmos.make_wrf. Instructs the resulting WRFSimulation to take analysis files as input, and calls the _set_wrf_proc_config to set I/O server and compute grid dimensions. @param wrf the wrf argument to __init__)r`rŽrœÚ analysis_inr“rÈ)r&r¤r0rar'r(rœws  zWRFAtmos.make_wrfcCs|j ¡dS)zq!deletes output files Deletes output files. See the hwrf.wrf.ExternalWRFTask.unrun for details.N)rÚunrunrÓr'r'r(r’…szWRFAtmos.unruncs t|_t|j_tt|ƒ ¡dS)aH!runs the uncoupled WRF forecast Performs all work needed to run the program. Sets the state to produtil.datastore.UNSTARTED. Creates the work directory, CD's to it, runs the initial_prerun, link_fix, link_all_inputs, make_namelist, final_prerun, run_exe, postrun and deliver_products.N)rÚstaterr`rŽrrÓrar'r(r‹sz WRFAtmos.runFcCs|jj||dS)aË!Determines the last output time at which all streams have completed their output. Asks the underlying hwrf.wrf.ExternalWRFTask.last_completed_time() the last time at which all streams have completed their output. If check=True, then all products are checked, otherwise cached information is used. @param streams a list of WRF stream names @param check if True, call produtil.datastore.Datum.check() on any products that are unavailable. Otherwise, cached information is used. @returns None if no times are complete, or a datetime.datetime of the last forecast time at which all streams are complete. )r;)rÚlast_completed_time)r&Ústreamsr;r'r'r(r”–szWRFAtmos.last_completed_timecks|jjf|ŽD] }|VqdS)zÑ!iterates over all WRF products. Iterates over all WRF products. See the hwrf.wrf.ExternalWRFTask.products() for details. @param kwargs passed to hwrf.wrf.ExternalWRFTask.products()N©rry©r&r+rQr'r'r(ry¦szWRFAtmos.productscks|jjf|ŽD] }|VqdS)zê!iterates over all WRF products. Synonym for products(). Iterates over all WRF products. See the hwrf.wrf.ExternalWRFTask.products() for details. @param kwargs passed to hwrf.wrf.ExternalWRFTask.products()Nr–r—r'r'r(Ú exproducts®szWRFAtmos.exproductscKs|jjf|ŽdS)a‚!checks the status of the WRF simulation. Checks the status of the WRF simulation. Should only be used while the simulation is running. This is intended to be run by jobs other than the WRF job, such as the post-processors, to monitor the WRF simulation as it progresses. @param kwargs passed to hwrf.wrf.ExternalWRFTask.wrf_check()N)rÚ wrf_check)r&r+r'r'r(r™¶s zWRFAtmos.wrf_checkr¤Ncs0|dkr| dd¡}tt|ƒj||||ddS)a-!runs the wrf program Called from run(), this runs the wrf program after the directory is set up. @param exename sent to getexe() to get the path to wrf if runner is unspecified @param not_allranks if True, only use one MPI rank. Do not use. @param runner an alternative produtil.prog.Runner to execute. @param sleeptime seconds to sleep between checks of the wrf executables completion. The default sleeptime, if none is specified is 60 seconds rather than the usual 30.Nrüé<)rr r rü)rr`rŽr)r&rr r rürar'r(rÁs  þzWRFAtmos.run_exec Cs.|j ¡}|j ¡|jj|_W5QRXdS)a!checks the wrf state and updates it in the HWRF database file Checks the state of the WRF simulation and copies that information to self.state in the produtil.datastore.Datastore. See hwrf.wrf.ExternalWRFTask.update_state() for details.N)r†Ú transactionrÚ update_stater“)r&r}r'r'r(rœÔs  zWRFAtmos.update_statecOsdS)ab!does nothing Has no effect. This is present only because it is a requirement of the parent class. No delivery is required because the products are all UpstreamFile objects, so the delivery state is set by the post-processors when calling the "check" function of each product. @param args,kwargs ignoredNr')r&Úargsr+r'r'r(rÝszWRFAtmos.deliver_products)Tr)F)r¤FNN)rIrJrKrLr)rœr’rr”ryr˜r™rrœrrlr'r'rar(rŽUs  ÿ   ÿ rŽcsˆeZdZdZd‡fdd„ Zdd„Zedd „ƒZed d „ƒZed d „ƒZ dd„Z ddd„Z dd„Z dd„Z ddd„Zd‡fdd„ Z‡ZS)Ú AnalysisCyclea!runs wrf for an analysis cycle This class is similar to a WRFGhost run, except that it runs a longer simulation (typically 1-6 hours), and provides wrfghost and wrfinput files at the end of it. It also requires wrfghost and wrfinput files as input. Hence, one can run another AnalysisCycle after this one, using the output of the previous. Note that this implementation relies on the fact that wrfghost, wrfanl and restart files are all exactly the same file format (just different times and domains).NFc sNtƒ|_|dk rt|ƒ}||_| |dd¡|_tt|ƒj|||||f|ŽdS)a!constructor for AnalysisCycle Constructor for the AnalysisCycle class. @param dstore the produtil.datastore.Datastore to use @param conf the hwrf.config.HWRFConfig that provides configuration ifnormation @param section the config section in conf @param wrf the hwrf.wrf.WRFSimulation object that is being run @param keeprun if True, the simulation temporary files are not deleted @param simlen simulation length in seconds @param kwargs passed unmodified to FcstTask.__init__()NZ use_restartT) rfÚ_AnalysisCycle__prodcacher"Ú_AnalysisCycle__simlenÚgetboolÚrestartr`ržr))r&r†r‡rˆr¤Úsimlenr¢r+rar'r(r)ós ÿÿzAnalysisCycle.__init__cCs†tƒ|_| ¡ ¡}| ¡}| ¡D]\}d| ¡f}tj |j |¡}||krft |j ||j |d|_ q$t |j ||j |d|j|<q$dS)zÀ!called from constructor, creates Products This is called from the WRFTaskBase constructor. It creates products that will be used by deliver_products() and products()úwrfinput_d%02dr7N)rfrgr¤rïr“rjr=r>rºrr r†r€r8)r&ròr0rPrirjr'r'r(rž s&  ÿ ÿzAnalysisCycle.make_productscCs | ¡ ¡S)z!analysis cycle end time The time at end of the analysis cycle, at which the output wrfanl and wrfinput files are available.)r¤rarÓr'r'r(Ú anlouttime'szAnalysisCycle.anlouttimecCs | ¡ ¡S)z—!analysis cycle start time The time at beginning of the analysis cycle, at which the input wrfanl and wrfinput files must be provided.rdrÓr'r'r(Ú anlintime/szAnalysisCycle.anlintimecCs |jdkrt| d¡ƒ|_|jS)zÓ!simulation length The requested length of the simulation. Note that this may not be the same as the actual simulation length (self.anlouttime-self.anlintime) due to the model timestep.Nr£)r r"r·rÓr'r'r(r£7s zAnalysisCycle.simlencCs| ¡}| ¡t|jƒ}|j|d|jr>|jd|||d|jd|||dd|j dd¡}|dkslt ‚|D]&}|  d¡s‚t ‚|jrp|  d¡spt ‚qp|D]}|j  d d d ¡qœ|  ¡}|  d ¡sÞ|jd |d ||dddD],}|  |¡}|D]}|  |¡rô| |¡qôqâ|S)an!creates the WRFSimulation object Called from the constructor. Generates and returns a new WRFSimulation object that meets these requirements: 1. Read in wrfanl and wrfinput files 2. Output a restart and wrfinput file after self.simlen time 3. Disable history and auxhist 1-3. @param wrf the wrf argument to __init__rUr¢)rXrtrVÚinputoutzwrfinputout_d)rXrtrVÚoutnameÚ time_controlÚ input_outnamerSÚntrackièrWrPrQ)rtrXrV)rÚauxhist2Úauxhist3rW)rÉr‘r"r£r`r¢rbr=rYr8Ú has_outputr>r“r…Ú hide_output)r&r¤r£rªrPr0r~rŠr'r'r(rœBs4    ÿ    zAnalysisCycle.make_wrfcCsF|r$|jd| ¡ ¡jdkr$dS|dk rBt||j|jƒrB|jSdS)a,!the wrfinput_d01 output Product for the specified time and domain @return the wrfinput output Product for the specified time and domain or None if that time and domain are not valid. @param atime the time @param domain the domain (integer, string name or WRFDomain)rdN)rer¤rïr!rer r8rGr'r'r(rYzs ÿzAnalysisCycle.wrfinput_at_timecCs|jS)zN!the wrfinput_d01 output product @returns the output wrfinput ProductrErÓr'r'r(rF‡szAnalysisCycle.get_wrfinputc Cs”| ¡ ¡}| ¡}|j}| dt|ƒf¡tjj|j |d| ¡D]D}d|  ¡f}t j   |j |¡}||kr²|jd||d  ¡}| dt|ƒ|f¡|jj||ddqH|jrø|jd||d  ¡}| d t|ƒ|f¡|j|j||dd| d ¡D]Š}|  ¡}| d t|ƒ|f¡t j   |j |¡} | d t|ƒ| f¡tjj|| d |d |js| ¡|kr|j|j||ddqqHdS)zŒ!delivers products from temporary areas Copies output products from temporary areas to long-term storage locations.z-Deliver products: rst and wrfinput at time %srâr¤r§zDeliver moad %s from %sFrpr¢zDeliver nest %s from %srWzDeliver nest %s to %sT)rr0N)r¤rïr“r¥rxrBr5rJrKrrjr=r>rºr„r@r8rMr¢rgÚ get_outputsrÚ validtime) r&ròr0ÚwhenrPrirjÚfromlocÚoutÚtolocr'r'r(rŒsR ÿ ÿÿ ÿzAnalysisCycle.deliver_productsc Cs ||jkr|j|S| ¡}|d}|dk s0t‚tj |tj | ¡¡¡}|j ¡.}t|j|j ||d}|  ¡|d<||d<W5QRX|r’||_ ||j|<|S)a)Converts a WRFOutput to a Product. Returns a Product for a WRFOutput. The Product is cached in self.__prodcache and as_product() will return that cached value if one is available. Otherwise, a new one is created. Call clear_cached_products() to clear the cache..rN)ÚcategoryÚprodnamer9r~r9) rŸr>r8r=rºr-r†r›r r€r~r9)r&ÚwrfoutÚrelocateÚrelrrjr}Úufr'r'r(Ú as_product±s"     ÿ  zAnalysisCycle.as_productc #sÞ|dk r€|D]p}|dkr:| |¡D]}|j||dVq"q |dkrd| |¡D]}|j||dVqLq |j| ||¡|dVq nZ|dkr’|jVn,|| ¡ ¡kr®|jVdS|dkr¾|jVtt|ƒ  |¡D] }|VqÎdS)au!iterates over all selected products Iterates over all products for the specified domain, or all products if the domain is unspecified or None. @param domain the requested domain (an integer, string name or WRFDomain) @param domains a list of domains of interest @param time the analysis time @param stream the output streamN)r¹) Úget_all_outputsr¼r°r„r8r¤rïr`ržry) r&rPr<r~rur¹ror´rQrar'r(ryÇs( ÿ  zAnalysisCycle.products)NF)NN)F)NNNNF)rIrJrKrLr)ržr/r¥r¦r£rœrYrFrr¼ryrlr'r'rar(ržès" ÿ   8 % rž)QrLÚ__all__r=r‚rãrär\Ú fractionsrÚsysrur2Ú produtil.cdr5Úprodutil.fileopÚprodutil.datastoreÚ produtil.runÚ produtil.progÚ hwrf.hwrftaskrgÚhwrf.exceptionsÚ hwrf.numericsÚhwrf.wrfÚ hwrf.wrfbaserrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#Úobjectr$rMrSrVr\r_rmrpÚhwrftaskÚHWRFTaskrr™r0ÚFractionr^rRrrr{r|rwrŒrŽržr'r'r'r(ÚsP$P  ((, K ..},p ]".