U gT@sXdZddlZddlZddlZddlZddlZddlm Z dgZ e Z Gddde Z dS)a!Base class of tasks run by HAFS. This module contains the HAFSTask class, a subclass of produtil.datastore.Task intended to be the base class of any HAFS task. It provides logging services, easy access to a config file, access to tcvitals, and extends the hafs.config.HAFSConfig string interpolation to include vitals information. It also provides a standard way of setting and querying the work and output directories of a task, and whether the task should scrub its output.N)TaskHAFSTaskcseZdZdZdddeeffdd ZeddZeddZd d Z d d Z ee e dd Z ddZ ddZ ee e ddZeddZeddZeddZddZddZddZefdd Zed!d"ZdHd$d%ZdId&d'ZdJd(d)ZdKd*d+ZdLd,d-ZdMd.d/ZdNd0d1ZdOd2d3ZdPd4d5ZdQd6d7Z dRd8d9Z!dSd:d;Z"dTdd?Z$ee$ddd@Z%dAdBZ&ee&dddCZ'dUdDdEZ(dFdGZ)Z*S)Vra!The base class of tasks run by the HAFS system. This class represents a task to be run by the HAFS system. It can be configured by an hafs.config.HAFSConfig object. Internal state information is stored in an produtil.datastore.Datastore. Each task has its own workdir, outdir and scrub flag, as well as its own vitals information. Execution of a task is done by calling run(), and the products generated by this task can be iterated with products()Nc  s|dkr |}||t|_|tk rD|D]\} } || | q.||_t||_||_ |dk rtt |tstt dt |tst d|dkr| dd}|dks|dkrt j|d|}|dkr| dd}|dks|dkrt j|d|}|jtt|j|f|||d| |dd} |d d} | rH| }n||d<| r\| }n||d <W5QRX|tkrt|d r|j|_nf|dk rt |trtj|_|j|d |j|n$t |tjjr||_nt d dS) a!Creates an HAFSTask @param dstore passed to Datum: the Datastore object for this Task @param conf the conf object for this task @param section the conf section for this task @param taskname Optional: the taskname in the datastore. Default: the section name @param workdir directory in which this task should run. Any value set in the database will override this value. @param outdir directory where output should be copied. This argument must not be changed throughout the lifetime of the HAFS datstore database file. @param storminfo the storm vitals information for the storm this task is running. @param taskvars additonal variables for string expansion, sent to the taskvars arguments of hafs.config.HAFSConfig member functions. @param kwargs passed to the parent class constructor.Nz2The taskname must be None or a basestring subclassz$The section be a basestring subclassworkdirWORKhafsintercom)tasknameloggeroutdirsyndatconfigzYThe storminfo argument to HAFSTask() must be None, a string or tcutil.storminfo.StormInfo)register_hafs_taskdict_HAFSTask__taskvars UNSPECIFIEDitemstvset_confstr_sectionZ _storminfo isinstance TypeErrorconfstrospathjoingetdir transactionsuperr__init__logmetahasattrr storminfotcutil StormInfoupdateZ parse_vitalscopy)selfdstoreconfsectionrrr r#taskvarskwargskvZmworkdirZmoutdir __class__:/lfs/h1/ops/para/packages/hafs.v2.0.8/ush/hafs/hafstask.pyr&s^               zHAFSTask.__init__cCsd}|jdddr|jddd}|dkrl||jjkr>d}q||jjkrtjd||jjfntjd|jj|S) z0Is this the fake storm of a multistorm HAFS run?Fr run_multistorm fakestormidnofakeidTzpMultistorm run: isfakestorm property can not be set since case mismatch between fakestormid: %s and stormid3: %szMultistorm run: isfakestorm property can not be set since No 'fakestormid' in config section of storm %s.Check if you defined a list of multistorm ids.) rgetboolgetstrr#stormid3upperhafs exceptionsHAFSConfigInsane)r( isfakestormr5r2r2r3r>rs$  zHAFSTask.isfakestormcCsd}|jdddrd}|S)zIs this a multistorm HAFS run?Fr r4T)rr7)r( ismultistormr2r2r3r?szHAFSTask.ismultistormcCs6|dd}|s&tj|d|j}|dks2t|S)z`!Returns the directory the class should work in, as set by the "workdir" metadata value.rrr/r!rrrrrAssertionError)r(rr2r2r3 get_workdirs   zHAFSTask.get_workdircCst||d<dS)z!Sets the directory the class should work in. This sets the "workdir" metadata value. @param val the new work directoryrNrr(valr2r2r3 set_workdirszHAFSTask.set_workdirz0!The directory in which this task should be run.cCs6|dd}|s&tj|d|j}|dks2t|S)ze!Gets the directory that should receive output data. This is in the "outdir" metadata value.r rrr@rA)r(r r2r2r3 get_outdirs   zHAFSTask.get_outdircCst||d<dS)z!Sets the directory that should receive output data. Sets the "outdir" metadata value. @param val the new output directoryr NrDrEr2r2r3 set_outdirszHAFSTask.set_outdirzB!The directory to which this task should deliver its final output.cCs |ddS)a!Is this job a real-time forecast job? Is this job running an actual, real-time forecast for an event happening now? This is different than self.conf.realtime: it allows this job's config section to locally override the value.realtimeTconfboolr(r2r2r3rJszHAFSTask.realtimecCs |ddS)z=!Should subprograms' outputs be redirected to separate files?redirectFrKrMr2r2r3rNszHAFSTask.redirectcCs |ddS)zJ!Should temporary files be deleted as soon as they are not needed?scrubTrKrMr2r2r3rOszHAFSTask.scrubcCs8t|}|dddkr*tjd|f||j|<dS)a!Sets a taskvar option's value. Sets an object-local (taskvar) value for option "opt" to value "val". This will override config settings from the HAFSConfig object. These are sent into the taskvars= parameter to the various HAFSConfig member functions (hence the "tv" in "tvset"). @param opt the name of the taskvar @param val the string value of the optionr__z.%s: invalid option name. Cannot begin with __N)rr;r<InvalidConfigOptNamer)r(optrFsoptr2r2r3rs zHAFSTask.tvsetcCst|}|j|=dS)za!Deletes an object-local value set by tvset. @param opt the name of the taskvar to deleteNrrr(rSrTr2r2r3tvdelszHAFSTask.tvdelcCst|}|j|S)z!Gets a taskvar's value Returns the value of an object-local (taskvar) option set by tvset. @param opt the taskvar whose value should be returnedrUrVr2r2r3tvgetszHAFSTask.tvgetcCs(|tkrt|jdkSt|}||jkS)a<!Is a taskvar set? If an option is specified, determines if the given option has an object-local (taskvar) value. If no option is specified, returns True if ANY object-local values (taskvars) exist for any options. @param opt Optional: the name of the taskvar being checked.r)rlenrrrVr2r2r3tvhaveszHAFSTask.tvhavecCs|jS)z>!The dict of object-local values used for string substitution.)rrMr2r2r3r,szHAFSTask.taskvarsFcCs(|dkr|j}|jj||||||jdS)a!Alias for self.conf.getint for section self.section. @param opt the option name @param section Optional: the section. Default: self.section. @param default if specified and not None, then the default is returned if an option has no value or the section does not exist @param badtypeok is True, and the conversion fails, and a default is specified, the default will be returned. @param morevars dict of more variables for string expansionNmorevarsr,)rrgetintrr(rSdefault badtypeokr+r\r2r2r3confints zHAFSTask.confintcCs(|dkr|j}|jj||||||jdS)a!Alias for self.conf.getstr for section self.section. @param opt the option name @param section Optional: the section. Default: self.section @param default if specified and not None, then the default is returned if an option has no value or the section does not exist @param badtypeok is True, and the conversion fails, and a default is specified, the default will be returned. @param morevars dict of more variables for string expansionNr[)rrr8rr^r2r2r3rs zHAFSTask.confstrcCs(|dkr|j}|jj||||||jdS)a!Alias for self.conf.getfloat for section self.section. @param opt the option name @param section Optional: the section. Default: self.section @param default if specified and not None, then the default is returned if an option has no value or the section does not exist @param badtypeok is True, and the conversion fails, and a default is specified, the default will be returned. @param morevars dict of more variables for string expansionNr[)rrgetfloatrr^r2r2r3 conffloats zHAFSTask.conffloatcCs(|dkr|j}|jj||||||jdS)a!Alias for self.conf.getbool for section self.section. @param opt the option name @param section Optional: the section. Default: self.section @param default if specified and not None, then the default is returned if an option has no value or the section does not exist @param badtypeok is True, and the conversion fails, and a default is specified, the default will be returned. @param morevars dict of more variables for string expansionNr[)rrr7rr^r2r2r3rL s zHAFSTask.confboolcCs(|dkr|j}|jj||||||jdS)a!Alias for self.conf.get for section self.section. @param opt the option name @param section Optional: the section. Default: self.section @param default if specified and not None, then the default is returned if an option has no value or the section does not exist @param badtypeok is True, and the conversion fails, and a default is specified, the default will be returned. @param morevars dict of more variables for string expansionNr[)rrgetrr^r2r2r3confget-s zHAFSTask.confgetcCs"|dkr|j}|jj|||jdS)z!Alias for self.conf.items for section self.section. @param section Optional: the section. Default: self.section. @param morevars variables for string substitutionNr[)rrrr)r(r+r\r2r2r3 confitems:szHAFSTask.confitemscKs|j|fd|i|S)z!Alias for self.icstr for backward compatibility @param string the string to expand @param section Optional: the section in which to expand it. Default: self.section. @param kwargs: more arguments for string substitutionr+)icstrr(stringr+r-r2r2r3 confstrinterpAszHAFSTask.confstrinterpcKs|j||f|||jd|S)au!Alias for self.timestr for backward comaptibility @param string the string to expand @param ftime: the forecast time @param atime: Optional: the analysis time. Default: self.conf.cycle @param section Optional: the section in which to expand it. Default: self.section. @param kwargs: more arguments for string substitution)atimer+r,)timestrrr(riftimerkr+r-r2r2r3conftimestrinterpIs zHAFSTask.conftimestrinterpcCs|dkr|j}|j|||S)a!Get a raw configuration value before string expansion. Returns the raw, uninterpolated value for the specified option, raising an exception if that option is unset. Will not search other sections, and will not search the taskvars, unlike other conf accessors. @param opt the option of interest @param section Optional: the section. Default: self.section @param default Optional: value to return if nothing is found.N)r+rgetraw)r(rSr_r+r2r2r3confrawTs zHAFSTask.confrawcKsD|dkr|j}|jr(d|kr(|jj|d<|jj||fd|ji|S)a!Expands a string in the given conf section. Given a string, expand it as if it was a value in the specified conf section. Makes this objects tcvitals, if any, available via the "vit" variable while interpolating strings. @param string the string to expand @param section Optional: the section in which to expand it. Default: self.section. @param kwargs: more arguments for string substitutionNvit __taskvars)rr#__dict__r strinterprrhr2r2r3rgas  zHAFSTask.icstrcKsj|dkr|j}|jr(d|kr(|jj|d<d|krF|jj||||f|S|jj||||fd|ji|SdS)a!Expands a string in the given conf section, including time vars Expands a string in the given conf section (default: self.section), and includes forecast and analysis time (default: conf.cycle) information in the variables that can be expanded. The mandatory ftime argument is the forecast time which will be used to expand values such as fHH, fYMDH, etc. The optional atime will be used to expand aHH, aYMDH, etc., and the two will be used together for forecast minus analysis fields like fahr. See hafs.config.timestrinterp for details As with self.icstr, this class's vitals are available via the "vit" variable while interpolating strings. @param string the string to expand @param ftime: the forecast time @param atime: Optional: the analysis time. Default: self.conf.cycle @param section Optional: the section in which to expand it. Default: self.section. @param kwargs: more arguments for string substitutionNrrr,rs)rr#rtr timestrinterprrmr2r2r3rlqs zHAFSTask.timestrcCs|jjd||||jdS)z!Alias for hafs.config.HAFSConfig.get() for the "dir" section. @param opt the option name @param default Optional: default value if nothing is found. @param morevars Optional: more variables for string substitutiondirr[rrdrr(rSr_r\r2r2r3rszHAFSTask.getdircCs|jjd||||jdS)z!Alias for hafs.config.HAFSConfig.get() for the "exe" section. @param opt the option name @param default Optional: default value if nothing is found. @param morevars Optional: more variables for string substitutionexer[rxryr2r2r3getexeszHAFSTask.getexecCs|jS)z7!Returns this HAFSTask's hafs.config.HAFSConfig object.)rrMr2r2r3getconfszHAFSTask.getconfz-!The HAFSConfig for this HAFSTask (read-only)cCs|jS)z8!Returns this HAFSTask's section name in the HAFSConfig.)rrMr2r2r3 getsectionszHAFSTask.getsectionz>!The confsection in self.section for this HAFSTask (read-only)cCs(|dkr|jS|j|jdt|S)aM!Obtain a logging domain. Creates or returns a logging.Logger. If subdom is None or unspecified, returns a cached logger for this task's logging domain. Otherwise, returns a logger for the specified subdomain of this task's logging domain. @param subdom Optional: the desired logging domainN.)_loggerrr rr)r(Zsubdomr2r2r3r sz HAFSTask.logccsdS)a !Iterates over all inputs required by this task. Iterates over dict-like objects suitable for input to hafs.input.InputSource.get. Each object contains the following keywords: * dataset: string name of the dataset (gfs, gdas1, gefs, enkf, etc.) * item: string name of the object (ie.: gfs_sf, gfs_sfcanl, bufr) * atime: self.conf.cycle * ftime: only present when relevant: the forecast time, in a format accepted by to_datetime_rel * enkfmem: only present when relevant: the ENKF member ID * obstype: only present when relevant: the bufr data type. Other keywords may be present if needed. They will be passed on by hafs.input.InputSource for string replacement.Nr2rMr2r2r3 inputiterszHAFSTask.inputiter)NFNN)NFNN)NFNN)NFNN)NFNN)NN)N)NN)NN)N)NNN)NN)NN)N)+__name__ __module__ __qualname____doc__rrpropertyr>r?rCrGrrHrIr rJrNrOrrWrXrZr,rarrcrLrerfrjrorqrgrlrr{r|r*r}r+r r __classcell__r2r2r0r3rs L                   )rrertcutil.numericsr$tcutil.storminfo produtil.logprodutilprodutil.datastorer__all__objectrrr2r2r2r3s