U  gj$@s*dZdddddddgZdd lZdd lZdd lZdd lZdd lZdd lZdd lZdd l Z dd l Z dd l Z dd l Z dd lZ dd lZ dd lZdd lmZdd lmZdd l mZmZdd l TddlmZddlmZddlmZmZeZGdddeZGdddeZeZGdddeZ ddZ!ddddddddddd d d!d!d!d"d"d#Z"ddddddddddd d d!d!d!d"d"d$Z#ddddddddddd d d!d!d!d"d"d%Z$ddddddddddd d d!d!d!d"d"d&Z%e&d'd(d)gZ'eZ(Gd*d+d+e Z)d,dZ*d3d.dZ+d4d/d0Z,Gd1ddeZ-Gd2dde j.jZ/d S)5a!Parses UNIX conf files and makes the result readily available The produtil.config module reads configuration information for a production system from one or more *.conf files, via the Python ConfigParser module. This module also automatically fills in certain information, such as fields calculated from the tcvitals or date. The result is accessible via the ProdConfig class, which provides many ways of automatically accessing configuration options. from_filez from-string confwalker ProdConfigZ fordriver ENVIRONMENTProdTaskN) ConfigParser)StringIO) DatastoreTask)*) to_datetime) Formatter) NoOptionErrorNoSectionErrorc@seZdZdZdS)DuplicateTaskNamez]!Raised when more than one task is registered with the same name in an ProdConfig object.N)__name__ __module__ __qualname____doc__rr (?: \' (?! \} ) | [^'] )* ) \' \} | \{ \" (?P (?: \" (?! \} ) | [^"] )* ) \" \} | (?P \{ (?P [^\}:!\['"\{] [^\}:!\[]* (?: \. [a-zA-Z_][a-zA-Z_0-9]+ | \[ [^\]]+ \] )* ) (?: ! (?P[rs]) )? (?: : (?P (?: [^\{\}]+ | \{[^\}]*\} )* ) )? \} ) | (?P \{\{ ) | (?P \}\} ) | (?P [^\{\}]+ ) | (?P . ) ) ZqescapeZdqescapeZleft_setrZZ right_set}rFZreplacement_fieldrGrHrIerrorz'Single '{' encountered in format stringz'Single '}' encountered in format stringzUnexpected %s in format string) rDr= TypeErrortyperr>listrefinditergroupappendr?)r6resultrFrGrHrImrrrr.sV              r. %Y%m%d%H%M%Y%m%d%H%Y%m%d%Y%y%C%m%d%H%M)ZfYMDHMZfYMDHZfYMDZfyearZfYYYYZfYYZfCCZfcenZfmonthZfMMZfdayZfDDZfhourZfcycZfHHZfminuteZfmin)ZaYMDHMZaYMDHZaYMDZayearZaYYYYZaYYZaCCZacenZamonthZaMMZadayZaDDZahourZacycZaHHZaminuteZamin)Zam6YMDHMZam6YMDHZam6YMDZam6yearZam6YYYYZam6YYZam6CCZam6cenZam6monthZam6MMZam6dayZam6DDZam6hourZam6cycZam6HHZ am6minuteZam6min)Zap6YMDHMZap6YMDHZap6YMDZap6yearZap6YYYYZap6YYZap6CCZap6cenZap6monthZap6MMZap6dayZap6DDZap6hourZap6cycZap6HHZ ap6minuteZap6minfahrfaminfahrmincs*eZdZdZdfdd ZddZZS)ConfTimeFormattera!internal function that implements time formatting Like its superclass, ConfFormatter, this class is part of the implementation of ProdConfig, and is used to interpolate strings in a way similar to string.format(). It works the same way as ConfFormatter, but accepts additional keys generated based on the forecast and analysis times: fYMDHM - 201409171200 = forecast time September 17, 2014 at 12:00 UTC fYMDH - 2014091712 fYMD - 20140917 fyear - 2014 fYYYY - 2014 fYY - 14 (year % 100) fCC - 20 (century) fcen - 20 fmonth - 09 fMM - 09 fday - 17 fDD - 17 fhour - 12 fcyc - 12 fHH - 12 fminute - 00 fmin - 00 Replace the initial "f" with "a" for analysis times. In addition, the following are available for the time difference between forecast and analysis time. Suppose the forecast is twenty-three hours and nineteen minutes (23:19) after the analysis time: fahr - 23 famin - 1399 ( = 23*60+19) fahrmin - 19 Fcstt|jt|ddS)z"!constructor for ConfTimeFormatterr1N)r(rr)boolr0r2rrr)Ws zConfTimeFormatter.__init__c Cst}|dd7<|dtjkr8t|d|d|zvt|trT||W`S||krh||}nd|kr|dr||dkr|d|}nd|kr|tkr|dt|}n\d|kr|tkr|dt|}n6d|kr|t kr|dt dd }|t |}nd|krT|t krT|dt dd }|t |}nd|krd|kr|t krtj|d|d\}}|d krt|}n@|d krt|d |}n$|d krt|}nt|d |}n2|d} | dkr2|d| } || dd} | s&|dd} | r>| }n |dd} |dd} | r| r| | |rr| | |}nB| | dr| | ddD] } | | |r| | |}q|tkr| d|r| d|}n| d|r| d|}|tkrtdt|t| ft|tr|ddksB|ddkrz&||||}|dk s`t|WWJStk r}z t|d|d|t|W5d}~XYnX|WS|dd8<XdS)a!return the value of a variable, or a substitution Never call this function. It is called automatically by str.format. It provides the value of an variable, or a string substitution. @param key the string key being analyzed by str.format() @param args the indexed arguments to str.format() @param kwargs the keyword arguments to str.format()rMrNrOrPrR__ftime__atimeri`Trr<rrSNrQrVrWrXrYz Cannot find key %s in section %srZr[)r`r\r]r^rDr_ FCST_KEYSstrftimeANL_KEYS ANL_M6_KEYSdatetime timedelta ANL_P6_KEYSTIME_DIFF_KEYSprodutilnumerics fcst_hr_minrr!rarbrcr>r=r-rCrInterpolationMissingOptionError)rrdr7r8rfZam6Zap6ihoursiminutesrgrhrirerjrkerrrr<[s                     zConfTimeFormatter.get_value)F)rrrrr)r<rmrrr2rr4s"rc Cst}t|g}t|dkr|d}||kr0q||||D]`\}} |||rb|||| ||krDt| dD]*} | } t| dkrx| |krx| | qxqDqdS)a!walks through a ConfigParser-like object performing some action Recurses through a ConfigParser-like object "conf" starting at section "start", performing a specified action. The special variable whose name is in recursevar specifies a list of additional sections to recurse into. No section will be processed more than once, and sections are processed in breadth-first order. For each variable seen in each section (including recursevar), this will call selector(sectionname, varname) to see if the variable should be processed. If selector returns True, then acceptor(section, varname, value) will be called. @param conf the ConfigParser-like object @param start the starting section @param selector a function selector(section,option) that decides if an option needs processing (True) or not (False) @param acceptor a function acceptor(section,option,value) run on all options for which the selector returns True @param recursevar an option in each section that lists more sections the confwalker should touch. If the selector returns True for the recursevar, then the recursevar will be sent to the acceptor. However, it will be scanned for sections to recurse into even if the selector rejects it.rrWN) setr=lenpopadditemsreversedrbstriprv) restartselectorZacceptorZ recursevartouchedZ requestedsecrdr%sec2trimrrrrs      FcCs.t|tstdtt|d}|||S)z!Reads the specified conf file into an ProdConfig object. Creates a new ProdConfig object and instructs it to read the specified file. @param filename the path to the file that is to be read @return a new ProdConfig objectz:First input to produtil.config.from_file must be a string.r)rDr=rprrread)filenamer1rerrrrs   cCs.t|tstdtt|d}|||S)z!Reads the given string as if it was a conf file into an ProdConfig object Creates a new ProdConfig object and reads the string data into it as if it was a config file @param confstr the config data @return a new ProdConfig objectzZ!d?d@Z"dAdBZ#dedCdDZ$dEdFZ%dfdGdHZ&dIdJZ'dgdKdLZ(dhdMdNZ)didOdPZ*djdQdRZ+dkdSdTZ,dldUdVZ-dmdWdXZ.dYdZZ/dnd[d\Z0dod]d^Z1dS)pra!a class that contains configuration information This class keeps track of configuration information for all tasks in a running model. It can be used in a read-only manner as if it was a ConfigParser object. All ProdTask objects require an ProdConfig object to keep track of registered task names via the register_task_name method, the current forecast cycle (cycle property) and the Datastore object (datastore property). This class should never be instantiated directly. Instead, you should use the produtil.config.from_string or produtil.config.from_file to read configuration information from an in-memory string or a file.NF;cCstd|_|j}t|_tt||_t t||_ d|_ t |_ |dkrZt||dn||_t|j_|jd|jdt|_dS)a!ProdConfig constructor Creates a new ProdConfig object. @param conf the underlying configparser.ConfigParser object that stores the actual config data. This was a SafeConfigParser in Python 2 but in Python 3 the SafeConfigParser is now ConfigParser. @param quoted_literals if True, then {'...'} and {"..."} will be interpreted as quoting the contained ... text. Otherwise, those blocks will be considered errors. @param strict set default to False so it will not raise DuplicateOptionError or DuplicateSectionError, This param was added when ported to Python 3.6, to maintain the previous python 2 behavior. @param inline_comment_prefixes, defaults set to ;. This param was added when ported to Python 3.6, to maintain the previous python 2 behavior. Note: In Python 2, conf was ConfigParser.SafeConfigParser. In Python 3.2, the old ConfigParser class was removed in favor of SafeConfigParser which has in turn been renamed to ConfigParser. Support for inline comments is now turned off by default and section or option duplicates are not allowed in a single configuration source.Z prodconfigN)strictinline_comment_prefixesrXrY)logging getLogger_logger threadingRLock_lockr'r _formatterr_time_formatter _datastorer _tasknamesr_confr= optionxform add_sectionrr_fallback_callbacks)rrer1rrloggerrrrr)s    zProdConfig.__init__cCs|jjo|jjSr4)rr1rr5rrrr1/szProdConfig.quoted_literalscCs.|ddd}|jD]}t||||}q|S)aC!Asks whether the specified fallback is allowed. May perform other tasks, such as alerting the operator. Calls the list of functions sent to add_fallback_callback. Each one receives the result of the last, and the final result at the end is returned. Note that ALL of the callbacks are called, even if one returns False; this is not a short-circuit operation. This is done to allow all reporting methods report to their operator and decide whether the fallback is allowed. Each function called is f(allow,name,details) where: - allow = True or False, whether the callbacks called thus far have allowed the fallback. - name = The short name of the fallback. - details = A long, human-readable description. May be several lines long. @param name the name of the emergency situation @warning This function may take seconds or minutes to return. It could perform cpu- or time-intensive operations such as emailing an operator. rXallow_fallbacksF)getboolrr)rnamedetailsZallowZfcrrrfallback4s zProdConfig.fallbackcCs|j|dS)aW!Appends a function to the list of fallback callback functions called by fallback() Appends the given function to the list that fallback() searches while determining if a workflow emergency fallback option is allowed. @param function a function f(allow,name,details) @see fallbacks()N)rrv)rfunctionrrradd_fallback_callbackUs z ProdConfig.add_fallback_callbackcCs$tt|}|j|||S)z!read config data and add it to this object Given a string with conf data in it, parses the data. @param source the data to parse @return self)rr=rreadfprB)rsourcefprrrras  zProdConfig.readstrTc Cst|}t|}t|}|}|r@|dkr6t}qHt|}n|sH|St}tt}|D]} t | tst dt | j t | f|r|| td| } | r|r\|r|d| d| dt | df| d|| d| d<q\|r"tj|| } tj| r"|| q\|r>|d|tfq\|d |tfq\|D]} tj| s|| d td nJtj| s|| d td n"tj| s|r|| d |r|dt | || qX|D] \} } | \}}| | ||qdS)af!Given a list of arguments, usually from sys.argv[1:], reads configuration files or sets option values. Reads list of strings of these formats: - /path/to/file.conf --- A configuration file to read. - section.option=value --- A configuration option to set in a specified section. Will read files in the order listed, and then will override options in the order listed. Note that specified options override those read from files. Also, later files override earlier files. @param args Typically argv[1:] or some other list of arguments. @param allow_files If True, filenames are allowed in args. Otherwise, they are ignored. @param allow_options If True, specified options (section.name=value) are allowed. Otherwise they are detected and ignored. @param rel_path Any filenames that are relative will be relative to this path. If None or unspecified, the current working directory as of the entry to this function is used. @returns self Nz}In produtil.ProdConfig.from_args(), the args argument must be an iterable of strings. It contained an invalid %s %s instead.zw(?x) (?P
[a-zA-Z][a-zA-Z0-9_]*) \.(?P