3 2g:D@sdZddlZddlZy ddlZWnek r4YnXddlZddlZddlZddlm Z ddl Z ddl j j ZddlmZddlZddlmZmZddlmZdd Zy*Gd d d ejZd d ZejeeWnek rYnXddZddZyejdeej dWnek r"YnXddZ!ddZ"dAddZ#ddZ$ddZ%dBddZ&dCd d!Z'dDd"d#Z(dEd$d%Z)d&d'Z*dFd(d)Z+d*d+Z,d,d-Z-dGd.d/Z.d0d1Z/dHd2d3Z0d4d5Z1dId7d8Z2d9d:Z3d;d<Z4dJd=d>Z5d?d@Z6dS)Ka% This file provides utilities for processing different configuration file formats. Supported formats include: a) YAML b) JSON c) SHELL d) INI e) XML Typical usage involves first loading the config file, then using the dictionary returnded by load_config to make queries. N)dedent)minidom) list_to_str str_to_list) run_commandc Cs$t|d}tj|}WdQRX|S)zSafe load a yaml filerN)openyamlZ safe_load) config_filefcfgrG/lfs/h1/ops/prod/packages/aqm.v7.0.11/ush/python_utils/config_parser.pyload_yaml_config/s rcs"eZdZdZdfdd ZZS) custom_dumperz.Custom yaml dumper to correct list indentationFcstt|j|dS)NF)superrincrease_indent)selfZflowZ indentless) __class__rrr=szcustom_dumper.increase_indent)FF)__name__ __module__ __qualname____doc__r __classcell__rr)rrr:srcCs,t|jdkr |jd|ddS|jd|S)Nrztag:yaml.org,2002:str|)style)len splitlinesZrepresent_scalar)Zdumperdatarrr str_presenter@sr cCstj|tdddS)z,Get contents of config file as a yaml stringF)Dumper sort_keysZdefault_flow_style)r dumpr)r rrrcfg_to_yaml_strKsr$cCs|j|}djdd|DS)z"Custom tag hangler to join stringscSsg|] }t|qSr)str).0irrr Vszjoin_str..)Zconstruct_sequencejoin)loadernodeseqrrrjoin_strSs r.z !join_str)LoadercCs tjj|S)z$A filter for jinja2 that joins paths)ospathr*)argrrr path_join_sr3cCstjjtj|djdS)zLA filter for jinja2 that gives us a date string for x number of days ago)daysz%Y%m%d00)datetimedatetoday timedeltastrftime)r2rrrdays_agoesr:c s|dkr |}t|tsdSxz|jD]l\}}t|trFt||q&t|tjr\t|jnt|tfddd D}|r&d|krg}n t j d}g}x|D]}t j t j t jd}t|jd<t|jd <|j|} y| jf||}Wnnt jjk r} zWYdd} ~ XnHtk r(Yn4tk r<Yn t|d |YnX|j|qWt|tjrd j||_q&d j|||<q&WdS) zi Updates yaml_dict inplace by rendering any existing Jinja2 templates that exist in a value. Nc3s|]}|kr|VqdS)Nr)r'Zele)v_strrr szextend_yaml..{{{%z {{[^}]*}}|\S)r+Z undefinedr3r:z: r%)r=r>) isinstancedictitems extend_yamlETElementr&textanyrefindalljinja2Z EnvironmentZ BaseLoaderZStrictUndefinedr3filtersr:Z from_stringZrender exceptionsZUndefinedError TypeErrorZeroDivisionErrorprintappendr*) Z yaml_dictZ full_dictkvZ is_a_templateZ templatesrtemplateZj2envZj2tmpler)r;rrBmsF         rBcCs\y$t|d}tj|}WdQRXWn2tjk rV}ztd|WYdd}~XnX|S)zLoad json config filerNzUnable to load json file )r jsonloadZJSONDecodeError Exception)r r r rSrrrload_json_configs   rWcCstj|ddddS)z,Get contents of config file as a json stringF)r"indent )rTdumps)r rrrcfg_to_json_strsr\cCst|d&}|j}|jdd}|jdd}WdQRXtjjtjdttjd}t|d }|j |WdQRXzt ||}Wdtj |X|S) z:Load shell config file with embedded structure in commentsrz# [[z\  Nz_temp.z.iniw) r readreplacer0r1r*getcwdr&getpidwriteload_ini_configremove) file_name return_stringfiler Z temp_filerrrload_shell_as_ini_configs  "  rjc Csyt||}|SYnXtj}td|d|d|d}t|\}}}|j}i}x@|D]8}|jd} |d| } t|| dd|} | || <q`W|S)aLoads old style shell config files. We source the config script in a subshell and gets the variables it sets Args: config_file: path to config file script Returns: dictionary that should be equivalent to one obtained from parsing a yaml file. z! #!/bin/bash t1="./t1.z" t2="./t2.z," (set -o posix; set) > $t1 { . z|; set +x; } &>/dev/null (set -o posix; set) > $t2 diff $t1 $t2 | grep "> " | cut -c 3- rm -rf $t1 $t2 =Nr)rjr0rcrrrfindr) r rhr pidcode_Z config_strlineslidxrPrQrrrload_shell_configs"       rscCsd}x|jD]\}}t|trb|r4|d|}n|}|d|d7}|t||7}|d7}qt|}t|tr||d|d7}q|jdd}|jdd }||d |d 7}qW|S) z2Get contents of config file as shell script stringr%.z# [z] rZrk'"r^z='z' )rAr?r@cfg_to_shell_strrlistra)r kname shell_strrPrQn_knamev1rrrrws"    rwcstjj|s ttd|dtjt_j |fddj D}x8|j D],\}}x"|j D]\}}t ||||<qjWqXW|S)zALoad a config file with a format similar to Microsoft's INI fileszS The specified configuration file does not exist: 'rucsi|]}tj||qSr)r@rA)r's)configrr <sz#load_ini_config..) r0r1existsFileNotFoundErrorr configparserZRawConfigParserr&Z optionxformr`ZsectionsrAr)r rhZ config_dictroZvsrPrQr)r~rre-s  recCs(||krtd|n |||SdS)z0Finds the value of a property in a given sectionzSection not found: N)KeyError)r~Zsectionkeyrrr get_ini_valueCs rcCsd}x|jD]\}}t|trb|r4|d|}n|}|d|d7}|t||7}|d7}qt|d}t|tr||d|d7}q||d|d 7}qW|S) z)Get contents of config file as ini stringr%rtr]z] rZTrkz='z' )rAr?r@cfg_to_ini_strrrx)r ryZini_strrPrQr{r|rrrrNs   rcCsNi}xD|D]<}tt|dkr4t||}|||j<q t|j|||j<q W|S)z!Convert an xml tree to dictionaryr)rrx xml_to_dicttagrrE)rootrhr childrrrrrgs   rcCsdtj|}xT|jD]H\}}t|trt|d}tj|dd}tj|}|jdd}|jdd}|S)z+Get contents of config file as a xml stringrunicode)encodingz )rYz"rv)rrCtostringrZ parseStringZ toprettyxmlra)r rrrrrcfg_to_xml_strs     rcCsRi}xH|jD]<\}}| s$||krt|trBt|}|j|q|||<qW|S)a"Flatten a recursive dictionary (e.g.yaml/json) to be one level deep Args: dictionary: the source dictionary keys: list of keys on top level whose contents to flatten, if None all of them Returns: A one-level deep dictionary for the selected set of keys )rAr?r@ flatten_dictupdate) dictionarykeysZ flat_dictrPrQrrrrrs    rcCsXi}xN|jD]B\}}t|tr8t||}|rP|||<q||jkr||||<qW|S)aStructure a dictionary based on a template dictionary Args: dict_o: dictionary to structure (flat one level structure) dict_t: template dictionary used for structuring Returns: A dictionary with contents of dict_o following structure of dict_t )rAr?r@structure_dictr)dict_odict_tZ struct_dictrPrQrrrrrs     rFcCsx|jD]\}}t|trHt|j|tr>t||||q|||<q ||jkr | s||dkst||dksd||kr |||<q WdS)zUpdate a dictionary with another Args: dict_o: flat dictionary used as source dict_t: target dictionary to update Returns: None Nrz{{)rAr?r@get update_dictrr)rrZprovide_defaultrPrQrrrrs      rcCshi}x^|jD]R\}}||jkrX||}t|tr`t|tr`t||}|r`|j|q|||<qW|S)aKCheck if a dictionary's structure follows a template. The invalid entries are returned as a list of lists. If all entries are valid, returns an empty dictionary Args: dict_o: target dictionary dict_t: template dictionary to compare structure to Returns: dict: Invalid key-value pairs. )rArr?r@check_structure_dictr)rrZinvalrPrQr|rrrrrs     rcsJg}x.|D]&}tj|}|tt|jj7}q Wfdd|D}|S)zFilter dictionary keys based on a list of keys Args: dict_o: the source dictionary keys_regex: list of keys to retain (could be regex exp.) csi|]}||qSrr)r'rP)rrrrszfilter_dict..)rGcompilerxfiltermatchr)rZ keys_regexrrPrrr)rr filter_dicts   rcCsrtjj|ddd}|dkr*t||S|dkrsf       I   -