U  gr?@sdZdgZddlZddlZddlZddlZddlZddlZ ddl Z ddl Z ddl Z ddl mZddl mZddlmZmZmZmZddlmZmZddlmZdd lmZmZdd lmZGd ddeZdS) zt!Contains Bufrprep, which converts data tanks to bufr files, and otherwise preprocesses the prepbufr and bufr files.BufrprepN)set_ecflow_event)NamedDir) make_symlink deliver_file isnonemptyfortlink)runexe)HWRFTask) to_datetimeto_datetime_rel) GSIInputErrorcseZdZdZdfdd ZddZdddZd d d Zd!d d Zd"d dZ ddZ d#ddZ ddZ ddZ ddZddZd$ddZZS)%rz!This is a HWRF task that preprocesses observations in data tanks to create bufr files suitable for input to hwrf.gsi classes. It also does other preprocessing of bufr and prepbufr files.Nc  stt|j|||fd|i||j}|dkr4|j}t||_|j|_d|_ d|_ d}d|kr|d}t |t j jr|||_ n4t |tr|}n$|dkrntdt|jt|f|j dkr|dkr|d}t j |j||j|_ dS)a!Bufrprep constructor @param dstore passed to Datum: the Datastore object for this Task @param conf the conf object for this task (passed to HWRFTask) @param section the conf section for this task (passed to HWRFTask) @param taskname Optional: the taskname for this product in the datastore @param atime the analsysis time @param kwargs Additional keyword arguments are passed to the hwrf.hwrftask.HWRFTask.__init__ constructortasknameN999 in_catalogzIn hwrf.bufrprep.Bufrprep.__init__, in_catalog must be None, a basestring or a DataCatalog. You provided an object of type %s, with value %s.catalog)superr__init__rcycler _atimeworkdir_dirname_stormid _in_catalog isinstancehwrfinput DataCatalogstr TypeErrortype__name__reprconfstrconf) selfdstorer%sectionratimekwargs incat_nameink __class__:/lfs/h1/ops/prod/packages/hmon.v3.2.7/ush/hwrf/bufrprep.pyrs@     zBufrprep.__init__cCs|}|jd}t|dd}|jj}|dkr@|jj|_nf|dkr`dd|jjf|_nF|dkrdd |jjf|_n&|d krdd |jjf|_nd |_| d |j|jS)z!The storm ID.z%Y%m%d%HriALz%s%02d1EP2CP3rzget input storm id %s) logrstrftimeint storminfo pubbasin2stnumrupperinfo)r&loggerr)yearbasinr/r/r0 getstormidEs     zBufrprep.getstormidcCsP|d}t}|dD]0}|}t|dkr||kr|j|||dqdS)z!Links or copies all needed bufr files to the local directory. @param atime the analysis time @param morevars passed to hwrf.config.HWRFConfig.get() and similar routines to define string replacementsobstypes,r)r)morevarsN)r$setsplitstriplengrab_obstype_section)r&r)rFolisttouchedosectiontrimr/r/r0 grab_bufrYs  zBufrprep.grab_bufrcs|}t|tst|}|dkr*|j}n t||j}|j|d}|j|d}|j|d}|d||||ft }|j |} t dd| Dfdd } | D]\} } | d krq| | q|D]} |d | ||ft| tst d t| jt| f|jj||||| d } | dks8| dkrNd| f}||qtj| rtj| }| || }t| ||ddqd| | f}||qdS)a!Copies or links observations specified in the obstype sections to the current working directory. The section listed in self.section should contain an obstype option, whose value is a comma separated list of section names. This method reads every section in that list. For each section, the option keys are the local directory filenames expected by GSI, while the values are the data type part of the operational filename (ie.: the satwind in gfs.t12z.tm00.satwind.bufr_d). There are a few special keys: * dataset - the name of the dataset for hwrf.input purposes * item - the name of the item for hwrf.input purposes * type - the type of observation: satellite, or anything else. At present, only "satellite" has any special meaning. If the type is "satellite" then the entire section will be skipped if sat_da=False in this task's config section. Once the section is parsed, the files are all linked to this directory. @param section Name of the section to read. @param atime Analysis time. @param morevars A dict for additional string replacements in the hwrf.config.HWRFConfig.get() family of functions.Ndatasetitemr!z6process obs section %s with dataset=%s item=%s type=%scSsg|]\}}||fqSr/r/).0kvr/r/r0 sz1Bufrprep.grab_obstype_section..cs|S)Nr/)ftotdictr/r0z/Bufrprep.grab_obstype_section..)rQrRr!z%Find obstype=%s in dataset=%s item=%szIn bufrprep.Bufrprep.link_bufr, the obstypes parameter must be an iterable container of basestrings. One of the elements was a %s (value %s) instead.r)r@obstypez/%s: Could not find a location for this obstype.T)r@forcez1%s: Observation file is empty or non-existant: %s)r8rrrr r%getlowerwarninglistitemsdictappendr r!r"r#rlocateprodutilfileoprospathbasenamer)r&r(r)rFr@rQrRotyperDrenamer localnamer^theremsgbnonr/rYr0rKfs^            zBufrprep.grab_obstype_sectioncKs|dkr|j}n t||j}|}|d}|d}|jj||f||d|}|dksd|dkrdt|t|f}||t|n&t j |s|d}| |t|t |dd |d dS) z!Links or copies the prepbufr file to the local directory. @param atime The analysis time. @param kwargs More string substitution variables for the hwrf.config.HWRFConfig family of functions.N bufr_dataset prepbufr_itemr)r@r_z5Could not find the prepbufr file (item=%s dataset=%s)z: is non-existant or emptyz prepbufr.ALLTkeepr@)rr r8r$rrhr#rcrrirjrerrorr)r&r)r*r@rurvrqrrr/r/r0 grab_prepbufrs.        zBufrprep.grab_prepbufrc Cs|dkr|j}n t||j}|}|jdd}tjtjdd}|j|d|}tj| dd}t |rt j d |t j j|d d |d n@|jdd drd}t jj|tj|nt j d|dS)z!Gets the dumped TDR data from the pure shell jhwrf_bufrdump job. @param atime The analysis time. @param morevars More string substitution variables for the hwrf.config.HWRFConfig.get() family of functions.Nconfig fcst_catalogDCOMROOTus007003dcomWORKhwrfz tldplr.ibmzTDR dump found: tldplrbufrTrxZ expect_tdrFzJTDR data is missing. This HWRF IT test requires that TDR data be present.zTDR dump missing: )rr r8r%rarkrljoinenvirongetdirrripostmsgrjrgetbooljloggercriticalr exceptions ExpectedTDR) r&r)rFr@ input_catalogdcomenvrZtdrbufrrrr/r/r0tdrdumps* zBufrprep.tdrdumpcCs|}tddd|dkrZd}||}|dd}tj|j|jj |jj |d}np|d kr| d }||}|dd}|d d}tj|j||d }n"|d krd}||}tj|}|dkr|d kr|j r|tk}tjj ||dn |ddS)z!pre-process prepbufr data Options: * option 0: make no change * option 1: remove some inner-core data * option 2: flag/unflag mass and dropsonde u, v data * option 3: unflag HS3 dropsonde dataz./prepbufr.ALLz ./prepbufr)3Zhwrf_rem_prepbufr_typ_in_circleRRADCg)ZRLATCZRLONCrZ!hwrf_change_prepbufr_qm_in_circleRBLDC)rrZhwrf_change_prepbufr_qm_typrr@z,no greater than 3 option, skip prep_prepbufrN)r8rgetexe conffloatrir r envr;latlon write_vitalsredirectcheckrunr?)r&optionr@fprogprogrcmdrr/r/r0 prep_prepbufrs<       zBufrprep.prep_prepbufrtcvitalsc Csb|dkr|}|dt|ft|d}||jdW5QRXtj |s^t dS)z!Writes the tcvitals (from self.storminfo) to the specified file. @param filename Name of the file to write @param logger A logging.Logger for log messagesNzWriting tcvitals to %swt ) r8r?r#openwriter; as_tcvitalsrkrlexistsAssertionError)r&filenamer@rWr/r/r0r s  zBufrprep.write_vitalsc Cs@|}z|d|jftj|jrH|d|jft|jt|j|j dt |j }|rt| n| tjdr||d|j|||n |d|dd}|dkr||||W5QRXWn<tk r:}z|jd t|d d W5d }~XYnXd S) z!Runs and delivers the results.zRun bufrprep in directory %szDelete old data in %s)ryrz storm id %szSkip read tdr bufr. prepbufrpreprzbufrprep failed: T)exc_infoN)r8r?rrkrlrshutilrmtreerscrubboolrealtimerrPisfilerCr readtdrstmid readtdrtime set_tdrstatusconfintr{rdeliver_products Exceptionrr)r&r@rrer/r/r0r s4        z Bufrprep.runcCsl|d|}d}||}d|j|jj|f}tj||j >}|j rX||k}tjj ||ddS)z$!Runs the hwrf_readtdrstmid program.rZhwrf_readtdrstmid%s/logs/%s_%s.logrN) r8r?rrr.r"rir r rrrr&r@rrr8rr/r/r0r9s zBufrprep.readtdrstmidcCsf|d|}d}||}d|j|jj|f}tj|}|j rR||k}tjj ||ddS)z#!Runs the hwrf_readtdrtime program.rZhwrf_readtdrtimerrN) r8r?rrr.r"rir r rrrr/r/r0rFs  zBufrprep.readtdrtimec Cs|d|}tj|jd}tj|jd}t|}t|}tj|jd| d}|r|st |d}| dW5QRX|dn|s|d n |d d S) z*!Create a TDR status file in com directoryrz stmid.datZtdrflagcomz{stormlabel}.tdrrzASSIMILATE_TDR=YES z8tdr bufr is available for this storm, ASSIMILATE_TDR=YESz1tdr bufr is not for this storm, ASSIMILATE_TDR=NOz:this tdr bufr file is a small edge dump, ASSIMILATE_TDR=NON) r8r?rkrlrrrr%ricstrrr)r&r@ZstmidouttimeoutZ rightstormZ smalledgedump tdrflagfilerWr/r/r0rSs   zBufrprep.set_tdrstatusc Ks|dkr|j}n t||j}|}tj|j|dddkr|d}|j j |f||d|}t j |j|}tjjd|d|dt|jr td r |jd d }|j j |||d d }t j |j|}tjjd |d|d|d}tjjd |d|dtddS)z!Delivers output products to the intercom directory. @param atime the analysis time @param kwargs Sent to hwrf.input.DataCatalog.parse()NrrrvrwprepbufrFrxrtdr_new_obstyperRtldplrr]Tz%{com}/{out_prefix}.tldplr.tm00.bufr_dZFoundTDR)rr r8rirjmakedirsoutdirrr$rparserkrlrrrrrr%rarr) r&r)r*r@rvrqitrRZtdrprodr/r/r0resR     zBufrprep.deliver_products)NN)NN)NN)N)NN)rN)N)r" __module__ __qualname____doc__rrCrPrKr{rrrr rrrr __classcell__r/r/r-r0rs) K  % "  )r__all__rkr hwrf.hwrftaskrhwrf.exceptionsdatetimeprodutil.fileopri produtil.cdprodutil.cluster produtil.logprodutil.ecflowrrrrrr produtil.runr r r hwrf.numericsr r rrr/r/r/r0s