o pgy@sdZddlZddlZddlZddlZddlZddlZddlZ ddl Z ddl Z ddl Z ddlZddlmZddZddZdd Zd d Zd d ZddZddZddZddZddZddZddZddZddZd d!Zd"d#Z d$d%Z!d&d'Z"d(d)Z#d*d+Z$d,d-Z%d.d/Z&d0d1Z'd2d3Z(d4d5Z)d6d7Z*d8d9Z+d:d;Z,dd?Z.d@dAZ/dBdCZ0dDdEZ1dFdGZ2dHdIZ3dJdKZ4dLdMZ5dNdOZ6dPdQZ7dRdSZ8dTdUZ9dVdWZ:dXdYZ;dZd[Zd`daZ?dbdcZ@dS)dz Name: global_det_atmos_util.py Contact(s): Mallory Row (mallory.row@noaa.gov) Abstract: This contains many functions used across global_det atmos. N)sleepcstddtfdddDrtjddd}nt}|jdkrBtd d|jd t|jt |jd Sd S) zu! Run shell command Args: command - list of agrument entries (string) Returns: z Running  c3s|] }|dvVqdS)rN)join).0ZmarkcommandM/lfs/h1/ops/prod/packages/evs.v1.0.19/ush/global_det/global_det_atmos_util.py sz$run_shell_command..)"'|*>Tshellr FATAL ERROR: z gave return code N) printrany subprocessrun returncodeargsstrsysexit)rZ run_commandrrr run_shell_commands    rcCs2tj|std|tj|ddddSdS)zr! Make a directory Args: dir_path - path of the directory (string) Returns: zMaking directory T)modeexist_okN)ospathexistsrmakedirs)Zdir_pathrrr make_dir*s r$c Cstjtjddd}tjtjddd}tjtjddtjdtjdtjd d tjd |}tj|sGtd |d td|d|d|}|S)z! Write out full call to METplus Args: conf_file_name - METplus conf file name (string) Returns: metplus_cmd - full call to METplus (string) METPLUS_PATHZushzrun_metplus.pyZPARMevsZmetplus_configz machine.confSTEP COMPONENTRUN_ VERIF_CASEr DOES NOT EXISTz -c r r!renvironr"rrr)Zconf_file_nameZ run_metplusZ machine_confZ conf_fileZ metplus_cmdrrr metplus_command7s"   r/cCsdtjtjdtjd|}tj|s!td|dtdd|}|D]}|d|}q'|S)a! Write out full call to python Args: python_script_name - python script name (string) script_arg_list - list of script agruments (strings) Returns: python_cmd - full call to python (string) USHevsr'rr+r,zpython rr-)Zpython_script_nameZscript_arg_listZ python_scriptZ python_cmdZ script_argrrr python_commandPs   r1cCsvd|vsd|vr d}nd}tj|r-tj|dkrd}|St|d|dd }|St|d|d d }|S) aK! Checks to see if file exists and has size greater than 0 Args: file_name - file path (string) Returns: file_good - boolean - True: file exists,file size >0 - False: file doesn't exist OR file size = 0 z/com/z/dcom/ZWARNINGZNOTErTz: z empty, 0 sizedF does not exist)r r!r"getsizer) file_nameZ alert_wordZ file_goodrrr check_file_exists_sizees  r5c Cstj|st|dm}|d|dkr7|d|dd|d|dd |d d d n'|d |d|dd|d|dd|d d|d d |d|d|d|dWdn1s{wYt|ddSdS)ab! This writes a missing model file to a log Args: log_missing_file - log of missing file (string) missing_file - missing file path (string) model - model name (string) init_dt - initialization date (datetime) fhr - forecast hour (string) w #!/bin/bash anlexport subject="z Analysis z!Data Missing for EVS global_det" echo "Warning: No z analysis was zavailable for valid date %Y%m%d%Hz" z > mailmsg zexport subject="Frz Forecast z forecast was zavailable for fecho "Missing file is " >> mailmsg !echo "Job ID: $jobid" >> mailmsg )cat mailmsg | mail -s "$subject" $MAILTO Nr)r r!r"openwriteupperchmod)log_missing_file missing_filemodelinit_dtfhrlmfrrr log_missing_file_models4     rKcCstj|sVt|d;}|d|d|dd|d|dd|d d |d |d |d |dWdn1sIwYt|ddSdS)a1! This writes a missing truth file to a log Args: log_missing_file - log of missing file (string) missing_file - missing file path (string) obs - observation name (string) valid_dt - initialization date (datetime) ar7r9z Data Missing for EVS z global_det" r:z data was available for z valid date r;z " > mailmsg r=r>r?r@Nr)r r!r"rArBrD)rErFobsZvalid_dtrJrrr log_missing_file_truths      rNcCs0t|rtd|d|t||dSdS)z! This copies a file from one location to another Args: source_file - source file path (string) dest_file - destination file path (string) Returns: zCopying  to N)r5rshutilcopy) source_file dest_filerrr copy_files rTcCDtd|dd|tjd}t|d|d|ddS)aW! Converts GRIB1 data to GRIB2 Args: grib1_file - string of the path to the GRIB1 file to convert (string) grib2_file - string of the path to save the converted GRIB2 file (string) Returns: zConverting GRIB1 file rto GRIB2 file CNVGRIBz -g12  > /dev/null 2>&1Nrr r.system) grib1_file grib2_filecnvgribrrr convert_grib1_grib2   r^cCrU)aE! Converts GRIB2 data to GRIB1 Args: grib2_file - string of the path to the GRIB2 file to convert grib1_file - string of the path to save the converted GRIB1 file Returns: Converting GRIB2 file rzto GRIB1 file rWz -g21 rXNrY)r\r[r]rrr convert_grib2_grib1r_racCrU)aK! Converts GRIB2 data to GRIB2 Args: grib2_fileA - string of the path to the GRIB2 file to convert grib2_fileB - string of the path to save the converted GRIB2 file Returns: r`rrVrWz -g22 rXNrY)Z grib2_fileAZ grib2_fileBr]rrr convert_grib2_grib2r_rbcCNtjd}tj|d|ddd}|jdkr#td|dd}|Sd }|S) a?! Checks if GRIB1 file is corrupt Args: grib1_file - string of the path to the GRIB1 file to convert Returns: file_is_corrupt - True means file is corrupt False means file is not corrupt WGRIBr 1> /dev/null 2>&1Trr WARNING: is corruptFr r.rrrr)r[rd chk_corruptfile_is_corruptrrr check_grib1_file_corrupt  rkcCrc) a?! Checks if GRIB2 file is corrupt Args: grib2_file - string of the path to the GRIB2 file to convert Returns: file_is_corrupt - True means file is corrupt False means file is not corrupt WGRIB2rreTrrrfrgFrh)r\rmrirjrrr check_grib2_file_corruptrlrncCs@tjd|ddd}|jdkrtd|dd}|Sd}|S) aU! Checks if netCDF file is corrupt Args: netcdf_file - string of the path to the netCDF file to convert Returns: file_is_corrupt - True means file is corrupt False means file is not corrupt zncks -H reTrrrfrgF)rrrr)Z netcdf_filerirjrrr check_netcdf_file_corrupt$s  rocCsxdd|D}dd|D}|dkr|}n|dkr|}|d} |d} t|dkr7tttj|td } nd } tj|| d } tj|| d } g}| }|| kr|dkrZ|}n|dkr`|}|D]I}|d krkd}nt|}|dkr||tj|d }n |dkr|tj|d }| d|vr| d|vri}||d<||d<t ||d<| |qb|tjt| d }|| ksS|S)a! Creates a list of dictionaries containing information on the valid dates and times, the initialization dates and times, and forecast hour pairings Args: date_start - verification start date (string, format:YYYYmmdd) date_end - verification end_date (string, format:YYYYmmdd) date_type - how to treat date_start and date_end (string, values:VALID or INIT) init_hr_list - list of initialization hours (string) valid_hr_list - list of valid hours (string) fhr_list - list of forecasts hours (string) Returns: time_info - list of dictionaries with the valid, initalization, and forecast hour pairings cSg|]}|dqSzfillrhrrrr Qz!get_time_info..cSrprqrsrurrr rwRrxVALIDINITrr,dtyper;r8hours%HZ valid_timeZ init_time forecast_hour) lennpmindiffarrayintdatetimestrptime timedeltastrftimerappend)Z date_startZdate_end date_type init_hr_list valid_hr_listfhr_listZvalid_hr_zfill2_listZinit_hr_zfill2_listZdate_type_hr_listZdate_type_hr_startZdate_type_hr_endZdate_type_hr_incZ date_start_dtZ date_end_dtZ time_infodate_dt valid_time_dt init_time_dtrIrtrrr get_time_info:sd        rcCsFd||d}|ddkr|}nd||d}|dkr!|d}|S)z! Get a initialization hour Args: valid_hour - valid hour (integer) forecast_hour - forecast hour (integer) r~rr) valid_hourr init_hourrrr get_init_hours rcCs>||d}|ddkr|}n||d}|dkr|d}|S)z! Get a valid hour Args: init_hour - intit hour (integer) forecast_hour - forecast hour (integer) r~rr)rrrrrr get_valid_hours   rcCs2d}gd}tt|dkr|t|}|dD]x}|D]k}|d|d} | dkrd} | | kr|dvri|d|ddd dd d} |d|ddd dd d} n|d|ddd d} |d kr|| } n|d kr| dkrt|dkr|d} n|} n| dkr|d} n| dkr|d} n|} n|dkr|| } n|dkr|d|ddd dd d} t t|t| }| dkrt|dkr|d} ns|} np| dkr|d} nf| dkr|d} n[|} nX|dkr2|d|ddd dd d} |t j t| d}|| } n.|dkr\|d|ddd dd d} |t j t| d}|| } n||} |dvrx| d|d| d| d | }n| d|d| d | }| d7} | | ks:q"t j||}q|S)a! Creates a filled file path from a format Args: unfilled_file_format - file naming convention (string) valid_time_dt - valid time (datetime) init_time_dt - initialization time (datetime) forecast_hour - forecast hour (string) str_sub_dict - other strings to substitue (dictionary) Returns: filled_file_format - file_format filled in with verifying time information (string) /)lead lead_shiftvalid valid_shiftinit init_shiftr{z?fmt=r,)rrrrr}zshift=?rrz%1H z%2Hz%3Hrrrrrz?shift=)rlistkeyssplitcount partitionrrrtrrrreplacer r!r)Zunfilled_file_formatrrrZ str_sub_dictZfilled_file_formatZformat_opt_listZfilled_file_format_chunkZ format_optZ nformat_optZformat_opt_countshiftZformat_opt_count_fmtZreplace_format_opt_countZforecast_hour_shiftZinit_shift_time_dtZvalid_shift_time_dtrrr format_fillers               lrcCsdtjtd|dd}t|rt|st||n t||d|t | dt ||dS)a!! Do prep work for FNMOC production files Args: source_file - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: atmos.rrrfnmocrN) r r!rgetcwd rpartitionr5rnrbrKrrtrTrRrSrHr prep_methodrE prepped_filerrr prep_prod_fnmoc_file*  rcCstjtd|dd}t|r,|dkr"t|s!t||nt|s+t||n t ||d|t | dt||dS)a! Do prep work for CMC production files Args: source_file - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rrrrprecipcmcrN) r r!rrrr5rnrTrkrKrrtrrrr prep_prod_cmc_fileGs   rcCdtjtd|dd}t|rt|st||n t||d|t | dt||dS)a(! Do prep work for CMC regional production files Args: source_file - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rrrr cmc_regionalrN r r!rrrr5rnrTrKrrtrrrr prep_prod_cmc_regional_filehrrcCr)a! Do prep work for IMD production files Args: source_file - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rrrrimdrNrrrrr prep_prod_imd_filerrc Cstjd}tjd}tj|d}tjtd|dd} | d} | d} |d kr|d kr4d } nt|d kr=d } nd |d} dD]D} |d| }| dkrT| }n| dkrZ| }t|r|t |s{t |d|d| d|d|d|gqEt ||d|t | dqEt| rt| rt || | | gn5d|vr|}t|rt |st |d|d|d|d|d| gn t ||d|t | dt| |dS)a/! Do prep work for JMA production files Args: source_file_format - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rdEXECevsZ jma_mergerrrr.tmp1.tmp2fullr8:anlr:rv)nsz {hem?fmt=str}rrr | grep "" | -i -grib -o jmarrz | grep "0-hr" | N)r r.r!rrrrrr5rkrrKrrtrT)source_file_formatrSrHrrrErdrZJMAMERGEr working_file1 working_file2 wgrib_fhrhemZhem_source_fileZ working_filerRrrr prep_prod_jma_files        rc Cstjd}tj|d}tj|d}tjd} tjtd|dd} | d} |d kr|d kr7d } nt|d kr@d } nd |d} t|rht|sgt | d|d| d| d|d| gn t ||d|t | dt| rt dd| gt dd| gt || | gn.d|vrt|rt|st |d|| gntt j ddkrt ||d|t | dtj| rt dd| gt dd| gt| |dS)a(! Do prep work for ECMWF production files Args: source_file - source file format (string) dest_file - destination file (string) init_dt - initialzation date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rZecm_gfs_look_alike_new pcpconformrdrrrrrrr8rrrrvrrrrecmwfrrD640chgrprstprodrrN)r r.r!rrrrr5rkrrKrrtrnowrr"rT) rRrSrHrrrErZECMGFSLOOKALIKENEW PCPCONFORMrdrrrrrr prep_prod_ecmwf_filesn       rc Cstjd}tjd}tjd}tj|d} tjtd|dd} | d} | d } |d krid d d d ddddddddddddddddddd dd!d"d#d"d$d%d&d'd(d)d*d+d,d-d.d/d0d1d0d2 } |t| vr|d kr| d }d }d }n| |}|}|d krd }n|d3}|d4|}t |rt |st |d5|d6|d7|d5|d8| gn t ||d9|t |d:t | rt | | | |gnbd;|vr=|}d<}t |rt|st |d5|d=d>| gn t ||d9|t |d:t | rt| | t | r=t||}t |d5| d6t |d?|d@|d5| d8| gt| |dAS)Ba1! Do prep work for UKMET production files Args: source_file_format - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rrdrmZukm_hires_mergerrrrrrrr8ZAAT06ZBBT12ZCCT18ZDDTZ24ZEETZ30ZFFTZ36ZGGTZ42ZHHTZ48ZIITZ54ZJJT60Z66ZKKTZ72Z78ZQQTZ84ZLLTZ90ZTTTZMMTZUUTZNNTZVVTZOOTZ11TZPPAZ22T) Z96Z102Z108Z114Z120Z126Z132Z138Z144rvz{letter?fmt=str}rrrrukmetrr z -if ":TWATP:" -set_var "APCP" z -fi -grib -rN)r r.r!rrrrrrr5rkrrKrrtrnrarrT)rrSrHrrrErrdrmZ UKMHIRESMERGErrrZukmet_fhr_id_dictZfhr_idZfhr_strrrRZsource_file_accumZsource_file_accum_fhr_startrrr prep_prod_ukmet_file's              rc Cstjd}tj|d}tjtd|dd}tjt|ddd} d|vrWt|r>t|s=t|| n t ||d|t | d t| rWt |d| |gt ||d S) a/! Do prep work for DWD production files Args: source_file_format - source file format (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rrrrrrz.tmprdwdrN)r r.r!rrrr5rnrarKrrtrrT) rRrSrHrrrErrrrrrr prep_prod_dwd_files.    rc Cs8tjd}tjtd|dd}tjtd|dd}tjtd|dddd} d|vrt| sbt|rUt||t|rTt d|gn t ||d |t | d t| rd } t || } t| st |d | d t | d|d|d | d|gt||dSdSdS)a#! Do prep work for METRFRA production files Args: source_file - source file(string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: rdrrrrz.gzrZgunzipmetfrarr~rrrrrN)r r.r!rrrrr5rTrrKrrtrrk) rRrSrHrrrErdrrrZ file_accumZfhr_accum_startrrr prep_prod_metfra_filesd        rcCstjtd|dd}t|rFt|st||t|rDt |d}|j dddd|j ddd<| t||dSdSd|vrMd }nd |vrSd }t ||d | |dS) a! Do prep work for OSI-SAF production files Args: daily_source_file - daily source file (string) daily_dest_file - daily destination file (string) date_dt - date (datetime object) log_missing_file - text file path to write that production file is missing (string) Returns: rrrrrLtimeNZ_nh_nhZ_sh_shzOSI-SAF )r r!rrrr5rorTnetcdfDatasetZ variablescloserNrC)Zdaily_source_fileZdaily_dest_filerrEr prepped_datarrrr prep_prod_osi_saf_files(   rcCstjtd|dd}t|rt|st||nt||d|t|rNt j |ddd}t j dd }|d d d d d |d d d <| t||d S)a}! Do prep work for GHRSST OSPO production files Args: source_file - source file (string) dest_file - destination file (string) date_dt - date (datetime object) log_missing_file - text file path to write that production file is missing (string) Returns: rrrrz GHRSST OSPOrLZNETCDF3_CLASSICformatz1981-01-01 00:00:00z%Y-%m-%d %H:%M:%SrNrr)r r!rrrr5rorTrNrrrrr)rRrSrrErrZghrsst_ospo_date_since_dtrrr prep_prod_ghrsst_ospo_files(  $rcCsTtjtd|dd}t|r!t|st||dSdStt |d|dS)a! Do prep work for ALEXI GET-D production files Args: source_file - source file (string) dest_file - destination file (string) date_dt - date (datetime object) log_missing_files - text file path to write that production file is missing (string) Returns: rrrrzGET-DN) r r!rrrr5rorTrNrE)rRrSrZlog_missing_filesrrrr prep_prod_get_d_file1s rcCs"t||||i}|dddd}|dddd}tj|d|ddddd}tj|st||||i} d| vrRt| |||d |d*Sd | vrat| |||d |d*Sd | vrpt | |||d |d*Sd | vrt | |||d |d*Sd | vrt| |||d|d*Sd| vrt | |||d|d*Sd| vrt | |||d|d*Sd| vrt | |||d|d*Sd| vrt | |||d|d*Sd|vr3d| vr3t |dvr3t| r&tjd| ddddd} | jdkr| jdd} tjd| d| dd|dd} d*Std td!| d"|t| |d*St|| |||d#d*Sd} d}|d$kra|ra|d%d&krTt |d'krTd(}d(} n t |d)dkrad(}d(} | rt| r{td!| d"|t| |d*S|rt|| |||d#d*Sd*Sd*Sd*S)+a! This get a model file and saves it in the specificed destination Args: valid_time_dt - valid time (datetime) init_time_dt - initialization time (datetime) forecast_hour - forecast hour (string) source_file_format - source file format (string) dest_file_format - destination file format (string) Returns: rrrr mail_missing_.r).shz dcom/navgemrz wgrbbul/jma_z wgrbbul/ecmwfzwgrbbul/ukmet_hiresz qpf_verif/jmarz qpf_verif/UWDzqpf_verif/ukmoz qpf_verif/dwdzqpf_verif/METFRA.precip.zcom/gfs)rzwgrib2 z | grep "APCP"Tutf8)rZcapture_outputencodingrz -match '^(z):' z-grib rz8Could not get APCP record number(s) linking files instedLinking rOrrr00HFr~N)rrr r!rrr"rrrrrrrr5rrrstdoutrrsymlinkrKrt isnumeric)rrrrdest_file_formatrSlog_missing_dirrGrErRZwgrib2_apcp_grepZfirst_apcp_recZwgrib2_apcp_matchZ link_fileZwrite_missing_filerrr get_model_fileIs         Crc Cst|||dgi}|dddd}dg}|dkr!|d|D]R} | dkr,|} n| dkr2|} tj|d| d|dd d dd } t| ||dgi} tj|sut| rnt d | d |t | |q#t | | ||q#dS)a$! This get a truth/observation file and saves it in the specificed destination Args: valid_time_dt - valid time (datetime) obs - observation name (string) source_prod_file_format - source productoin file format (string) source_arch_file_format - source archive file format (string) evs_run_mode - mode EVS is running in (string) dest_file_format - destination file format (string) Returns: r8rrprodZ productionZarchrr)rrrrrrON) rrrr r!rrr"r5rrrN) rrMZsource_prod_file_formatZsource_arch_file_format evs_run_moderrSrZsource_loc_listZ source_locrrErRrrr get_truth_filesD    r c Cs tj|d|dd}tj|d|dd|d}|d}g}i}i}|d d d d }d d|DD]U}i|t|<i|t|<|tj|d} |ddkrtj|d||dd} |ddkrtj|d|ddd|d|dd|dd|dd} tj|d|ddd|d|dd|dd|dd} |ddkr|dd kr| d!d"vr|d#d$kr|| t|d%|t|d&<||tj|d'dt|d'd%|t|d(<|| t|d%|t|d&<||tj|d'dt|d'd%|t|d(<nt|dd)vrq|} |d#}|}d$}|| krp||tj|dt|d%|t|d*t|d+<||tj|dt|d%|t|d*t|d+<|d+7}|d'7}|| ks/n|| t|d%|t|d&<|| t|d%|t|d&<|dd,kr|dd-kr|| t|d%|t|d&<tj|d|dd.||dd/|dd|dd0d1} tj|d|dd.||dd/|dd|dd0d1} || t|d%|t|d&<q:|dd2kr|ddkr|dd3vr`tj|d||d4d5d} |dd6d }tj|d|dd.||dd7|dd|d8d1} tj|d|dd.||dd7|dd|d8d1} nD|ddkr|dd9krtj|d|ddd|d|dd:|dd|dd;d d0d<d=} tj|d|dd.||dd>|dd|dd?d@dA} tj|d|dd.||dd>|dd|dd?d@dA} n|dd)vrQtj|d|ddd|d|dd|dd|dd;d dBdCdDdE} tj|d|dd.||dd>|dd|dd?dFdA} tj|d|dd.||dd>|dd|dd?dFdA} nStj|d||dd} |ddGkrtj|d|dd.||dd7|dd|dd0d1} tj|d|dd.||dd7|dd|dd0d1} |dd3vr3t |dd6d dHd }t|g}|dId$dJkr||d$kr| t||n;t |dI|krt |t |dI}d+}||kr||d+t |dI}|d$kr | t||d+7}||ks|D]}| |d+}|| t|d%|t|d*t|<qn`|ddGkrY|| t|d%|t|d&<|| t|d#d%|t|d(<n:|ddkr|dd9kr| d!d"vr|d#d$kr|| t|d%|t|d&<n|| t|d%|t|d&<|| t|d%|t|d&<q:|dd,krtj|d||dd} |ddKkrL|ddLkrL|| t|d%|t|d&<||tj|d'dt|d'd%|t|d(<tj|d|dd.||ddM|dd|ddN} tj|d|dd.||ddM|dd|ddN} || t|d%|t|d&<||tj|d'dt|d'd%|t|d(<q:|dd-kr|| t|d%|t|d&<tj|d|dd.||ddO|dd|dd0d1} tj|d|dd.||ddO|dd|dd0d1} || t|d%|t|d&<q:|ddPkr|ddkre|ddQkrdR|dvrtj|d|dd.||ddS|dT|ddU} tj|d|dd.||ddS|dT|ddU} n;tj|d|dd.||dd|dd|ddN} tj|d|dd.||dd|dd|ddN} || t|d%|t|d&<|ddkr}|dd9kr}tj|d|dd.||dd>|dd|dd?d@dA} n|ddkr|ddVkrtj|d|dd.||ddW|dd|ddX} n|dd3vr|dd6d }tj|d|dd.||dd7|dd|d8d1} n|ddQkrtj|d|dd.||dd>|ddY|dZ dd[dFdA} nn|ddGkr.tj|d|dd.||dd7|dd\|d]d0d1} nF|dd^krXtj|d|dd.||dd>|dd|dd?dFdA} ntj|d||d_} n|dd,krt|dd-kr|dd`krtj|d|ddd|d|ddO|dd|ddd[d=} nC|ddKkr|ddakrtj|d|ddd|d|dd:|dd|dd;d d0d<db} n tj|d||d_} |ddKkr9|ddakr9tj|d|dd.||ddS|ddc|ddd|ddU} tj|d|dd.||ddS|ddc|ddd|ddU} n;tj|d|dd.||ddM|dd|ddN} tj|d|dd.||ddM|dd|ddN} || t|d%|t|d&<|| t|d%|t|d&<q:t|D]}g}t||D]d}t| |||de|||df|||dgi}tj| r| dh|ddkr|ddivr| |||dgq|dd2kr|ddjvr| |||dgq| dkqtdldm|D rt|d$k r| |qttjttj|t dntdn}t|}g}t|D]}t||D]t}t| |||de|||df|||dgi}t| |||de|||df|||dgi}tj| rt|||||dg|v r||||dg qH|||dg|v r||f|v r| ||f qH q>t|d$k rdh}ndk}|ddk r|ddk r|dd k rt |dd'd$k s|ddVk r|}|dd2k r |dd,k r |ddLk r |}|||fS)oaB! Check what model files or don't exist Args: job_dict - dictionary containing settings job is running with (strings) Returns: model_files_exist - if non-zero number of model files exist or not (boolean) fhr_list - list of forecast hours that model files exist for (string) model_copy_output_DATA2COMOUT_list - list of file to copy from DATA to COMOUT DATEvalid_hr_startr;DATAr*r)r&MODELrr r, cSg|]}t|qSr)rrirrr rwz%check_model_files..r JOB_GROUP reformat_datadataz.{init?fmt=%Y%m%d%H}.zf{lead?fmt=%3H} grid2gridMETplus_outputr(rz{valid?fmt=%Y%m%d}Z grid_stat_ VERIF_TYPEjob_namez8_{lead?fmt=%2H}0000L_{valid?fmt=%Y%m%d_%H%M%S}V_pairs.ncCOMOUT pres_levs GeoHeightAnomr)rrr~r) valid_date init_daterZfile1rZfile2)sea_icesstfiler,grid2obsptypez.{valid?fmt=%Y%m%d}regrid_data_plane__initz({init?fmt=%Y%m%d%H}_fhr{lead?fmt=%3H}.nc assemble_data)precip_accum24hrprecip_accum3hrrz{init?fmt=%Y%m%d%H}. precip_accum pcp_combine_Z Accum_initZDailyAvg_GeoHeightAnomZanomaly_Z DailyAvg_z{init?fmt=%Y%m%d%H}_zfhr{lead?fmt=%3H}.ncZ daily_avg_z_init{init?fmt=%Y%m%d%H}_z)valid{valid_shift?fmt=%Y%m%d%H?shift=-12}zto{valid?fmt=%Y%m%d%H}.ncz_{lead?fmt=%2H}0000L_z{valid?fmt=%Y%m%d}_z{valid?fmt=%H}0000V_zpairs.ncz)valid{valid_shift?fmt=%Y%m%d%H?shift=-24}snowrvZ MODEL_accumrsfc TempAnom2mZ point_stat_z4_{lead?fmt=%2H}0000L_{valid?fmt=%Y%m%d_%H%M%S}V.statZ merged_ptype_generate_statsr DailyAvg_ExtentZstat_analysis_fcstZ _obsosi_saf_z _SL1L2_{valid?fmt=%Y%m%d%H}.statZ WindShearZ wind_shear_z-_init{init?fmt=%Y%m%d%H}_fhr{lead?fmt=%3H}.ncZ_DailyAvg_Concentration hemispherezinit{init?fmt=%Y%m%d%H}_Z _24hrAccum_Z file_name_varr!z$.{init?fmt=%Y%m%d%H}.f{lead?fmt=%3H}ZPtypeDailyAvg_TempAnom2mzfhr{lead?fmt=%3H}.statZ _obsprepbufr_prepbufrrrrT)rZConcentrationNHZConcentrationSHZSST)r.Fcs|]}|dkVqdSTNrrxrrr r )z$check_model_files..r|)rrr r!rrrrrrrrindexrCrrrr"allrrZasarrayuniquerQZdeepcopyrTremove) job_dict valid_date_dtverif_case_dirrGrZfhr_check_input_dictZfhr_check_output_dictZjob_dict_fhr_listrIZ init_date_dtinput_file_formatZoutput_DATA_file_formatZoutput_COMOUT_file_formatZ fhr_avg_endZ fhr_avg_startZ fhr_in_avgZnfr*Zfhr_in_accum_listZnfiles_in_accumZfhr_nfZ fhr_in_accumZfile_numZfhr_keyZfhr_key_input_files_exist_listZ fhr_fileN_keyZ fhr_fileNZinput_fhr_listZ"model_copy_output_DATA2COMOUT_listZfhr_fileN_DATAZfhr_fileN_COMOUTZmodel_files_existrrr check_model_filess                                                                                                           !    rAc Cstj|d|dd}tj|d|dd|d}g}g}|dd kr|dd kr|d d krRtj|d |d|dd|dd}||nu|d dkrtj|d dd|dd|tjdddd|dd}||tj|d|dd|dd|dd|d d|dd|tjdddd|dd}tj|d|dd|dd|dd|d d|dd|tjdddd|dd}|||fn|dd kr|d d!vrd"|dvr|dd"d# } tj|d d$| d%| d|d} || tj|d|dd|dd&|dd'|d d|d&d|dd} tj|d|dd|dd&|dd'|d d|d&d|dd} || | fn=|dd(krW|dd kr|d d)kr|dd*krd+} d,}|d+krtj|d d-d.|tj|d,d/dd}|||d,7}|d+kstj|d|dd|dd-|dd0|dd}tj|d|dd|dd-|dd0|dd}|||fn|dd krU|d d!vrUtj|d|dd|dd&|dd'|d d|d&d|dd}||np|dd1kr|dd kr|d d krtj|d |d|dd|dd}||n>|d d)krtj|d|dd|dd-|dd2|d d3|dd}||n |d d4krtj|d d-d5|d}||n|d dkrFd6|dvrtj|d dd|dd|tjdddd|dd}n9d7|dvr@tj|d|dd|dd|dd8|d d|tjdddd|dd}||n|d d9kratj|d d:d;|d}||nf|d d|tjdddd|dd}||n<|dd kr|d d!vrtj|d|dd|dd&|dd'|d d|d&d|dd}||g}|}|D]]}tj |d,r$|d?t |d,|d@|dd kr|dd kr|d d!vrd"|dvrtj |d@rt dAdB|d@gt dCdD|d@g||q|dEqtdFdG|DrBt|d@krBdE}||fSg}|D]}tj |rV|d?qF|dEqFtdHdG|Drtt|d@krtd?}||fSdE}||fS)Ia! Args: job_dict - dictionary containing settings job is running with (strings) Returns: all_truth_file_exist - if all needed truth files exist or not (boolean) truth_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r r r;r r*r)r&rrrrrrrrz.truthr osi_safzosi_saf.multi.r1r~rtoz.ncrr(%Y%m%dr%r_validrr#)rr-r$ZPrepbufrrZ prepbufr_z prepbufr.r3Zpb2nc_r'r(24hrCCPAr,Zccpazccpa.6H.rZ+pcp_combine_precip_accum24hr_24hrCCPA_validr/r+Z_24hrCCPA_validr)zccpa.3H.ZDailyAvg_Concentrationr0Z0regrid_data_plane_sea_ice_DailyAvg_Concentrationr,Znohrscz nohrsc.24H.r! ghrsst_ospoz ghrsst_ospo.TrrDrrrFcsr4r5rr6rrr r ;r8z$check_truth_files..csr4r5rr6rrr r Er8)rrr r!rrrrrlowerrCr"rTrr<r:r)r=r>r?Ztruth_input_file_listZtruth_output_file_listZmodel_truth_fileZ osi_saf_fileZosi_saf_DATA_outputZosi_saf_COMOUT_outputZ prepbufr_nameZ prepbufr_fileZpb2nc_DATA_outputZpb2nc_COMOUT_outputZ nccpa_filesrZ nccpa_fileZccpa_DATA_outputZccpa_COMOUT_outputZ pb2nc_fileZ ccpa_fileZ nohrsc_fileZghrsst_ospo_fileZtruth_output_files_exist_listZ"truth_copy_output_DATA2COMOUT_listZtruth_file_tupleZall_truth_file_existZtruth_input_files_exist_listZ truth_filerrr check_truth_filesds                                       rJc Csrtj|d|dd|dd|dd|d|d |d}ttj|d }t|d kr5d }|Sd }|S)a! Check for MET .stat files Args: job_dict - dictionary containing settings job is running with (strings) Returns: stat_files_exist - if .stat files exist or not (boolean) r r*r)r&rr(rr rz*.statrTF)r r!rglobr)r=Zmodel_stat_file_dirZstat_file_listZstat_files_existrrr check_stat_filesLs   rLc Csddddddddddddddddddddddddd}|t|vr<||d}||d }||d }n td |d td |||fS)al! This returns the valid hour start, end, and increment information for a given observation Args: obs - observation name (string) Returns: valid_hr_start - starting valid hour (integer) valid_hr_end - ending valid hour (integer) valid_hr_inc - valid hour increment (integer) rr~)r  valid_hr_end valid_hr_incrr)rF3hrCCPA 24hrNOHRSCOSI-SAF GHRSST-OSPOZGET_Dr rMrNzFATAL ERROR: Cannot get z valid hour informationr,)rrrrr)rMZobs_valid_hr_dictr rMrNrrr get_obs_valid_hrscs@     rTc Csd}tjt|d}tjjtjt|d}td|d|td|t|d|dkrit d| d d d |d |d |d d|d|d d|d|d d|d} n2|dvrt d| d d|d d|d|d d|d |d|d|d d} d\} } | | |krt | tdt | | d dt t|d t j| d!d"d#} | d$d%krd'S| d&7} | | |ksd'Sd'S)(a! This submits a job to the transfer queue to get data that does not reside on current machine Args: job_file - path to job submission file (string) job_name - job submission name (string) job_output - path to write job output (string) machine - machine name (string) user - user name (string) queue - submission queue name (string) account - submission account name (string) Returns: r)Zminutesz Submitting rOzOutput sent to rZWCOSS2zqsub -V -l walltime=z%H:%M:%Srz-q z -A z -o z-e z -N z-l select=1:ncpus=1 zqselect -s QR -u z-N z | wc -l)ZHERAZORIONZS4ZJETzsbatch --ntasks=1 --time=z --partition=z --account=z --output=z --job-name=z squeue -u z -n z-t R,PD -h | wc -l)r,rzWalltime checker: zout of z secondsTUTF-8)rrrrr,N)rrr total_secondsrrrr rDrZrrrrZ check_output) Zjob_filerZ job_outputmachineuserZqueueZaccountZwalltimeZwalltime_secondsZ job_check_cmdZ sleep_counterZ sleep_checkerZ check_jobrrr get_off_machine_datas     rYcCsgd}|dvr0tjtjdtjddtjdddtjd <ttjd |gd n|d vrE|d d g|dkrE|dgi}|D] }tj|||<qI|d vr[d|d<||d<||d<||d<|dvr|dttjvrtj|dd}n+t t tj|dt tj|dt tj|dt tj|d}dd|D}|dvrd||d<n d d|d |d<|d!vr(tj|d"d} tjdd#kr|d$krd%|vs|d&vr| D]} t | d'd(kr| | q| d( d)|d*<| d+ d)|d,<t | d-krtttj| t d.} nd/} t| |d0<n^|d1kr5td2\} } }n9|d3krBtd4\} } }n,|d5krOtd6\} } }n|d7kr\td8\} } }n|d9kritd:\} } }nd;\} } }t|  d)|d*<t|  d)|d,<t||d0<tj|d<d}|d( d)|d=<|d+ d)|d><t |d-krtttj|t d.}nd/}t||d?<|S)@ad! This initializes a dictionary of environment variables and their values to be set for the job pulling from environment variables already set previously Args: verif_type - string of the use case name group - string of the group name verif_case_step_abbrev_type - string of reference name in config and environment variables job - string of job name Returns: job_env_dict - dictionary of job settings )rWZevs_verZHOMEevsZFIXevsr0r COMROOTNETr(r*r&r'ZCOMINSENDCOMrr )rr'r/Z gather_statsr r*r)r&rtmp MET_TMP_DIR)r%MET_ROOTr^rZ)condense_stats filter_stats make_plots tar_imagesr_Zmet_verrcZKEEPDATADEBUGZplot_verbosityrrr)rr'r/rarbZ _fhr_listrZ_fhr_minZ_fhr_maxZ_fhr_inccSrr)rrrrr rwrz*initalize_job_env_dict..)rarbrrr )rZmeansr-r$Z_valid_hr_listr#r-CAPE)Z PBLHeightr2rrrrr r{rMr,r|r~rNr(rFr)rPr,rQr rRr!rS)rrZ _init_hr_list init_hr_start init_hr_end init_hr_inc)r r!rr.r$extendrrrrangerr<rtrrrrrrrT)Z verif_typegroupZverif_case_step_abbrev_typeZjobZjob_env_var_listZ job_env_dictZenv_varrZ fhr_rangeZverif_type_valid_hr_listZvhZverif_type_valid_hr_incr rMrNZverif_type_init_hr_listZverif_type_init_hr_incrrr initalize_job_env_dicts                   rmcCsbtdd}t|}|dtj|dd}||||d|}t||||S)z! Get logger Args: log_file - full path to log file (string) Returns: logger - logger object zL%(asctime)s.%(msecs)03d (%(filename)s:%(lineno)d) %(levelname)s: %(message)sz%m/%d %H:%M:%SrdrL)rz Log file: ) loggingZ FormatterZ getLoggerZsetLevelZ FileHandlerZ setFormatterZ addHandlerrinfo)Zlog_fileZ log_formatterloggerZ file_handlerZ logger_inforrr get_loggerA s      rqc Cs|dkrtj||d} tj||d} tjt|d} n|dkr=tj||d} tj||d} tjt| d} t| | | | tj}|dkr[|}|tjt| d}n|dkrk|}|tjt| d}g}ddtt|t|t|t|D}tt|D]}|| d|vr| |qt ||}t ||}g}ddtt|t|t| t| D}tt|D]}|| d|vr| |qt ||}t ||}||fS) a! This builds the dates to include in plotting based on user configurations Args: logger - logger object date_type - type of date to plot (string: VALID or INIT) start_date - plotting start date (string, format: YYYYmmdd) end_date - plotting end date (string, format: YYYYmmdd) valid_hr_start - starting valid hour (string) valid_hr_end - ending valid hour (string) valid_hr_inc - valid hour increment (string) init_hr_start - starting initialization hour (string) init_hr_end - ending initialization hour (string) init_hr_inc - initialization hour incrrement (string) forecast_hour - forecast hour (string) Returns: valid_dates - array of valid dates (datetime) init_dates - array of initalization dates (datetime) ryr;rrzcSg|] }t|dqSrqrrtrurrr rw z"get_plot_dates..rcSrrrqrsrurrr rw rt) rrrrrZarangeastyperkrrrdelete)rprZ start_dateend_dater rMrNrgrhrir start_date_dt end_date_dtZdt_incZdate_type_dates valid_datesZ init_datesZvalid_remove_idx_listrdZinit_remove_idx_listrrrr get_plot_datesW sx            r|c Cs|ddkr|dd}n |ddkr|}tj|dddd|d }tj|rSt|}|D]}||vrB|d d }nq3Wd n1sMwYn||d dt d| ddd} | S)a! Get the MET columns for a specific line type and MET verison Args: logger - logger object met_root - path to MET (string) met_version - MET version number (string) met_line_type - MET line type (string) Returns: met_version_line_type_col_list - list of MET versoin line type colums (strings) rrrrr,ZshareZmetZ table_filesZmet_header_columns_Vz.txtz : r{Nz DOES NOT EXISTS, z*cannot determine MET data column structure rr) rrr r!rr"rArerrorrrr) rpZmet_rootZ met_versionZ met_line_typeZmet_minor_versionZmet_minor_version_col_filer<lineZline_type_colsmet_version_line_type_col_listrrr get_met_line_type_cols s2      rcCsp|ddddddddd d d d }|ddddddddd d d d }||fS) z! Format threshold with letter and symbol options Args: thresh - the threshold (string) Return: thresh_symbol - threshold with symbols (string) thresh_letters - treshold with letters (string) gez>=gtreqz==nez!=lez<=lt<)r)ZthreshZ thresh_symbolZ thresh_letterrrr format_thresh s rc Csiddddddddddd d d d d d ddddddddddddddddddid d!d"d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;d;dd?d@dAdBdCdDdEdFdG}|dH}|dI}|dJ}dK|dLdM}|dN} |dO} |dPkr|dQdRvr|dS} n|dTdUdVdWdX} n|dYdUdVdWdX} |dOdZkr| d[d\d]d^} ||d_} |d`vr tj||dX|da|dbdU|dc|dX||| | dX| | } n2|dPkr>|dd}tj||dX|da|dbdU|dc|dX||| | dX| | | } | tj||dX|da|dbdU|dc|}| |fS)ea"! Get directories for the plotting job Args: DATA_base_dir - path to DATA directory (string) COMOUT_base_dir - path to COMOUT directory (string) job_group - plotting job group: condense_stats, filter_stats, make_plots (string) plot_job_env_dict - dictionary with plotting job environment variables to be set Returns: DATAjob_dir - path to plotting job's DATA directory COMOUTjob_dir - path to plotting job's COMOUT directory ZAlaskaZalaskaZ AppalachiaZbuk_aplZ ANTARCTICZ antarcticZARCTICZarcticZATL_MDRZal_mdrZconusZCONUSZ buk_conusZ CONUS_EastZ buk_conus_eZ CONUS_CentralZ buk_conus_cZ CONUS_SouthZ buk_conus_sZ CONUS_WestZ buk_conus_wZCPlainsZbuk_cplZ DeepSouthZbuk_dsZEPAC_MDRZep_mdrZGLOBALZglbZ GreatBasinZbuk_grbZ GreatLakesZbuk_grlkZhawaiiZ MezqutialZbuk_mezZ MidAtlanticZbuk_matlZN60N90Zn60ZNAOZnaoZNHEMZnhemZ NorthAtlanticZbuk_neZNPlainsZbuk_nplZNPOZnpoZNRockiesZbuk_nrkZ PacificNWZbuk_npwZ PacificSWZbuk_pswZPrairieZbuk_praZpricoZS60S90Zs60ZSAOZsaoZshemZbuk_seZbuk_swZbuk_splZspoZbuk_srkZtropics)ZSHEMZ SoutheastZ SouthwestZSPlainsZSPOZSRockiesZTROPICSr&r*rZlastZNDAYSdays line_type fcst_var_namerbZplot)Z stat_by_levelZ lead_by_levelZ vert_profileZfcst_var_level_listrprr)fcst_var_levelreZz0Zl0Zp90_0Zl90vx_mask)r`raZ plot_outputr(rwstat)rIrr r!r)Z DATA_base_dirZCOMOUT_base_dirZ job_groupZplot_job_env_dictZregion_savefig_dictZdir_stepZdir_verif_caseZdir_verif_typeZ dir_ndaysZ dir_line_typeZ dir_parameterZ dir_levelZ dir_regionZ DATAjob_dirZdir_statZ COMOUTjob_dirrrr get_plot_job_dirs s      !"# +                 rc Cs|}||krctj||d|dd|d|dd|dd}tj||d|d|dd}tj|sUt|rUtd |d |t|||tj d d }||ksd Sd S)ah! Link model daily stat files Args: model_name - name of model (string) source_stats_base_dir - full path to stats/global_det source directory (string) dest_model_name_stats_dir - full path to model destintion directory (string) verif_case - grid2grid or grid2obs (string) start_date_dt - month start date (datetime obj) end_date_dt - month end date (datetime obj) Returns: rrDz evs.stats.z.atmos.v.statZ_atmos_Z_vrrOr,)rN) r r!rrr"r5rrrr) Z model_nameZsource_stats_base_dirZdest_model_name_stats_dirZ verif_caserxryrZsource_model_date_stat_fileZdest_model_date_stat_filerrr get_daily_stat_fileN s>  rc Cs tj|||d} tj| dd} tj|d|d| d|d|ddddd|d } t| d krP|d | d Stj| s| d | t | d  }| }Wd n1sswYd}d|dd|dd|dd|dd|dd| d}| D]/}| d|t jd|d||dt jt jdd}| d|j||d }q| d|d| t | d}|||Wd d S1swYd S| | dd S)a! Condense the individual date model stat file and thin out unneeded data Args: logger - logger object input_dir - path to input directory (string) output_dir - path to output directory (string) model - model name (string) obs - observation name (string) vx_mask - verification masking region (string) fcst_var_name - forecast variable name (string) fcst_var_level - forecast variable level (string) obs_var_name - observation variable name (string) obs_var_leve - observation variable level (string) line_type - MET line type (string) Returns: z_*.statT) recursivecondensed_stats_r)rrrrrzNO STAT FILES IN MATCHING z$Condensing down stat files matching Nrrz "zGetting data from z grep -R "z " rU)rrstderrrzRan z Condensed z .stat file at r6z exists)r r!rrKrIrrwarningr"rorAreadlinedebugrPopenPIPEZSTDOUTrZ communicaterB)rp input_dir output_dirrGrMrrr obs_var_name obs_var_levelrZmodel_stat_files_wildcardZmodel_stat_files output_fileZmsfZmet_header_colsZall_grep_outputZ grep_optsZmodel_stat_fileZpsr<rrr condense_model_stat_filesr s       "rcE Cs t||d|d| }t|D]}|d||dd||d}tjj|g|gddgd}||}tj|d ||d d | d | d | d d  d d d | d}t |dkrd|dd |||d d|dd || | d d| d d| d d| d d||d | |d dd|d dd d| d d d  d d  dd dd d d! d"d!d}tj||}tj||}tj|r|}n|}tj|s d#}d#}n d$}d#}nd$}d$}tj|r| d%krtj|d&d'd#d$d(d)d*} t | dkrt| d+d}!g}"|D]Y}#|#d,krN|"d-q@|#d.krd'}$d'}%d'}&|&|!|!kr|"d/t|$d d0t|%|%|!kr{|%d'7}%n |%|!krd'}%|$d'7}$|&d'7}&|&|!|!ks`q@|"|#q@|"}|r|d1krt|\}'}(n|}'|})| d1krt| \}*}+n| }*| },tj|ri|d2|tj|d&d'd#|d$d(d)d3}-|-|-d4|dk|-d5| k@|-d6| d7d8k@|-d9|k@|-d:|k@|-d;|k@|-d<| k@|-d=|dk@|-d>| k@|-d?|k@|-d@|k@|-dA|'k@|-dB|*k@|-dC| k@}.|.|.dD|}.tj|.dDdEdF|.dD<|.jdDdG}.|.dDj dE|.dD<|.j||d)d&dHdIn||dJtj|r|dK|ddLdM|n|dN|tjtj||dO}/|rtj|r|dP|dQ|dtj|d&d'd#|d1gd)dR}0i}1|dS}2|D]}#||#}3|3|2krt|1|#<qtj|1|#<q|0|1}0|D]'}4|0j|0dD|4k }5t |5dkrq|0j!|5dd)d)|/j!||4f<q|dTvr>|dUvr>| dVvr>t"dWdX}6|dYkr-d}7n | dZkr5d}7nd[}7d#}8d\}9d/}:n|d]vrS| d^vrSd_}6d}7d#}8d`}9da}:nd$}8|8r| dbkr||/j!|/dc|9kddf};|/j!|/dc|9kdef}n| dZkr|/j!|/dc|9kdgf};|/j!|/dc|9kdhf}ns| djkr|/j!|/dc|9kdkf}?|/j!|/dc|9kdlf}@|/j!|/dc|9kdmf}A|/j!|/dc|9kdnf}Bgdo}=gdp}>n9| dqkr|/j!|/dc|9kdrf}?|/j!|/dc|9kdsf}@|/j!|/dc|9kdtf}A|/j!|/dc|9kduf}Bgdv}=gdw}>|=D]}#|6|/j!|/dc|9k|#f|7|/j!|/dc|9k|#f<q|>D]}#|#dxvrG|6|7|;|6|7|<}CnM|#dyvrUd7|6|7|;}Cn?|#dzvrcd7|6|7|<}Cn1|#d{vru|6|7|?|@|A|B}Cn|#d|vrd7|6|7|?|@}Cn|#d}vrd7|6|7|A|B}C|6d7|/j!|/dc|9k|#f|C|7d7|/j!|/dc|9k|#f<q3|:|/j!|/dc|9kdcf<n||dJ|d~kr|/}Dqt#|D|/g}Dq|DS)a! Build the data frame for all model stats, Read the model parse file, if doesn't exist parse the model file for need information, and write file Args: logger - logger object input_dir - path to input directory (string) output_dir - path to output directory (string) model_info_dict - model infomation dictionary (strings) met_info_dict - MET information dictionary (strings) fcst_var_name - forecast variable name (string) fcst_var_level - forecast variable level (string) fcst_var_tresh - forecast variable treshold (string) obs_var_name - observation variable name (string) obs_var_level - observation variable level (string) obs_var_tresh - observation variable treshold (string) line_type - MET line type (string) grid - verification grid (string) vx_mask - verification masking region (string) interp_method - interpolation method (string) interp_points - interpolation points (string) date_type - type of date (string, VALID or INIT) dates - array of dates (datetime) met_format_valid_dates - list of valid dates formatted like they are in MET stat files fhr - forecast hour (string) Returns: all_model_df - dataframe of all the information rootversionrnameZ plot_namerGrz)namesrr)rrrrrZfcstrMZobs_nameZlinetypegridZvxmaskZinterpz %Y%m%d%H%M%SrCr{rIrz&&andz||orz0,*,*rz*,*TFMCTCrr,rN)sepskiprowsskipinitialspacekeep_default_nar}headerz(N_CAT)ZN_CATzF[0-9]*_O[0-9]*FOZNAz Parsing file )rrrrrr}rrZDESCZ FCST_LEADrrZ0000ZFCST_VARZFCST_LEVZOBS_VARZOBS_LEVZOBTYPEZVX_MASKZ INTERP_MTHDZ INTERP_PNTSZ FCST_THRESHZ OBS_THRESHZ LINE_TYPEZFCST_VALID_BEGz %Y%m%d_%H%M%Sr)Zbyr6)rr9rrr2zParsed z file zat zCould not create )r9columnszReading z for )rrrrZ na_valuesrTOTAL)ZTMPZDPTTMP_ANOM_DAILYAVGZ SST_DAILYAVGZTSOIL)ZZ0ZZ2zZ0.1-0)SL1L2SAL1L2g"@g@rrgQ|K)ZUGRDZVGRDZ UGRD_VGRDZWNDSHRZGUST)rrVL1L2VAL1L2g,?zm/sZktrZ FCST_UNITSFBAROBAR)FOBARFFBAROOBARFABAROABAR)FOABARFFABAROOABARrUFBARVFBARUOBARVOBAR)rrrr)UVFOBARUVFFBARUVOOBARrUFABARVFABARUOABARVOABAR)rrrr)UVFOABARUVFFABARUVOOABAR)rr)rr)rr)rr)rr)rrZmodel1)$rrrpdZ MultiIndexZ from_productr r!rrIrrrrtr"Zread_csvrrrrrZisinZ to_datetimeZ sort_valuesZdtZto_csvZ DataFramernanr9Zfloat64rutolistlocZdivideconcat)ErprrZmodel_info_dictZ met_info_dictrrZfcst_var_threshrrZobs_var_threshrrrZ interp_methodZ interp_pointsrZdatesZmet_format_valid_datesrIrZ model_numZmodel_num_nameZmodel_num_df_index model_dictZcondensed_model_fileZparsed_model_stat_file_nameZinput_parsed_model_stat_fileZoutput_parsed_model_stat_fileZparsed_model_stat_fileZwrite_parse_stat_fileZread_parse_stat_fileZtmp_dfZncatZ"new_met_version_line_type_col_listZcolZfcountZocountZtotcountZfcst_var_thresh_symbolZfcst_var_thresh_letterZfcst_vat_thresh_letterZobs_var_thresh_symbolZobs_var_thresh_letterZobs_vat_thresh_letterZcondensed_model_dfZparsed_model_dfZ model_num_dfZmodel_stat_file_dfZ df_dtype_dictZ float_idxZcol_idxrZ&model_stat_file_df_valid_date_idx_listZcoefZconstZconvertZ units_oldZ units_newZ fcst_avg_oldZ obs_avg_oldZ col1_listZ col2_listZ uf_avg_oldZ vf_avg_oldZ uo_avg_oldZ vo_avg_oldZconst2Z all_model_dfrrr build_df s #                                                        rcCs|dkr3|jddd}|jddd}|jddd}|jddd}|jddd} n|dkrf|jddd } |jddd } |jddd } |jddd } |jddd } n|dkr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd }|jddd!} |jddd"}!|jddd#}"|jddd$}#|jddd%}$|jddd&}%|jddd'}&|jddd(}'|jddd)}(|jddd*})|jddd+}*|jddd,}+|jddd-},|jddd.}-|jddd/}.|jddd0}/|jddd1}0|jddd2}1|jddd3}2|jddd4}3|jddd5}4|jddd6}5|jddd7}6|jddd8}7|jddd9}8|jddd:}9|jddd;}:|jddd<};|jddd=}<|jddd>}=|jddd?}>|jddd@}?|jdddA}@|jdddB}A|jdddC}B|jdddD}C|jdddE}D|jdddF}E|jdddG}F|jdddH}G|jdddI}H|jdddJ}I|jdddK}J|jdddL}K|jdddM}L|jdddN}M|jdddO}N|jdddP}O|jdddQ}P|jdddR}Q|jdddS}R|jdddT}S|jdddU}T|jdddV}U|jdddW}V|jdddX}W|jdddY}X|jdddZ}Y|jddd[}Z|jddd\}[|jddd]}\|jddd^}]|jddd_}^|jddd`}_|jddda}`|jdddb}a|jdddc}b|jdddd}c|jddde}d|jdddf}e|jdddg}f|jdddh}g|jdddi}h|jdddj}i|jdddk}j|jdddl}k|jdddm}l|jdddn}mnl|dokr7|jdddp}n|jdddq}o|jdddr}p|jddds}q|jdddt}r|jdddu}s|jdddv}t|jdddw}u|jdddx}vn|dykrY|jdddz}w|jddd{}x|jddd|}yn|d}vr|jddd~}z|jddd}{|jddd}||jddd}}|dkr|jddd}~n|dvr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|dkr|jddd}~nh|dkr|jddd}ېnX|dkr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jdddz}w|jddd}|jddd}|jddd|}y|jddd}|jddd}n|dkr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}ni|dk rA|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}n |dk rK|jddd}|jddd}|jddd}|jddd }|jddd }|jddd }|jddd}|jddd}|jddd }|jddd }|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd} |jddd} |jdddY}X|jdddZ}Y|jddd[}Z|jddd\}[|jdddi}h|jdddj}i|jdddk}j|dk r|dk rm| | | t| | | | | | } n |dv ry| } n|dk r|t||} n|dv r|dk r||} n|dk r|*} n|dk rt|t|} n|dk r|dk r|||} |||} |||t| | } n|dk r|dk r|z|z|{||} n|dk r|dk r|ې} nr|dv rB|dk r8|z|{|||}}|z|{|z|||}|z||z|{|||} nA|dk r@|} n7|dk rQ|dk rO|} n(|dk rp|dk rf|z|{|z||} n|dk rn|} n |dk r~|dk r}|ߐ} n|d~k r|dk r|z} n|dk r|dk r|z|{|||}}|z|{|z||}|||}|{|}}|||}|z|}|||} n|dk r|dk r|} n|dk r|dk r|z|z||} n|dAk r|dk rt||d|} nv|dk r |@} nm|dk rt||d|} nZ|dtk r-|dok r,|r} nL|dk rC|dk rBd|{|{|z} n6|dk rk|dk rjt||||||d|d||} n||d tdd!}i}|| jjk rt| j | |d"t |<|d7}|| jjk s| jjdk rɐ| j |d#}| |fS| jjdk r| j |d#|d$}| |fS(%a! Calculate the statistic from the data from the read in MET .stat file(s) Args: data_df - dataframe containing the model(s) information from the MET .stat files line_type - MET line type (string) stat - statistic to calculate (string) Returns: stat_df - dataframe of the statistic stat_array - array of the statistic rNrrrrrrrrrrrCNTFBAR_NCLFBAR_NCUFBAR_BCLFBAR_BCUFSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCUOBAR_NCLOBAR_NCUOBAR_BCLOBAR_BCUOSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCUPR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCUSP_CORRKT_CORRRANKS FRANKS_TIES ORANKS_TIESMEME_NCLME_NCUME_BCLME_BCUESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCUMBIAS MBIAS_BCL MBIAS_BCUMAEMAE_BCLMAE_BCUMSEMSE_BCLMSE_BCUBCRMSE BCRMSE_BCL BCRMSE_BCURMSERMSE_BCLRMSE_BCUE10E10_BCLE10_BCUE25E25_BCLE25_BCUE50E50_BCLE50_BCUE75E75_BCLE75_BCUE90E90_BCLE90_BCUIQRIQR_BCLIQR_BCUMADMAD_BCLMAD_BCU ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCUME2ME2_BCLME2_BCUMSESS MSESS_BCL MSESS_BCURMSFA RMSFA_BCL RMSFA_BCURMSOA RMSOA_BCL RMSOA_BCUANOM_CORR_UNCNTRANOM_CORR_UNCNTR_BCLANOM_CORR_UNCNTR_BCUSISI_BCLSI_BCUZGRADFGBAROGBARMGBAREGBARS1S1_OG FGOG_RATIODXDYZFHOF_RATEH_RATEO_RATE)CTCZNBRCTCFY_OYFY_ONFN_OYFN_ONrGEC_VALUE)CTSZNBRCTSBASER BASER_NCL BASER_NCU BASER_BCL BASER_BCUFMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCUACCACC_NCLACC_NCUACC_BCLACC_BCUFBIAS FBIAS_BCL FBIAS_BCUPODYPODY_NCLPODY_NCUPODY_BCLPODY_BCUPODNPODN_NCLPODN_NCUPODN_BCLPODN_BCUPOFDPOFD_NCLPOFD_NCUPOFD_BCLPOFD_BCUFARFAR_NCLFAR_NCUFAR_BCLFAR_BCUCSICSI_NCLCSI_NCUCSI_BCLCSI_BCUGSSGSS_BCLGSS_BCUHKHK_NCLHK_NCUHK_BCLHK_BCUHSSHSS_BCLHSS_BCUODDSODDS_NCLODDS_NCUODDS_BCLODDS_BCULODDS LODDS_NCL LODDS_NCU LODDS_BCL LODDS_BCUORSSORSS_NCLORSS_NCUORSS_BCLORSS_BCUEDSEDS_NCLEDS_NCUEDS_BCLEDS_BCUSEDSSEDS_NCLSEDS_NCUSEDS_BCLSEDS_BCUEDIEDI_NCLEDI_NCUEDI_BCLEDI_BCUSEDISEDI_NCLSEDI_NCUSEDI_BCLSEDI_BCUBAGSS BAGSS_BCL BAGSS_BCUrMrF1_O1ZNBRCNTFBSFBS_BCLFBS_BCUFSSFSS_BCLFSS_BCUAFSSAFSS_BCLAFSS_BCUUFSSUFSS_BCLUFSS_BCU F_RATE_BCL F_RATE_BCU O_RATE_BCL O_RATE_BCUrrrrrrrrrrrrrrrr FA_SPEED_BAR OA_SPEED_BARVCNTFS_RMSOS_RMSMSVERMSVEFDIRZODIR FBAR_SPEED OBAR_SPEED VDIFF_SPEED VDIFF_DIR SPEED_ERR SPEED_ABSERRDIR_ERR DIR_ABSERR ANOM_CORR)rr)ZBIASrZCORR)ZETSryZPODrrZSRATIOr,Z STDEV_ERRz IS NOT AN OPTIONrr9Zindex0Zindex1)rrsqrtr~rrr9ZnlevelsrZget_level_valuesr;rvaluesZreshape(rpZdata_dfrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrHrIrJrKrLrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrZORDIRrrrrrrrrrZstat_dfZvar_fZvar_orCZCAZCBidxZidx_dictZ stat_arrayrrr calculate_stat s:                                 "                          .    rc Cstj}|dkrtj|}|S|dkrV|jsTt|j dddftj tj|j dddf}t ||j ddddf dg|||\}}|d}|S| |dd|S) ag! Calculate average of dataset Args: logger - logger object average_method - method to use to calculate the average (string: mean, aggregation) line_type - line type to calculate stat from stat - statistic to calculate (string) df - dataframe of values Returns: meanZ aggregationNrsumrz not recongnized...z(use mean, or aggregation...returning NaN)rrZmaZmasked_invalidrZisnullrr:rrZ count_maskedrZaggr) rpZaverage_methodrrZdfZ average_valueZndaysZavg_dfZ avg_arrayrrr calculate_average s( " r)A__doc__r rZnumpyrrrPrZnetCDF4rrKZpandasrrnrQrrrr$r/r1r5rKrNrTr^rarbrkrnrorrrrrrrrrrrrrrrrrr rArJrLrTrYrmrqr|rrrrrrrrrrrr s   E !E@m*0"!b(i)2I$t$?Z