o ng@sdZddlZddlZddlZddlZddlZddlZddlZ ddlZddl Z ddl Z ddl Z ddlZddlmZddZddZdd Zd d Zd d ZddZddZddZddZddZddZddZddZddZd d!Zd"d#Z d$d%Z!d&d'Z"d(d)Z#d*d+Z$d,d-Z%d.d/Z&d0d1Z'd2d3Z(d4d5Z)d6d7Z*d8d9Z+d:d;Z,dd?Z.d@dAZ/dBdCZ0dDdEZ1dFdGZ2dHdIZ3dJdKZ4dLdMZ5dNdOZ6dPdQZ7dRdSZ8dTdUZ9dVdWZ:dXdYZ;dZd[Zd`daZ?dbdcZ@dddeZAdS)fzw Name: subseasonal_util.py Contact(s): Shannon Shields Abstract: This contains many functions used across subseasonal. N)sleepcstddtfdddDrtjddd}nt}|jdkrBtd d|jd t|jt |jd Sd S) zu! Run shell command Args: command - list of argument entries (string) Returns: z Running  c3s|] }|dvVqdS)rN)join).0ZmarkcommandI/lfs/h1/ops/prod/packages/evs.v1.0.19/ush/subseasonal/subseasonal_util.py sz$run_shell_command..)"'|*>T)shellr FATAL ERROR: z gave return code N) printrany subprocessrun returncodeargsstrsysexit)rZ run_commandrrr run_shell_commands    rcCstjtjddd}tjtjddd}tjtjddtjdtjdtjd |}tj|s@td |d td |d |d |}|S)z! Write out full call to METplus Args: conf_file_name - METplus conf file name (string) Returns: metplus_cmd - full call to METplus (string) METPLUS_PATHZushzrun_metplus.pyZPARMevsZmetplus_configz machine.confSTEP COMPONENT VERIF_CASEr DOES NOT EXISTz -c ospathrenvironexistsrrr)Zconf_file_nameZ run_metplusZ machine_confZ conf_fileZ metplus_cmdrrr metplus_command+s"   r'cCsdtjtjdtjd|}tj|s!td|dtdd|}|D]}|d|}q'|S)a! Write out full call to python Args: python_script_name - python script name (string) script_arg_list - list of script agruments (strings) Returns: python_cmd - full call to python (string) USHevsrrr r!zpython rr")Zpython_script_nameZscript_arg_listZ python_scriptZ python_cmdZ script_argrrr python_commandDs   r)cCsvd|vsd|vr d}nd}tj|r-tj|dkrd}|St|d|dd }|St|d|d d }|S) aK! Checks to see if file exists and has size greater than 0 Args: file_name - file path (string) Returns: file_good - boolean - True: file exists,file size >0 - False: file doesn't exist OR file size = 0 z/com/z/dcom/ZWARNINGZNOTErT: z empty, 0 sizedF does not exist)r#r$r&getsizer) file_nameZ alert_wordZ file_goodrrr check_file_exists_sizeYs  r.c Cstj|sbt|dG}|d|d|d|dd|d|dd |d d |d d |d|d|d|dWdn1sUwYt|ddSdS)ab! This writes a missing model file to a log Args: log_missing_file - log of missing file (string) missing_file - missing file path (string) model - model name (string) init_dt - initialization date (datetime) fhr - forecast hour (string) w #!/bin/bash zexport subject="Frz Forecast z"Data Missing for EVS subseasonal" echo "Warning: No z forecast was zavailable for %Y%m%d%Hfz" z > mailmsg echo "Missing file is " >> mailmsg !echo "Job ID: $jobid" >> mailmsg )cat mailmsg | mail -s "$subject" $MAILTO Nr#r$r&openwriteupperchmod)log_missing_file missing_filemodelinit_dtfhrlmfrrr log_missing_file_modelts"     rDcCstj|st|dk}|d|dkr9|d|ddd|d|d d d |d d n#|d|ddd|d|dd d |d d |d|d|d|dWdn1sywYt|ddSdS)a/! This writes a missing obs file to a log Args: log_missing_file - log of missing file (string) missing_file - missing file path (string) obs - observation name (string) valid_dt - initialization date (datetime) r/r0Znamzexport subject="z prepbufr Data z Missing for zEVS subseasonal" r1z prepbufr data zwas available for z valid date r2z " > mailmsg z Analysis Data z Analysis data r4r5r6r7Nr8r9)r>r?obsZvalid_dtrCrrr log_missing_file_obss<      rFcCs0t|rtd|d|t||dSdS)z! This copies a file from one location to another Args: source_file - source file path (string) dest_file - destination file path (string) Returns: zCopying  to N)r.rshutilZcopy2) source_file dest_filerrr copy_files rKcCDtd|dd|tjd}t|d|d|ddS)aW! Converts GRIB1 data to GRIB2 Args: grib1_file - string of the path to the GRIB1 file to convert (string) grib2_file - string of the path to save the converted GRIB2 file (string) Returns: zConverting GRIB1 file rto GRIB2 file CNVGRIBz -g12  > /dev/null 2>&1Nrr#r%system) grib1_file grib2_filecnvgribrrr convert_grib1_grib2   rUcCrL)aE! Converts GRIB2 data to GRIB1 Args: grib2_file - string of the path to the GRIB2 file to convert grib1_file - string of the path to save the converted GRIB1 file Returns: Converting GRIB2 file rzto GRIB1 file rNz -g21 rONrP)rSrRrTrrr convert_grib2_grib1rVrXcCrL)aK! Converts GRIB2 data to GRIB2 Args: grib2_fileA - string of the path to the GRIB2 file to convert grib2_fileB - string of the path to save the converted GRIB2 file Returns: rWrrMrNz -g22 rONrP)Z grib2_fileAZ grib2_fileBrTrrr convert_grib2_grib2rVrYcCsxdd|D}dd|D}|dkr|}n|dkr|}|d} |d} t|dkr7tttj|td } nd } tj|| d } tj|| d } g}| }|| kr|dkrZ|}n|dkr`|}|D]I}|d krkd}nt|}|dkr||tj|d }n |dkr|tj|d }| d|vr| d|vri}||d<||d<t ||d<| |qb|tjt| d }|| ksS|S)a! Creates a list of dictionaries containing information on the valid dates and times, the initialization dates and times, and forecast hour pairings Args: date_start - verification start date (string, format:YYYYmmdd) date_end - verification end_date (string, format:YYYYmmdd) date_type - how to treat date_start and date_end (string, values:VALID or INIT) init_hr_list - list of initialization hours (string) valid_hr_list - list of valid hours (string) fhr_list - list of forecasts hours (string) Returns: time_info - list of dictionaries with the valid, initalization, and forecast hour pairings cSg|]}|dqSzfillrZhrrrr z!get_time_info..cSrZr[r]r_rrr r`raVALIDINITrr!dtyper2anlhours%H valid_time init_time forecast_hour) lennpmindiffarrayintdatetimestrptime timedeltastrftimerappend)Z date_startZdate_end date_type init_hr_list valid_hr_listfhr_listZvalid_hr_zfill2_listZinit_hr_zfill2_listZdate_type_hr_listZdate_type_hr_startZdate_type_hr_endZdate_type_hr_incZ date_start_dtZ date_end_dtZ time_infodate_dt valid_time_dt init_time_dtrBrntrrr get_time_infosd        rcCsFd||d}|ddkr|}nd||d}|dkr!|d}|S)z! Get a initialization hour Args: valid_hour - valid hour (integer) forecast_hour - forecast hour (integer) rgrr) valid_hourrn init_hourrrr get_init_hour3s rcCs>||d}|ddkr|}n||d}|dkr|d}|S)z! Get a valid hour Args: init_hour - init hour (integer) forecast_hour - forecast hour (integer) rgrr)rrnrrrr get_valid_hourCs   rcCsd}gd}tt|dkr|t|}|dD]M}|D]@}|d|d} | dkrcd} | | krc|dvr[|dd d d} |d|dd d d} n|d|dd d d} |d kru|| } n|d kr| dkrt|dkr|d} n|} n| dkr| d } n| dkr| d} n|} n|dkr|| } n|dkr|dd d d} t t|t| }| dkrt|dkr|d} n`|} n]| dkr| d } nS| dkr| d} nI|} nF|dkr|dd d d} |t j t| d}|| } n%|dkr1|dd d d} |t j t| d}|| } n||} |dvrM| d|d| d| d | }n| d|d| d | }| d7} | | ks:q"tj||}q|S)a! Creates a filled file path from a format Args: unfilled_file_format - file naming convention (string) valid_time_dt - valid time (datetime) init_time_dt - initialization time (datetime) forecast_hour - forecast hour (string) str_sub_dict - other strings to substitue (dictionary) Returns: filled_file_format - file_format filled in with verifying time information (string) /)lead lead_shiftvalid valid_shiftinit init_shiftr{z?fmt=r!)rrrzshift=r\}?rrz%1H z%2Hz%3Hrrrrirz?shift=)rolistkeyssplitcount partition rpartitionrxrtr^rrurwreplacer#r$r)Zunfilled_file_formatrrrnZ str_sub_dictZfilled_file_formatZformat_opt_listZfilled_file_format_chunkZ format_optZ nformat_optZformat_opt_countshiftZformat_opt_count_fmtZreplace_format_opt_countZforecast_hour_shiftZinit_shift_time_dtZvalid_shift_time_dtrrr format_fillerSs                arc Cstjd}tjd} |d} |dkr|dkrd} n|} gd} t|rqt|rqtd|g| D]?} t|d d | d |d |d |d | gtd| d|gt|d d | d |d |d |d | gtd| d|gt| q/dSt||d|t|ddSdS)a! Do prep work for GEFS production files Args: source_afile - source a file format (string) source_bfile - source b file format (string) prepped_file - prepped file (string) dest_file - destination file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_file - text file path to write that production file is missing (string) Returns: WGRIB2EXECevs.tmp1fullrrh) HGT:500 mbULWRF:top of atmosphere APCP:surface ICETK:surface ICEC:surface TMP:surfaceTMP:2 m above groundTMAX:2 m above groundTMIN:2 m above ground UGRD:850 mb UGRD:200 mbUGRD:10 m above ground VGRD:850 mb VGRD:200 mbVGRD:10 m above groundr-matchr r -i-gribcat>>gefsrNr#r%r.rremoverDrr^)Z source_afileZ source_bfile prepped_filerJrArn prep_methodr>rr working_file1 wgrib_fhrthin_var_level_listthin_var_levelrrr prep_prod_gefs_files@        rc Ctjd}tjd}|d} |dkr`|dkrd} n|} gd} t|rQtd|g| D]#} t|d d | d |d |d |d | gtd| d|gt| q+dSt||d|t|ddSdS)a{! Do prep work for CFS production pgbf files Args: source_pfile - source pgbf file format (string) prepped_pfile - prepped pgbf file (string) dest_pfile - destination pgbf file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_pfile - text file path to write that production file is missing (string) Returns: rrrrrrh)rrrrrrrrr r rrrrcfsrNr) Z source_pfileZ prepped_pfileZ dest_pfilerArnrZlog_missing_pfilerrrrrrrrr prep_prod_cfs_pfiles0      rc Cr)a{! Do prep work for CFS production flxf files Args: source_ffile - source flxf file format (string) prepped_ffile - prepped flxf file (string) dest_ffile - destination flxf file (string) init_dt - initialization date (datetime) forecast_hour - forecast hour (string) prep_method - name of prep method to do (string) log_missing_ffile - text file path to write that production file is missing (string) Returns: rrrrrrh) rrrrrrrrrrrr r rrrrrrNr) Z source_ffileZ prepped_ffileZ dest_ffilerArnrZlog_missing_ffilerrrrrrrrr prep_prod_cfs_ffileJs0      rc Cstjd}tjd}tjtd|dd}|d}gd}t|rStd|g|D]#} t|d d | d |d |d |d |gtd|d|gt|q.nt ||d|t ||dS)a! Do prep work for GFS analysis production files Args: source_file - source file format (string) dest_file - destination file (string) date_dt - date (datetime object) log_missing_file - text file path to write that production file is missing (string) Returns: rratmos.rr\r)rrrrrrrr r rrrrGFSN) r#r%r$rgetcwdrr.rrrFrK) rIrJr~r>rrrrrrrrr prep_prod_gfs_files*    rc CsLtjd}tjd}tjtd|dd}dD]V}|d|}|dd|d } tjtd| dd} t|rYttj|d d d tj|d d|| gn t ||d| ||dkrl| } q|dkrr| } qt| rt| rt | } t | }t j |ddd}| D]6}|dkr||| |d||q|dkr||dq|dkr||dq||| |qt| jD] }||t| j|qdD]J}||| j|j| j|j}| j| D]}||| j||iq|dkr| j|ddd|dd<q| j|dd|dd<qdD]l}||| j|jd }| j| D](}|d!kr\||| j||d"d#iqB||| j||iqBtjt|j|d$dd%ddf| j|d$d%dddff| j|j}||dd<q.|t||dS)&a! Do prep work for OSI-SAF production files Args: daily_source_file_format - daily source file format (string) daily_dest_file - daily destination file (string) date_dt - date (datetime object) log_missing_file - text file path to write that production file is missing (string) Returns: FIXevsCDO_ROOTrrr\)nhshz {hem?fmt=str}zmulti..binZcdoz remapbil,Z cdo_gridsz G003.gridzOSI-SAF rrr/NETCDF3_CLASSICformathistoryrZsouthernmost_latitudez-90ZareaZGlobal)time time_bndslatlonrN)Zice_concZraw_ice_conc_valuesZ status_flagZtotal_uncertaintyZsmearing_uncertaintyZalgorithm_uncertainty)rrZ long_nameznorthern hemisphereZgloberZ) r#r%r$rrrrr.rrFr<netcdfDatasetZncattrs setncattrZ getncattrrZ dimensionsrZcreateDimensionroZcreateVariable variablesZdatatypeZ setncattsrpmaZ masked_equalZvstackZ _FillValuecloserK)Zdaily_source_file_formatdaily_dest_filer~r>rrdaily_prepped_fileZhemZhem_source_fileZ hem_dest_fileZhem_prepped_fileZnh_prepped_fileZsh_prepped_fileZnh_dataZsh_dataZ merged_dataattrZdimvarZ merged_varkZmerged_var_valsrrr prep_prod_osi_saf_files              rcCstjtd|dd}t|rt||nt||d|t|rJtj |ddd}t j dd }|d d d d d |d d d <| t||d S)a! Do prep work for GHRSST OSPO production files Args: daily_source_file - daily source file (string) daily_dest_file - daily destination file (string) date_dt - date (datetime object) log_missing_file - text file path to write that production file is missing (string) Returns: rrr\z GHRSST OSPOarrz1981-01-01 00:00:00%Y-%m-%d %H:%M:%SrNrr) r#r$rrrr.rKrFrrrurvr)Zdaily_source_filerr~r>rZdly_prepped_dataZghrsst_ospo_date_since_dtrrr prep_prod_ghrsst_ospo_file s$  $rc Chtjtd|dd}g}|D]}tj|r"||qtd|ddqt|dkrdg}|D]}||q8|d ||t |t |rt j |d d d }| d |ddtjdd}|d||jdddddg|jddd<|n td|dd|tj|rtd|d|t||dSdS)a! Gather weekly OSI-SAF production files Args: weekly_source_file_list - list of daily files to make up weekly average file weekly_dest_file - weekly destination file (string) weekly_dates - date span for weekly period (tuple of datetimes) Returns: rrr\ WARNING:  does not exist, not using in weekly average filencea-orrr start_daterr1978-01-01 00:00:00rNr!"WARNING: Not enough files to make r*rLinking rGr#r$rrrr&ryrrorr.rrrrxrurv total_secondsrrsymlink) weekly_source_file_listweekly_dest_file weekly_datesweekly_prepped_filencea_weekly_source_file_listweekly_source_file ncea_cmd_list weekly_dataosi_saf_date_since_dtrrr weekly_osi_saf_file)R         rc Cr)ak! Gather monthly OSI-SAF production files Args: monthly_source_file_list - list of daily files to make up monthly average file monthly_dest_file - monthly destination file (string) monthly_dates - date span for monthly period (tuple) Returns: rrr\rr!not using in monthly average filergrrrrrrrrrrNr!rr*rrrGr) monthly_source_file_listmonthly_dest_file monthly_datesmonthly_prepped_filencea_monthly_source_file_listmonthly_source_filer monthly_datarrrr monthly_osi_saf_file]rrcCtjtd|dd}g}|D]}tj|r"||qtd|ddqt|dkrdg}|D]}||q8|d ||t |t |rt j |d d d }| d |dd| d|dd| d|dd| d|dd|dddd|ddd<|n td|dd|tj|rtd|d|t||dSdS)a! Gather weekly GHRSST OSPO production files Args: weekly_source_file_list - list of daily files to make up weekly average file weekly_dest_file - weekly destination file (string) weekly_dates - date span for weekly period (tuple of datetimes) Returns: rrr\rrrrrrrrr start_timerrtime_coverage_start stop_timer!time_coverage_endrNrdrr*rrrGr#r$rrrr&ryrrorr.rrrrxrr)rrrrrrrrrrr weekly_ghrsst_ospo_file^            r cCr)ao! Gather monthly GHRSST OSPO production files Args: monthly_source_file_list - list of daily files to make up monthly average file monthly_dest_file - monthly destination file (string) monthly_dates - date span for monthly period (tuple) Returns: rrr\rrrrgrrrrrrrrr r r!r rNrdrr*rrrGr )rrrrrrrrrrr monthly_ghrsst_ospo_filerrcCst||||i}tj|sEt||||i}d|vr#t|||ddStj|r;td|d|t||dStd|ddSdS)a! This gets a model file and saves it in the specified destination Args: valid_time_dt - valid time (datetime) init_time_dt - initialization time (datetime) forecast_hour - forecast hour (string) source_file_format - source file format (string) dest_file_format - destination file format (string) Returns: z wgrbbul/ecmwfrrrGrr N)rr#r$r&Zprep_prod_ecmwf_filerr)rrrnsource_file_formatdest_file_formatrJrIrrr get_model_files  rcCsxt|||dgi}tj|s:t|||dgi}tj|r0td|d|t||dStd|ddSdS)aC! This gets a truth/obs file and saves it in the specified destination Args: valid_time_dt - valid time (datetime) source_file_format - source file format (string) dest_file_format - destination file format (string) Returns: rhrrGrr N)rr#r$r&rr)rrrrJrIrrr get_truth_file!s   rcCstj|d|dd}|d}|d}t|d}t|d}d}|}g}i} ||kri| t|<|dd krtj|d d |d||d d |d dd} tj|d d |d||d d|d dd} ||t|d| t|d<||7}||ks-t| D]i} g} t| | D]\}t | | | |d| | |d| | |di}t | | | |d| | |d| | |di}tj |r| dt d|ddq| d| | | |dqqtt jt t j|tdtd}|S)aB! Check if COMOUT GEFS prep files exist and adjust fhr_list Args: job_dict - dictionary containing settings job is running with (strings) Returns: fhr_list - adjusted list of forecast hours that are needed for prep (string) ZINITDATE init_hr_startr2MODEL fhr_startfhr_end JOB_GROUPZ retrieve_dataCOMOUTrzens30.tzz .pgrb2.0p50.f{lead?fmt=%3H}zens*.t valid_date init_daternfile1rrrnTz COMOUT files z exist and will notz be generated in prep restartFre)rurvrtrr#r$rrrrr&ryrrpasarrayunique)job_dict init_date_dtr@Zinithrfhr_minfhr_maxfhr_incrBr}fhr_check_output_dictoutput_COMOUT_file_formatZrestart_COMOUT_file_formatfhr_keyZfhr_key_output_files_exist_list fhr_fileN_keyfhr_fileN_COMOUTZfhr_fileN_restartrrr check_prep_files9s           r-c Cs8tj|d|dd}tj|d|dd}tj|d|dd|d }|d }|d }g}i}i}d d |ddDD]} i|t| <i|t| <|tj| d}|ddkr|ddkr|ddkr|ddkrt|d} tj|d||d| dd} tj|d|dd|d|d |dd |dd|dd!} tj|d"|dd|d|d |dd |dd|dd!} ||t| d#|t| d$<||t| d#|t| d$<qEt | D]l}g}t || D]I}t | |||d%|||d&|||d'i}tj |rD| d(|ddkrB|ddkrB| |||d'q| d)qtd*d+|Drat|d,kra| |qt tjttj|td-td-}t|}g}t | D]}t || D]~}t | |||d%|||d&|||d'i}t | |||d%|||d&|||d'i}tj |rt||td.|d/d0|||d'|vr||||d'q|||d'|vr||f|vr| ||fqqt|d1krd(}nd)}|||fS)2aN! Check if model files exist for daily reformat step Args: job_dict - dictionary containing settings job is running with (strings) Returns: model_files_exist - if correct number of model files exist or not (boolean) fhr_list - list of forecast hours that model files exist for (string) model_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT DATEvalid_hr_startr2CORRECT_INIT_DATErDATAr_rrmemberscSg|]}t|qSrrtrirrr r`z+check_daily_model_files..CORRECT_LEAD_SEQ,rir reformat_data grid2grid VERIF_TYPEsstjob_nameSSTr\data.ens.{init?fmt=%Y%m%d%H}.rMETplus_outputRUNr grid_stat_8_{lead?fmt=%2H}0000L_{valid?fmt=%Y%m%d_%H%M%S}V_pairs.ncrrr rrrnTFcs|]}|dkVqdSTNrrxrrr r z*check_daily_model_files..rreCOMOUT restart file  exists" and will be used in stats restartrrurvr#r$rrrrwr^rrrr&ryallrorpr!r"rtcopydeepcopyrKrrr# valid_date_dtr$verif_case_dirr@r3r}fhr_check_input_dictr(rBmbinput_file_formatoutput_DATA_file_formatr)r*fhr_key_input_files_exist_listr+ fhr_fileNinput_fhr_list"model_copy_output_DATA2COMOUT_listfhr_fileN_DATAr,model_files_existrrr check_daily_model_filess(                    % rac Cstj|d|dd}tj|d|dd}tj|d|dd|d }|d }|d }g}i}i}d d |ddDD]} i|t| <i|t| <|tj| d}|ddkr|ddvre|ddvre|ddvret|d} |ddkr|d dkrtj|d||d| dd} ntj|d||d| dd} |dd krtj|d!|d"d#|d|d |dd$|dd|dd%} tj|d&|d"d#|d|d |dd$|dd|dd%} |dd'krItj|d!|d"d#|d|d |dd(|ddd)d*d+} tj|d&|d"d#|d|d |dd(|ddd)d*d+} ||t| d,|t| d-<||t| d,|t| d-<|dd kr|dd.kr}tj|dd/d0}qE|ddkrtj|dd1d2}qE|dd3kr2|dd'kr2|dd4kr2|dd5kr2tj|d!|d"d#|d|d |dd(|dd6d7d*d8d9} tj|d!|d"d#|d|d |dd:|dd|dd;} tj|d&|d"d#|d|d |dd:|dd|dd;} ||t| d,|t| d-<||t| d,|t| d-<qEt | D]}g}t || D]}t | |||d<|||d=|||d>i}|ddkr|dd?vrt ||||d<|||d<d@gi}tj |rtj |r| dA| |||d>qE| dBqE|ddkr|ddCvrtj |r| dA| |||d>qE| dBqE|dd3kr|dd5krtj |r| dA| |||d>qE| dBqEtdDdE|Drt|dFkr| |q9t tjttj|tdGtdG}t|}g}t | D]}t || D]~}t | |||d<|||d=|||d>i}t | |||d<|||d=|||d>i}tj |rt||tdH|dIdJ|||d>|vr||||d>qG|||d>|vr||f|vr| ||fqGq=t|dKkrdA}ndB}t|dFkrdB}|||fS)LaQ! Check if model files exist for weekly reformat and assemble step Args: job_dict - dictionary containing settings job is running with (strings) Returns: model_files_exist - if 80% of model files exist or not (boolean) fhr_list - list of forecast hours that model files exist for (string) model_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r0rr1rr2rrr3cSr4rr5r6rrr r`?r8z,check_weekly_model_files..r9r:rirr;r<grid2obsr=)seaicer>anomprepbufr pres_lvlsr?) Concentrationr@ TempAnom2m GenEnsProd GeoHeightAnomr\rgrrA .pgbf.ensrCrrBr<rDrErrFrGrrc gen_ens_prod_GenEnsProd_FHR{lead?fmt=%3H}_{valid?fmt=%Y%m%d}_{valid?fmt=%H}0000V_ens.ncrr reecmwfecmwf.{valid?fmt=%Y%m%d%H}.anlgfsgfs.{valid?fmt=%Y%m%d%H}.anl assemble_datarfri _GenEnsProd_FHR{lead?fmt=%3H}_{valid?fmt=%H}0000V_ens.nc point_stat_4_{lead?fmt=%2H}0000L_{valid?fmt=%Y%m%d_%H%M%S}V.statrrrnrirkrhTF)rhr@rjcsrHrIrrJrrr r rLz+check_weekly_model_files..rrerMrNrOrrPr#rUr$rVr@r3r}rWr(rBrXrYrZr)truth_file_formatr*r[r+r\ truth_filer]r^r_r,r`rrr check_weekly_model_files s<                                     % rc Cs8tj|d|dd}tj|d|dd}tj|d|dd|d }|d }|d }g}i}i}d d |ddDD]} i|t| <i|t| <|tj| d}|ddkr|ddkr|ddvr|ddvrt|d} tj|d||d| dd} tj|d|dd|d|d |dd |dd|dd!} tj|d"|dd|d|d |dd |dd|dd!} ||t| d#|t| d$<||t| d#|t| d$<qEt | D]l}g}t || D]I}t | |||d%|||d&|||d'i}tj |rD| d(|ddkrB|ddvrB| |||d'q| d)qtd*d+|Drat|d,kra| |qt tjttj|td-td-}t|}g}t | D]}t || D]~}t | |||d%|||d&|||d'i}t | |||d%|||d&|||d'i}tj |rt||td.|d/d0|||d'|vr||||d'q|||d'|vr||f|vr| ||fqqt|d1krd(}nd)}|||fS)2aE! Check if model files exist for monthly reformat step Args: job_dict - dictionary containing settings job is running with (strings) Returns: model_files_exist - if 80% of model files exist or not (boolean) fhr_list - list of forecast hours that model files exist for (string) model_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r0rr1rr2rrr3cSr4rr5r6rrr r`Lr8z-check_monthly_model_files..r9r:rirr;r<r=rdr>r?rhr@r\rArBrCrrDrErrFrGrrr rrrnTFcsrHrIrrJrrr r rLz,check_monthly_model_files..rrerMrNrO1rPrTrrr check_monthly_model_files-s(                    % rc Ctj|d|dd}tj|d|dd}tj|d|dd|d }|d }|d }g}i}i}d d |ddDD]} i|t| <i|t| <|tj| d}|ddkr|ddvre|ddvre|ddvret|d} |ddkr|d dkrtj|d||d| dd} ntj|d||d| dd} |dd krtj|d!|d"d#|d|d |dd$|dd|dd%} tj|d&|d"d#|d|d |dd$|dd|dd%} |dd'krItj|d!|d"d#|d|d |dd(|ddd)d*d+} tj|d&|d"d#|d|d |dd(|ddd)d*d+} ||t| d,|t| d-<||t| d,|t| d-<|dd kr|dd.kr}tj|dd/d0}qE|ddkrtj|dd1d2}qE|dd3kr2|dd'kr2|dd4kr2|dd5kr2tj|d!|d"d#|d|d |dd(|dd6d7d*d8d9} tj|d!|d"d#|d|d |dd:|dd|dd;} tj|d&|d"d#|d|d |dd:|dd|dd;} ||t| d,|t| d-<||t| d,|t| d-<qEt | D]}g}t || D]}t | |||d<|||d=|||d>i}|ddkr|dd?vrt ||||d<|||d<d@gi}tj |rtj |r| dA| |||d>qE| dBqE|ddkr|ddCkrtj |r| dA| |||d>qE| dBqE|dd3kr|dd5krtj |r| dA| |||d>qE| dBqEtdDdE|Drt|dFkr| |q9t tjttj|tdGtdG}t|}g}t | D]}t || D]~}t | |||d<|||d=|||d>i}t | |||d<|||d=|||d>i}tj |rt||tdH|dIdJ|||d>|vr||||d>qG|||d>|vr||f|vr| ||fqGq=t|dKkrdA}ndB}t|dFkrdB}|||fS)LaT! Check if model files exist for days 6-10 reformat and assemble step Args: job_dict - dictionary containing settings job is running with (strings) Returns: model_files_exist - if 80% of model files exist or not (boolean) fhr_list - list of forecast hours that model files exist for (string) model_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r0rr1rr2rrr3cSr4rr5r6rrr r`r8z.check_days6_10_model_files..r9r:rirr;rbr=rerfrgr?rirjrkr\rgrrArlrCrrBr<rDrErrFrGrrcrmrnrorprr rerqrrrsrtrurfrirvrwrxryrzr{rrrnr|rhTFrjcsrHrIrrJrrr r rLz-check_days6_10_model_files..rrerMrNrO rPr}rrr check_days6_10_model_files<                                     % rc Cr)LaT! Check if model files exist for weeks 3-4 reformat and assemble step Args: job_dict - dictionary containing settings job is running with (strings) Returns: model_files_exist - if 80% of model files exist or not (boolean) fhr_list - list of forecast hours that model files exist for (string) model_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r0rr1rr2rrr3cSr4rr5r6rrr r`r8z.check_weeks3_4_model_files..r9r:rirr;rbr=rr?rr\rgrrArlrCrrBr<rDrErrFrGrrcrmrnrorprr rerqrrrsrtrurfrirvrwrxryrzr{rrrnr|rhTFrjcsrHrIrrJrrr r rLz-check_weeks3_4_model_files..rrerMrNrOrPr}rrr check_weeks3_4_model_filesrrc Cstj|d|dd}tj|d|dd|d}|d}|d }t|d }t|d }d }|}g} i} i} ||k ri| t|<i| t|<|tj|d } |ddkr|ddkr|ddvr|ddvrt|d} tj|d||d| dd}d}|dkr|d|dkr|tjd|d | t|d|dt t|dt|d<|d7}|dks n@|| t|dt t|d< n0|dd kr|ddkr|dd!vra|dd"vratj|d#|d$d%|d|d|dd&|dd|d d'd(d)d*d+}tj|d#|d$d%|d||dd,|dd|dd-d.d/}tj|d0|d$d%|d||dd,|dd|dd-d.d/}n|dd!vr|dd1vrtj|d#|d$d%|d|d|dd&|dd|d d2d(d)d*d+}tj|d#|d$d%|d||dd3|dd|dd-d4d/}tj|d0|d$d%|d||dd3|dd|dd-d4d/}n)|dd!vrm|dd5vrmtj|d#|d$d%|d|d|dd&|dd|d d6d(d)d*d+}tj|d#|d$d%|d||dd7|dd|dd-d8d/}tj|d0|d$d%|d||dd7|dd|dd-d8d/}n|dd9kr|dd:krtj|d#|d$d%|d|d|dd;|dd|d dd?d@}tj|d#|d$d%|d||ddA|dd|dd-dBd/}tj|d0|d$d%|d||ddA|dd|dd-dBd/}n|ddvr||ddCvr|tj|d#|d$d%|d|d|dd;|dd|d d'd(d=d>d?d@}tj|d#|d$d%|d||dd,|dd|dd-d.d/}tj|d0|d$d%|d||dd,|dd|dd-d.d/}n|ddvr|ddDvrtj|d#|d$d%|d|d|dd;|dd|d dEd(d=d>d?d@}tj|d#|d$d%|d||ddF|dd|dd-dGd/}tj|d0|d$d%|d||ddF|dd|dd-dGd/}n tj|d||dd}|dd9krg|dd:krgd}|dkrW|d|dkrR|tjd|d | t|d|d| t|dt|d<|d7}|dks%|| t|d| t|d<n|ddHvr|ddIvrt|dJvrd}|dKkr|tjd|d | t|d|d| t|dt|d<|d7}|dKks|| t|d| t|d<nA|ddvr|ddDvr|dLkr d}|dMkr|tjd|d | t|d|d| t|dt|d<|d7}|dMks|| t|d| t|d<n|dd!vrb|dd1vrb|dNkr`d}|dOkrR|tjd|d | t|d|d| t|dt|d<|d7}|dOks)|| t|d| t|d<n|dd!vr|dd5vr|dPkrd}|dQkr|tjd|d | t|d|d| t|dt|d<|d7}|dQks||| t|d| t|d<nH|| t|d| t|d<n8|ddRk r|ddkrLtj|d#|d$d%|d||dd;|dd|ddS}tj|d0|d$d%|d||dd;|dd|ddS}|dd!vrK|dd"vrKtj|d#|d$d%|d||dd,|dd|dd-d.d/}n|dd!vr|dd1vrtj|d#|d$d%|d||dd3|dd|dd-d4d/}n|dd!vr|dd5vrtj|d#|d$d%|d||dd7|dd|dd-d8d/}n|ddTvr|ddUvrtj|d#|d$d%|d||dd,|dd|dd-d.d/}ns|ddTvr|ddVvrtj|d#|d$d%|d||ddF|dd|dd-dGd/}n?|dd9krO|dd:krOtj|d#|d$d%|d||ddA|dd|dd-dBd/}n tj|d||dW}|dd9kr|dd:kr|| t|d| t|d<|| t|d| t|d<nw|ddHvr|ddIvrt|dJvr|| t|d| t|d<|| t|d| t|d<nD|ddvr|ddDvr|dLkr|| t|d| t|d<|| t|d| t|d<n|dd!vr|dd1vr|dNkr|| t|d| t|d<|| t|d| t|d<n|dd!vrJ|dd5vrJ|dPkrJ|| t|d| t|d<|| t|d| t|d<n|ddXk r|ddYkr|ddZkrtj|d#|d$d%|d|d|dd&|dd|d d'd(d)d*d[}tj|d#|d$d%|d||dd\|dd]|dYd|dd^d_}tj|d0|d$d%|d||dd\|dd]|dYd|dd^d_}n%|ddYk rn|dd`k rntj|d#|d$d%|d|d|dd&|dd|d d2d(d)d*d[}tj|d#|d$d%|d||dd\|dd]|dYd|ddad_}tj|d0|d$d%|d||dd\|dd]|dYd|ddad_}n|ddYk r|ddbk rtj|d#|d$d%|d|d|dd&|dd|d d6d(d)d*d[}tj|d#|d$d%|d||dd\|dd]|dYd|ddcd_}tj|d0|d$d%|d||dd\|dd]|dYd|ddcd_}n tj|d||dW}|ddYk rZ|ddZk rZt|dJv rYd}|dKk rK|tjd|d | t|d|d| t|dt|d<|d7}|dKk s"|| t|d| t|d<n|ddYk r|dd`k r|dNk rd}|dOk r|tjd|d | t|d|d| t|dt|d<|d7}|dOk st|| t|d| t|d<nQ|ddYk r|ddbk r|dPk rd}|dQk r|tjd|d | t|d|d| t|dt|d<|d7}|dQk s|| t|d| t|d<||7}||ksAt | D]}g}t | | D]x}t || ||dd| ||de| ||dfi}tj| r|dg|ddk rZ|ddhv rZ| | ||df q|dd k ru|ddiv ru| | ||df q|ddRk r| | ||df q|dj qtdkdl|D r|dgdk r| | q t tjttj| tdmtdm} t| }g}t | D]j}t | | D]^}t || ||dd| ||de| ||dfi}t || ||dd| ||de| ||dfi}tj| rt|| q| ||df| v r.||f|v r.|||f qѐ qt| dk r DailyAvg_SSTrFZ DailyAvg_z_{lead?fmt=%2H}0000L_z{valid?fmt=%Y%m%d}_z{valid?fmt=%H}0000V_zpairs.ncZ daily_avg_z)valid{valid_shift?fmt=%Y%m%d%H?shift=-24})WeeklyAvg_Concentration WeeklyAvg_SST)MonthlyAvg_ConcentrationMonthlyAvg_SSTZ MonthlyAvg_Z monthly_avg_z*valid{valid_shift?fmt=%Y%m%d%H?shift=-720})rdr>rgre)rrrr)168336504672840i<rigenerate_statsr{)r>rd)rr)rrz$.{init?fmt=%Y%m%d%H}.f{lead?fmt=%3H}rcrfrzfhr{lead?fmt=%3H}.statZstat_analysis_fcstZ _obsprepbufr_z1_SL1L2_valid{valid_shift?fmt=%Y%m%d%H?shift=-168}zto{valid?fmt=%Y%m%d%H}.statrz1_SL1L2_valid{valid_shift?fmt=%Y%m%d%H?shift=-120}rz1_SL1L2_valid{valid_shift?fmt=%Y%m%d%H?shift=-336}rrrnT)rkrhr@r|FcsrHrIrrJrrr r rLz$check_model_files..re)rurvr#r$rrtrrwr^Zfhr_check_dictrrrrr&ryrrrpr!r"rRrSrKro)r#rUrVr@r3r%r&r'rBr}rWr(r$rXmodel_file_formatnfrYrZr)r*r[r+r\r]r^r_r,r`rrr check_model_filessZ                                                                                                                 %          rc Cstj|d|dd}tj|d|dd|d}g}g}|dd krs|dd kr|d d krltj|d dd}d}|dkrjt||tjd|d|tjd|ddgi}|||d7}|dksDnu|d dkrtj|d dd}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksn8|ddkrr|d dkrrtj|d dd}d}|dkrt||tjd|d|tjd|ddgi} || |d7}|dkstj|d|dd|d d|dd!|d d"d#} tj|d$|dd|d d|dd!|d d"d#} d}|dkrrt| |tjd|d|tjd|ddgi} t| |tjd|d|tjd|ddgi} || | f|d7}|dks0nn|dd%kr|ddkr|d dkr|d&d'krtj|d|dd|d d|dd!|d d|dd(d#}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksg}|}|D]}tj |dr|d)| |q|d*qt d+d,|Drt |dkrd)}||fSg}|D]}tj |r0|d)q |d*q t d-d,|DrO|d)dkrOd)}||fSd*}||fS).a! Check if obs files exist for weekly reformat and assemble step Args: job_dict - dictionary containing settings job is running with (strings) Returns: truth_files_exist - if 80% of truth files exist or not (boolean) truth_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r1rr2rrr;r<r=rerArqrrrrrrirhr!rgrsrtrcrf prepbufr_nam!prepbufr.nam.{valid?fmt=%Y%m%d%H}rDrEr%Y%m%dpb2nc_ _nam_valid{valid?fmt=%Y%m%d%H}.ncrrur?ri_validTFcsrHrIrrJrrr r ` rLz+check_weekly_truth_files..csrHrIrrJrrr r j rLrurvr#r$rrrwryrxr&rrQrorrr#rUrVtruth_input_file_listtruth_output_file_listr~rrprepbufr_file_format prepbufr_filepb2nc_DATA_output_file_formatpb2nc_COMOUT_output_file_formatpb2nc_DATA_outputpb2nc_COMOUT_outputpb2nc_file_format pb2nc_filetruth_output_files_exist_list"truth_copy_output_DATA2COMOUT_listtruth_file_tupletruth_files_existtruth_input_files_exist_listrrr check_weekly_truth_files >                       rc Cstj|d|dd}tj|d|dd|d}g}g}|dd krs|dd kr|d d krltj|d dd}d}|dkrjt||tjd|d|tjd|ddgi}|||d7}|dksDnu|d dkrtj|d dd}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksn8|ddkrr|d dkrrtj|d dd}d}|dkrt||tjd|d|tjd|ddgi} || |d7}|dkstj|d|dd|d d|dd!|d d"d#} tj|d$|dd|d d|dd!|d d"d#} d}|dkrrt| |tjd|d|tjd|ddgi} t| |tjd|d|tjd|ddgi} || | f|d7}|dks0nn|dd%kr|ddkr|d dkr|d&d'krtj|d|dd|d d|dd!|d d|dd(d#}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksg}|}|D]}tj |dr|d)| |q|d*qt d+d,|Drt |dkrd)}||fSg}|D]}tj |r0|d)q |d*q t d-d,|DrO|d)d.krOd)}||fSd*}||fS)/a! Check if obs files exist for Days 6-10 reformat and assemble step Args: job_dict - dictionary containing settings job is running with (strings) Returns: truth_files_exist - if 80% of truth files exist or not (boolean) truth_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r1rr2rrr;r<r=rerArqrrrrrrirhr!rgrsrtrcrfrrrDrErrrrrrrur?rirTFcsrHrIrrJrrr r rLz-check_days6_10_truth_files..csrHrIrrJrrr r rLrrrrrr check_days6_10_truth_filesq rrc Cs^tj|d|dd}tj|d|dd|d}g}g}|dd kr|dd kr|d d krltj|d dd}d}|dkrjt||tjd|d|tjd|ddgi}|||d7}|dksDn|d dkrtj|d dd}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksn^|ddkr|d dkrd|dvr|ddd }tj|d d|d|d d!} d}|dkrt| |tjd|d|tjd|ddgi} || |d7}|dkstj|d"|d#d | d$d|dd%|d d|dd&d'} tj|d(|d#d | d$d|dd%|d d|dd&d'} d}|dkrt| |tjd|d|tjd|ddgi} t| |tjd|d|tjd|ddgi}|| |f|d7}|dksVnn|dd)kr|ddkr|d dkr|dd*krtj|d"|d#d | d$d|dd%|d d|dd&d'}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksg}|}|D]&}tj |dr-|d+t |d|d| |q |d,q td-d.|DrKt|dkrKd+}||fSg}|D]}tj |r_|d+qO|d,qO|dd kr|ddkrtd/d.|Drt|dkrd+}||fSd,}||fStd0d.|Dr|d+d1krd+}||fSd,}||fS)2a! Check if obs files exist for Weeks 3-4 reformat and assemble step Args: job_dict - dictionary containing settings job is running with (strings) Returns: truth_files_exist - if 80% of truth files exist or not (boolean) truth_copy_output_DATA2COMOUT_list - list of files to copy from DATA to COMOUT r.r/r2r1rr2rrr;r<r=rerArqrrrrrrirhr!rgrsrtrcrfPrepbufrr?r prepbufr_ prepbufr.r{valid?fmt=%Y%m%d%H}rDrErrrrrruriTFcsrHrIrrJrrr r rLz-check_weeks3_4_truth_files..csrHrIrrJrrr r rLcsrHrIrrJrrr r rLr)rurvr#r$rrrwryrlowerrxr&rKrrQrorr)r#rUrVrrr~rr prepbufr_namerrrrrrrrrrrrrrrr check_weeks3_4_truth_files sr                             rc Csdtj|d|dd}tj|d|dd|d}g}|dd kr|dd kr|d d krjtj|d dd}d}|dkrht||tjd|d|tjd|ddgi}|||d7}|dksBn|d dkrtj|d dd}d}|dkrt||tjd|d|tjd|ddgi}|||d7}|dksn|ddkr|d dkrd|dvr|ddd }tj|d d|d|d d!}d}|dkrt||tjd|d|tjd|ddgi} || |d7}|dksސnx|dd"krv|ddkrt|d dkrt|dd#krttj|d$|d%d | d&d|dd'|d d|dd(d)} d}|dkrtt| |tjd|d|tjd|ddgi} || |d7}|dksMn|dd*kr~|dd krB|d d+kr|dd,krtj|d d-d.} t| ||dgi} || n|d d+kr|dd/krtj|d d-d0} t| ||dgi} || n|d d1kr|dd2krtj|d d3d4}t|||dgi}||n|d d1kr|dd5krtj|d d3d6}t|||dgi}||nb|d d1krA|dd7krAtj|d d3d8}t|||dgi}||n<|ddkr~|d dkr~tj|d$|d%d | d&d|dd'|d d|dd(| dd9} || g}|D]}tj |r|d:q|d;qt da! Args: job_dict - dictionary containing settings job is running with (strings) Returns: all_truth_file_exist - if all needed truth files exist or not (boolean) r.r/r2r1rr2rrr;r<r=rerArqrrrrrrirhr!rgrsrtrcrfrr?rrrrrrurirDrErrrrrrdrosi_safzQosi_saf.multi.{valid_shift?fmt=%Y%m%d%H?shift=-168}to{valid?fmt=%Y%m%d%H}_G003.ncrzQosi_saf.multi.{valid_shift?fmt=%Y%m%d%H?shift=-720}to{valid?fmt=%Y%m%d%H}_G003.ncr>r ghrsst_ospozIghrsst_ospo.{valid_shift?fmt=%Y%m%d%H?shift=-24}to{valid?fmt=%Y%m%d%H}.ncrzJghrsst_ospo.{valid_shift?fmt=%Y%m%d%H?shift=-168}to{valid?fmt=%Y%m%d%H}.ncrzJghrsst_ospo.{valid_shift?fmt=%Y%m%d%H?shift=-720}to{valid?fmt=%Y%m%d%H}.ncz.ncTFcsrHrIrrJrrr r ^rLz$check_truth_files..)rurvr#r$rrrwryrrrxr&rQro)r#rUrVZtruth_file_listr~rrrrrrrZosi_saf_file_formatZ osi_saf_fileZghrsst_ospo_file_formatZghrsst_ospo_fileZtruth_files_exist_listZall_truth_file_existrrr check_truth_files s                                rc Csrtj|d|dd|dd|dd|d|d |d}ttj|d }t|d kr5d }|Sd }|S)a! Check for MET .stat files Args: job_dict - dictionary containing settings job is running with (strings) Returns: stat_files_exist - if .stat files exist or not (boolean) r1rr2rrDrErr.rz*.statrTF)r#r$rglobro)r#Zmodel_stat_file_dirZstat_file_listZstat_files_existrrr check_stat_fileses   rcCsddddddddddddddddddddd}|t|vr7||d}||d}||d}n td |d td |||fS) al! This returns the valid hour start, end, and increment information for a given observation Args: obs - observation name (string) Returns: valid_hr_start - starting valid hour (integer) valid_hr_end - ending valid hour (integer) valid_hr_inc - valid hour increment (integer) rrg)r/ valid_hr_end valid_hr_incr)OSI-SAF GHRSST-OSPOECMWFrBUFRr/rrzFATAL ERROR: Cannot get z valid hour informationr!)rrrrr)rEZobs_valid_hr_dictr/rrrrr get_obs_valid_hrs|s8     rcCsgd}i}|D] }tj|||<q||d<tjd|d<tjd|d<tjdd}|d d |d <|d d |d <tjdd}|d d |d<|d d |d<|S)a! This initializes a dictionary of environment variables and their values to be set for the job pulling from environment variables already set previously Args: verif_type - string of the use case name group - string of the group name job - string of job name Returns: job_env_dict - dictionary of job settings )machineevs_verHOMEevsrr(r1NETrErrZ COMINgefsrSENDCOMZ gefs_membersrr%rr&rZvhr_listrrr\r/rdrZ inithour_listr init_hr_end)r#r%rr^) verif_typegroupjobjob_env_var_list job_env_dictenv_varr|r{rrr initialize_prep_job_env_dicts$ rcCsgd}|dvr:tjtjdtjddtjdddtjd <tjtjd s2ttjd |gd n |d vrE|d d gi}|D] }tj|||<qI|d vr[d|d<||d<|dvr`||d<|d vrq||d<n||d<tj|d|d<tj|d|d<tj|d|d<|dvrtj|dd}|dd|d <|d!d|d"<t |d#krt t t j |td$}nd%}t||d&<nZ|d'vrtd(\} } } n6|d)krtd*\} } } n*|d+krtd,\} } } n|d-krtd.\} } } n|d/kr td0\} } } nd1\} } } t| d|d <t| d|d"<t| |d&<tj|d2d} | dd|d3<| d!d|d4<t | d#krXt t t j | td$} nd%} t| |d5<|S)6ad! This initializes a dictionary of environment variables and their values to be set for the job pulling from environment variables already set previously Args: verif_type - string of the use case name group - string of the group name verif_case_step_abbrev_type - string of reference name in config and environment variables job - string of job name Returns: job_env_dict - dictionary of job settings )rrrrr(r1COMROOTrrErrrCOMINrrr3)r;rurZ gather_statsr1rr2rrDZtmp MET_TMP_DIR)rMET_ROOTrr)condense_stats filter_stats make_plots tar_imagesrZmet_verZINFOZplot_verbosityr)r;rurrrrrr=Zjob_varr?_fhr_minr_fhr_maxr_fhr_incr')ZmeansZsfcZ _vhr_listrrr\r/rdrr!rergr)rdZsea_icerr>rrerrgrrfr)rrr_inithour_listrr init_hr_inc)r#r$rr%r&makedirsextendrr^rorprqrrrsrtrr)rrZverif_case_step_abbrev_typerrrrZverif_type_valid_hr_listZverif_type_valid_hr_incr/rrZverif_type_init_hr_listZverif_type_init_hr_incrrr initialize_job_env_dicts                 rcCsbtdd}t|}|dtj|dd}||||d|}t||||S)z! Get logger Args: log_file - full path to log file (string) Returns: logger - logger object zL%(asctime)s.%(msecs)03d (%(filename)s:%(lineno)d) %(levelname)s: %(message)sz%m/%d %H:%M:%SZDEBUGr)modez Log file: ) loggingZ FormatterZ getLoggerZsetLevelZ FileHandlerZ setFormatterZ addHandlerrinfo)Zlog_fileZ log_formatterloggerZ file_handlerZ logger_inforrr get_loggerAs      rc Cs|dkrtj||d} tj||d} tjt|d} n|dkr=tj||d} tj||d} tjt| d} t| | | | tj}|dkr[|}|tjt| d}n|dkrk|}|tjt| d}g}ddtt|t|t|t|D}tt|D]}|| d|vr| |qt ||}t ||}g}ddtt|t|t| t| D}tt|D]}|| d|vr| |qt ||}t ||}||fS) a! This builds the dates to include in plotting based on user configurations Args: logger - logger object date_type - type of date to plot (string: VALID or INIT) start_date - plotting start date (string, format: YYYYmmdd) end_date - plotting end date (string, format: YYYYmmdd) valid_hr_start - starting valid hour (string) valid_hr_end - ending valid hour (string) valid_hr_inc - valid hour increment (string) init_hr_start - starting initialization hour (string) init_hr_end - ending initialization hour (string) init_hr_inc - initialization hour incrrement (string) forecast_hour - forecast hour (string) Returns: valid_dates - array of valid dates (datetime) init_dates - array of initalization dates (datetime) rbr2rirccSg|] }t|dqSr[rr^r_rrr r`z"get_plot_dates..rkcSrr[rr_rrr r`r) rurvrwrtrpZarangeastyperangerorxrydelete)rrzrend_dater/rrrrrrnZ start_date_dtZ end_date_dtZdt_incZdate_type_dates valid_datesZ init_datesZvalid_remove_idx_listr|dZinit_remove_idx_listr{rrr get_plot_datesWsx            rc Cs|ddkr|dd}n |ddkr|}tj|dddd|d }tj|rSt|}|D]}||vrB|d d }nq3Wd n1sMwYn|d |dddt d| ddd} | S)a! Get the MET columns for a specific line type and MET version Args: logger - logger object met_root - path to MET (string) met_version - MET version number (string) met_line_type - MET line type (string) Returns: met_version_line_type_col_list - list of MET version line type columns (strings) rr\rr!ZshareZmetZ table_filesZmet_header_columns_Vz.txtz : rdNz FATAL ERROR rzDOES NOT EXIST, z*cannot determine MET data column structure r) rrr#r$rr&r:rerrorrrr) rZmet_rootZ met_versionZ met_line_typeZmet_minor_versionZmet_minor_version_col_filer3lineZline_type_colsmet_version_line_type_col_listrrr get_met_line_type_colss6     rcCsp|ddddddddd d d d }|ddddddddd d d d }||fS) z! Format threshold with letter and symbol options Args: thresh - the threshold (string) Return: thresh_symbol - threshold with symbols (string) thresh_letters - treshold with letters (string) gez>=gtreqz==nez!=lez<=lt<)r)ZthreshZ thresh_symbolZ thresh_letterrrr format_threshs r c Cstj|||d} tj| dd} t| dkr!|d| dStj|s|d| t| d } | } Wdn1sDwYd}d |d d |d d |d d |d d |d d | d }| D]/}|d |t j d |d ||dt j t j dd}|d|j||d}qq|d|d|t|d}|| |WddS1swYdSdS)a! Condense the individual date model stat file and thin out unneeded data Args: logger - logger object input_dir - path to input directory (string) output_file - path to output file (string) model - model name (string) obs - observation name (string) grid - verification grid (string) vx_mask - verification masking region (string) fcst_var_name - forecast variable name (string) obs_var_name - observation variable name (string) line_type - MET line type (string) Returns: z_*.statT) recursiverzNO STAT FILES IN MATCHING z$Condensing down stat files matching Nrz | grep "z "zGetting data from z grep -R "z " zUTF-8)rstdoutstderrencodingzRan z Condensed z .stat file at r/)r#r$rrrowarningr&debugr:readlinerPopenPIPEZSTDOUTrZ communicater;)r input_dirZ output_filer@rEgridvx_mask fcst_var_name obs_var_name line_typeZmodel_stat_files_wildcardZmodel_stat_filesZmsfZmet_header_colsZall_grep_outputZ grep_optsZmodel_stat_fileZpsr3rrr condense_model_stat_filess|     "rc> Cst||d|d| }t|D]}|d||dd||d}tjj|g|gddgd}||}tj||d |dd }t |d krtj|d |dd |||d d |dd || | d d| d d| d d| d d||d | |d  dd|d dd d| dd }tj|| dd}tj|r|}n|}tj|sd}d}n d}d}nd}d}tj|r\| dkr\tj|ddddddd }t |d kr\t|d!d } g}!|D]Y}"|"d"kr|!d#q|"d$krSd}#d}$d}%|%| | krQ|!d%t|#d d&t|$|$| kr;|$d7}$n |$| krFd}$|#d7}#|%d7}%|%| | ks q|!|"q|!}|rH|d'krkt|\}&}'n|}&|}'| d'kr{t| \}(})n| }(| })tj|r(|d(|tj|ddd|dddd)}*|*|*d*|dk|*d+| k@|*d,| dd-k@|*d.|k@|*d/|k@|*d0|k@|*d1| k@|*d2|dk@|*d3| k@|*d4|k@|*d5|k@|*d6|&k@|*d7|(k@|*d8| k@}+|+|+d9|}+tj|+d9d:d;|+d9<|+jd9d<}+|+d9j d:|+d9<|+j||ddd=d>tj|r@|d?|dd@dA|n|dB|tjtj||dC},|rtj|r|dD|dE|dtj|ddd|d'gddF}-i}.|dG}/|D]}"||"}0|0|/krt|.|"<qtj|.|"<q|-|.}-|D]Z}1|-j|-d9|1k }2t |2d kr|dH|1dI|qt |2dkr|dJ|1dK|ddLn |dM|1dK||-j!|2d dd|,j!||1f<q|dNvr(|dOvr(| dPvr(t"dQdR}3|dSvrd }4n | dTkrd }4ndU}4d}5dV}6d%}7nd}5|5r| dWkrQ|,j!|,dX|6kdYf}8|,j!|,dX|6kdZf}9dYdZg}:gd[};n#| dTkrt|,j!|,dX|6kd\f}8|,j!|,dX|6kd]f}9d\d]g}:gd^};|:D]}"|3|,j!|,dX|6k|"f|4|,j!|,dX|6k|"f<qv|;D]N}"|"d_vr|3|4|8|3|4|9}rrZ output_dirZmodel_info_dictZ met_info_dictrZfcst_var_levelZfcst_var_threshrZ obs_var_levelZobs_var_threshrrrZ interp_methodZ interp_pointsrzZdatesZmet_format_valid_datesrBrZ model_numZmodel_num_nameZmodel_num_df_indexZ model_dictZcondensed_model_fileZinput_parsed_model_stat_fileZoutput_parsed_model_stat_fileZparsed_model_stat_fileZwrite_parse_stat_fileZread_parse_stat_fileZtmp_dfZncatZ"new_met_version_line_type_col_listZcolZfcountZocountZtotcountZfcst_var_thresh_symbolZfcst_vat_thresh_letterZobs_var_thresh_symbolZobs_vat_thresh_letterZcondensed_model_dfZparsed_model_dfZ model_num_dfZmodel_stat_file_dfZ df_dtype_dictZ float_idxZcol_idxrZ&model_stat_file_df_valid_date_idx_listZcoefZconstZconvertZ units_oldZ units_newZ fcst_avg_oldZ obs_avg_oldZ col1_listZ col2_listZconst2Z all_model_dfrrr build_dfs#                                                rAcCs.|dkr3|jddd}|jddd}|jddd}|jddd}|jddd} n|dkrf|jddd } |jddd } |jddd } |jddd } |jddd } n|dkr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd }|jddd!} |jddd"}!|jddd#}"|jddd$}#|jddd%}$|jddd&}%|jddd'}&|jddd(}'|jddd)}(|jddd*})|jddd+}*|jddd,}+|jddd-},|jddd.}-|jddd/}.|jddd0}/|jddd1}0|jddd2}1|jddd3}2|jddd4}3|jddd5}4|jddd6}5|jddd7}6|jddd8}7|jddd9}8|jddd:}9|jddd;}:|jddd<};|jddd=}<|jddd>}=|jddd?}>|jddd@}?|jdddA}@|jdddB}A|jdddC}B|jdddD}C|jdddE}D|jdddF}E|jdddG}F|jdddH}G|jdddI}H|jdddJ}I|jdddK}J|jdddL}K|jdddM}L|jdddN}M|jdddO}N|jdddP}O|jdddQ}P|jdddR}Q|jdddS}R|jdddT}S|jdddU}T|jdddV}U|jdddW}V|jdddX}W|jdddY}X|jdddZ}Y|jddd[}Z|jddd\}[|jddd]}\|jddd^}]|jddd_}^|jddd`}_|jddda}`|jdddb}a|jdddc}b|jdddd}c|jddde}d|jdddf}e|jdddg}f|jdddh}g|jdddi}h|jdddj}i|jdddk}j|jdddl}k|jdddm}l|jdddn}mnl|dokr7|jdddp}n|jdddq}o|jdddr}p|jddds}q|jdddt}r|jdddu}s|jdddv}t|jdddw}u|jdddx}vn|dykrY|jdddz}w|jddd{}x|jddd|}yn|d}vr|jddd~}z|jddd}{|jddd}||jddd}}|dkr|jddd}~n|dvr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|dkr|jddd}~nh|dkr|jddd}ېnX|dkr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jdddz}w|jddd}|jddd}|jddd|}y|jddd}|jddd}n|dkr|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}ni|dk rA|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}n |dk rK|jddd}|jddd}|jddd}|jddd }|jddd }|jddd }|jddd}|jddd}|jddd }|jddd }|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd}|jddd} |jddd} |jdddY}X|jdddZ}Y|jddd[}Z|jddd\}[|jdddi}h|jdddj}i|jdddk}j|dk r|dk rm| | | t| | | | | | } n|dv ry| } n|dk r|t||} n|dv r|dk r||} n|dk r|*} n|dk rt|t|} no|dk r|dk r|z|z|{||} nZ|dk r|dk r|ې} nK|dv r|dk r |z|{|||}} |z|{|z||| } |z| |z|{||| } n|dk r|} n|dk r&|dk r$|} n|dk rC|dk r:|z|{|z||} n|dk rB|} n|dk rQ|dk rP|ߐ} n|d~k r_|dk r^|z} n|dk r|dk r|z|{|||}} |z|{|z||}|||}|{|}}||| } |z|}| | | } n|dk r|dk r|} n||dk r|dk r|z|z||} ni|dAk r|dk rt||d|} nQ|dk r|@} nH|dk rt||d|} n5|dtk r|dok r|r} n'|dk r|dk rd|{|{|z} n|d|dtdd }i}|| jjk r\t| j | |d!t |<|d7}|| jjk s7| jjdk rw| j |d"}| |fS| jjdk r| j |d"|d#}| |fS($a! Calculate the statistic from the data from the read in MET .stat file(s) Args: data_df - dataframe containing the model(s) information from the MET .stat files line_type - MET line type (string) stat - statistic to calculate (string) Returns: stat_df - dataframe of the statistic stat_array - array of the statistic r/Nr2r3r4r5r6r0r7r8r9r:r;CNTFBAR_NCLFBAR_NCUFBAR_BCLFBAR_BCUFSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCUOBAR_NCLOBAR_NCUOBAR_BCLOBAR_BCUOSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL OSTDEV_BCUPR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCUSP_CORRKT_CORRRANKS FRANKS_TIES ORANKS_TIESMEME_NCLME_NCUME_BCLME_BCUESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL ESTDEV_BCUMBIAS MBIAS_BCL MBIAS_BCUMAEMAE_BCLMAE_BCUMSEMSE_BCLMSE_BCUBCRMSE BCRMSE_BCL BCRMSE_BCURMSERMSE_BCLRMSE_BCUE10E10_BCLE10_BCUE25E25_BCLE25_BCUE50E50_BCLE50_BCUE75E75_BCLE75_BCUE90E90_BCLE90_BCUIQRIQR_BCLIQR_BCUMADMAD_BCLMAD_BCU ANOM_CORR_NCL ANOM_CORR_NCU ANOM_CORR_BCL ANOM_CORR_BCUME2ME2_BCLME2_BCUMSESS MSESS_BCL MSESS_BCURMSFA RMSFA_BCL RMSFA_BCURMSOA RMSOA_BCL RMSOA_BCUANOM_CORR_UNCNTRANOM_CORR_UNCNTR_BCLANOM_CORR_UNCNTR_BCUSISI_BCLSI_BCUZGRADFGBAROGBARMGBAREGBARS1S1_OG FGOG_RATIODXDYZFHOF_RATEH_RATEO_RATE)CTCZNBRCTCFY_OYFY_ONFN_OYFN_ONrEC_VALUE)CTSZNBRCTSBASER BASER_NCL BASER_NCU BASER_BCL BASER_BCUFMEAN FMEAN_NCL FMEAN_NCU FMEAN_BCL FMEAN_BCUACCACC_NCLACC_NCUACC_BCLACC_BCUFBIAS FBIAS_BCL FBIAS_BCUPODYPODY_NCLPODY_NCUPODY_BCLPODY_BCUPODNPODN_NCLPODN_NCUPODN_BCLPODN_BCUPOFDPOFD_NCLPOFD_NCUPOFD_BCLPOFD_BCUFARFAR_NCLFAR_NCUFAR_BCLFAR_BCUCSICSI_NCLCSI_NCUCSI_BCLCSI_BCUGSSGSS_BCLGSS_BCUHKHK_NCLHK_NCUHK_BCLHK_BCUHSSHSS_BCLHSS_BCUODDSODDS_NCLODDS_NCUODDS_BCLODDS_BCULODDS LODDS_NCL LODDS_NCU LODDS_BCL LODDS_BCUORSSORSS_NCLORSS_NCUORSS_BCLORSS_BCUEDSEDS_NCLEDS_NCUEDS_BCLEDS_BCUSEDSSEDS_NCLSEDS_NCUSEDS_BCLSEDS_BCUEDIEDI_NCLEDI_NCUEDI_BCLEDI_BCUSEDISEDI_NCLSEDI_NCUSEDI_BCLSEDI_BCUBAGSS BAGSS_BCL BAGSS_BCUrr F1_O1ZNBRCNTFBSFBS_BCLFBS_BCUFSSFSS_BCLFSS_BCUAFSSAFSS_BCLAFSS_BCUUFSSUFSS_BCLUFSS_BCU F_RATE_BCL F_RATE_BCU O_RATE_BCL O_RATE_BCUZVL1L2UFBARVFBARUOBARVOBARUVFOBARUVFFBARUVOOBARZVAL1L2UFABARVFABARUOABARVOABARUVFOABARUVFFABARUVOOABAR FA_SPEED_BAR OA_SPEED_BARVCNTFS_RMSOS_RMSMSVERMSVEFDIRZODIR FBAR_SPEED OBAR_SPEED VDIFF_SPEED VDIFF_DIR SPEED_ERR SPEED_ABSERRDIR_ERR DIR_ABSERR ANOM_CORR)rBr3)ZBIASr_)ZETSrZPODr\ZSRATIOr!z FATAL ERROR, z IS NOT AN OPTIONrr)Zindex0Zindex1)r?rpsqrtrrrr)ZnlevelsroZget_level_valuesr"rvaluesZreshape(rZdata_dfrstatr2r3r4r5r6r7r8r9r:r;rCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryrzr{r|r}r~rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r4r5r6r7r8ZORDIRr9r:r;r<r=r>r?r@rAZstat_dfr+CZCAZCBZidxZidx_dictZ stat_arrayrrr calculate_stat>s"                              "                            rFc Cstj}|dkrtj|}|S|dkrV|jsTt|j dddftj tj|j dddf}t ||j ddddf dg|||\}}|d}|S| |dd|S) ao! Calculate average of dataset Args: logger - logger object average_method - method to use to calculate the average (string: mean, aggregation) line_type - line type to calculate stat from stat - statistic to calculate (string) df - dataframe of values Returns: meanZ aggregationNr+sumrz not recognized...z(use mean, or aggregation...returning NaN)rpr=rZmasked_invalidrGZisnullrCrQror?Z count_maskedrFZaggr) rZaverage_methodrrDZdfZ average_valueZndaysZavg_dfZ avg_arrayrrr calculate_averages( " rI)B__doc__r#ruZnumpyrprrHrZnetCDF4rrZpandasr<rrRrrrr'r)r.rDrFrKrUrXrYrrrrrrrrrrrrr rrrr-rarrrrrrrrrrrrrrrrr rrArFrIrrrr s  ED47(a449:U  "@&%zI%6/