U g@stddlZddlZddlZddlZddlZddlZddlZddZddZ ddZ dd Z d d Z d d Z ddZdS)NcCs(t|}|j}t|gtj}t|gtj} t|gtj} td|D]} || } t|| k} t| ddkrd|| |_t qHt| d|| <t|| k||k@}t|d| | <t| | t|| d| | <| | dkrHd|| dd|j d|_d |_ t qHd |_ | |fS) z Function designed to calculate the percent of each RFC region that contains missing data from the Stage IV dataset being evaluated. Function will return an array with a length equal to the number of RFC regions. rz8RFC Mask contains no valid pixel cells for RFC region: gY@?z7Found greater than 1.0 percent missing data from RFC: z for accumulation z timestep: z%Y-%m-%d %H:00:00FT) lengribNdvnpemptyintfloatrangewhereerrMsg ExceptiondEndAccstrftimeemcAccOk) statusMetagridMetaZst4GridrfcMskrfcValsrfcNamesZ numRegionsZndvCheckZ regionCountZ noDataCount noDataPercentZrfcZ rfcValTmpZindRfcZindNdvr>/lfs/h1/ops/prod/packages/nwm.v3.0.13/ush/mpe_refine/mpeMod.pypercentMissingInRfcs>   rc Cstt|jdt|j}|j}td|D]&}|jdkrV|jd|dd}n4|jd|dd |d d |dd}t j |sd |d d|_ zt |Wnt |YnX|tj|jdd}q,d||_ zt |Wnt |YnX|jd d|dd}|jd d|dd}zt j |rht |WnBd||_zt |Wnt |YnXtYnXzt j |rt |WnBd||_zt |Wnt |YnXtYnXzt||WnJd|d||_zt |Wnt |YnXtYnXz>t|d(} t|d} t| | W5QRXW5QRXWnBd||_zt |Wnt |YnXtYnXzt j |rt |WnBd||_zt |Wnt |YnXtYnX|sz6t}|jdkrx|||dn|||dWn8zt |Wnt |YnXtYnXt|tdkrt !|j"|j#gt j$}d|ddddf<t !|j"|j#gt j$} |j%| ddddf<zF|jdkrTt&|||ddddd| nt&|||dd ddd| Wn8zt |Wnt |YnXtYnXz4t '| d!k} t '| d"k} |j%| | <|j%| | <WnBd#||_zt |Wnt |YnXtYnX|dkrPt '| |j(k| |j%k@}d$||<nt '||j%k}|j%| |<d||<t '| |j%k}d||<t '| |j(k|d$k@}d||<t '||j%k| |j%k@|d$k@| |j(k@}d$||<|tj|jdd}zt j |rt |Wq,d||_zt |Wnt |YnXtYq,Xq,d} d} d} d}d}d}d}d}t|tdkrd%|_ zt |Wnt |YnXd&|_)nd'|_)|||fS)(a8 Function to loop over the RQI step time in minutes specified by the user and open MRMS GC files. Each file will be checked for their RQI values and checked against the user-specified RQI threshold. This grid will be upated as files are opened with a final mask being returned to the user. gN@rz+/RadarQualityIndex/RadarQualityIndex_00.00_ %Y%m%d-%H%M 00.grib2.gzz/RadarQualityIndex/%Y/%mz/MRMS_RadarQualityIndex_00.00_zExpected MRMS file: z not found. z Moving to next MRMS timestep.secondszRQI file being used: zMRMS_RadarQualityIndex_00.00_00.grib2Unable to remove old file: &Unable to remove temporary GRIB file: Unable to copy:  to: rbwbUnable to unzip: ZRadarQualityIndex0 m above mean sea levelNZ%RadarQualityIndex_0mabovemeansealevelunknownZvar209_8_0_0mabovemeansealevelggz/Unable to calculate missing data coverage for: rzONo MRMS Files found in accumulation period. Will not use MRMS for disagregationFT)*rr accDuration rqiStepMindBegAccr ncepWcossmrmsDirrospathisfile statusMsgerrMod logWarning errOutScreendatetime timedeltalogMsgtmpDirremover r shutilcopygzipopen copyfileobjgribMod2gribMetareadMetatyperrnynxfloat64rgetGribVariabler rqiThresh useRqiMsk)rr mrmsOkGridZ numMrmsStepsdMrmsmrmsStep mrmsFileCheck mrmsFileTmpGz mrmsFileTmpfTmpGzfTmpZ mrmsTmpGridZ indNoCoverageZ indNoDataZ indContinueZ indRemove1Z indRemove2Z indRemove3rrr genRqiMask;sl           $               rTcCs |s:|jdzt|Wnt|YnXt|j}t|j|j t |j gtj }t|j|j gtj }t|j|j gtj }|j |ddddddf<|j |ddddf<tdt |j D]}|tjdd}|jdkr|jd|dd |d d } n4|jd |d d |dd|d d} tj| sd| d|_zt|Wnt|YnXtd| |_zt|Wnt|YnX|jdkrJ|jd d|d d } zt| | WnJd| d| |_zt|Wnt|YnXtYnXn|jd d|d d} |jd d|d d} |jd d|d d } ztj| rt| WnBd| |_zt|Wnt|YnXtYnXztj| rt| WnBd| |_zt|Wnt|YnXtYnXzt| | WnJd| d| |_zt|Wnt|YnXtYnXz>t| d(} t| d}t| |W5QRXW5QRXWnBd| |_zt|Wnt|YnXtYnXztj| rVt| WnBd| |_zt|Wnt|YnXtYnXzt !|| | Wn8zt|Wnt|YnXtYnXzt| WnBd| |_zt|Wnt|YnXtYnXtj| sd| d|_zt|Wnt|YnXtz0t "|| |d d!d"dd#|dddd|f Wn8zt|Wnt|YnXtYnXz|j ||j#|j$|f<Wn>d$|_zt|Wnt|YnXtYnX|dkrv|dddd|f|ddddf<nt%||j k|dddd|f|j kB}t%||j k|dddd|f|j k@}|j ||<|dddd|f}||||||<z|dkr(d|ddddf<n|dkr |dddd|f|dddd|df}t&|}t%|dddd|f|j k|dddd|df|j k@|dddd|fd%k@|dddd|dfd%k@||j k@||j'k@|d%k@}d||<n|dddd|f|dddd|df}t&|}t%|dddd|f|j k|dddd|df|j k@|dddd|fd%k@|dddd|dfd%k@|d%k@}||d||<Wnd}YnXd}d}d}d}d}d}ztj|  rt| Wqd| |_zt|Wnt|YnXtYqXqt%|t |j dk}d||<t%|t |j dk}d||<d&|_(|||||fS)'zw This is a function to read in hourly Stage IV data over the course of the accumulation duration period. z3Grid metadata for Stage IV is unexpectedly missing.Nr @rrz/pcpanl.%Y%m%dz /st4_conus.z%Y%m%d%Hz .01h.grb2rrz/ST4.z.01h.gzzExpected Stage IV file: z? not found. Hourly Stage IV will not be used in disaggregation.zStageIV file being used: z st4_conus.zERROR: Unable to copy: r%zST4.z.01h.grbr"r#r&r'r(z!Unable to remove temporary file: zExpected file: z not found.APCP APCP_surfacesurfacezkg/m^2zeUnable to assign Channel Islands to missing for: Hourly Stage IV will not be used in dissagregation.r*T))r4r5r6r7r r.rrrFrGrr,rHrr r8r9r/st4Dirrr1r2r3r:r;r=r>r r<r?r@rArBgribOneToGribTworICIColCIRowr absolutethreshMM haveHourlyMPE)rrZst4AccDurationgeoConstZdSt4 st4HrlyPrecipst4HrlyTotalPrecip st4FlatIndZst4StepZ st4FileCheckZ st4FileTmp2Z st4FileTmpGzZ st4FileTmprRrS indNdvTmp indValidTmpZst4TmpZdiffTmpindTmprrrreadHourlyStageIVs          $          (,,   0  0   rhcCsv|j}d}tdt|jD]6}|tjdd}|jdkrT|jd|dd}n4|jd |d d |d d |dd}t j |sd|d|_ zt |Wnt |YnXtd||_ zt |Wnt |YnX|jdkrD|jd d|dd}|jd d|dd}n8|jd d|dd}|jd d|dd}zt j |rt |WnBd||_zt |Wnt |YnXtYnXzt j |rt |WnBd||_zt |Wnt |YnXtYnXzt||WnJd|d||_zt |Wnt |YnXtYnXz>t|d(} t|d} t| | W5QRXW5QRXWnBd||_zt |Wnt |YnXtYnXzt j |r4t |WnBd||_zt |Wnt |YnXtYnX|sz6t}|jdkr|||dn|||dWn8zt |Wnt |YnXtYnXt|tdkrht|j |j!t|jgtj"}t|j |j!gtj"} |j#|ddddddf<|j#| ddddf<zj|jdkrt$|||ddddd|dddd|f n,t$|||ddddd|dddd|f Wn8zt |Wnt |YnXtYnXt%|d k} |j#|| <|dkrV|dddd|f| ddddf<nt%| |j#k|dddd|f|j#kB} t%| |j#k|dddd|f|j#k@}|j#| | <|dddd|f}| |||| |<zt j |rt |WnBd||_zt |Wnt |YnXtYnX|tjdd}qd} d} d}d}d!|_&||| |fS)"z This is a function to read in hourly MRMS Gage-Corrected data over the course of the accumulation duration period. NrrUrrz5/MultiSensorQPE/MRMS_MultiSensor_QPE_01H_Pass2_00.00_rrz/GaugeCorr_QPE_01H/rrrz/MRMS_GaugeCorr_QPE_01H_00.00_z#Expected MRMS MultiSensorQPE file: z> not found. Hourly MRMS GC will not be used in disaggregation.z MultiSensorQPE file being used: z%MRMS_MultiSensor_QPE_01H_Pass2_00.00_r!zMRMS_GaugeCorr_QPE_01H_00.00_r"r#r$r%r&r'r(ZMultiSensorQPE01Hr)Z%MultiSensorQPE01H_0mabovemeansealevelr+Zvar209_6_9_0mabovemeansealevelr*T)'r.r rr,r8r9r/r0rr1r2r3r4r5r6r7r r:r;r<r r=r>r?r@rArBrCrDrErrrFrGrHrrIr useMrmsGc)rrrMmrmsHrlyPreciprNZ dMrmsNextrOrPrQrRrSmrmsHrlyTotalPrecipZ indFilterrerfZmrmsTmprrrreadHourlyMrmsGc'sV          $       .0   (     rlc Cs|j}d}|jdkr|}n6|jdkr6|tjdd}n|jdkrP|tjdd}tdt|jD]8}t|jd}|j dkr|j d | d d | d d |d}nX|j d| dd| dd| dd | d d| d d |d}t j |sPd|d|_zt|Wnt|YnXtd||_zt|Wnt|YnX|szt}|||dWn8zt|Wnt|YnXtYnXt|tdkrRt|j|jt|jgtj}t|j|jgtj}|j|ddddddf<|j|ddddf<z0t|||ddddd|dddd|f Wn8zt|Wnt|YnXtYnX|dkr|dddd|f|ddddf<nt ||jk|dddd|f|jkB} t ||jk|dddd|f|jk@} |j|| <|dddd|f} || | | || <|tjdd}|tjdd}q`d} d} d} d|_!||||fS)z This is a function to read in hourly HRRR forecasted precipitation data over the course of the accumulation duration period. NrrUr @rz/hrrr.rVz /conus/hrrr.t%Hz z.wrfsfcf.grib2rrr%d.tzWARNING: Expected HRRR file: z; not found. Hourly HRRR will not be used in disaggregation.zHRRR file being used: rWrXrYkg m**-2T)"r. hrrrFcstHrr8r9r rr,strzfillr/hrrrDirrr1r2r3r4r5r6r7r r:rBrCrDrErrrFrGrHrrIr useHrrr) rrZdHrrrhrrrHrlyPrecipZ dHrrrCycleZhrrrStep fcstHrStrZhrrrFilehrrrHrlyTotalPreciprerfZhrrrTmprrrreadHourlyHrrrPrecips        (,, r~c Cs|j}d}|jdkr|}n6|jdkr6|tjdd}n|jdkrP|tjdd}tdt|jD]8}t|jd}|j dkr|j d | d d | d d |d}nX|j d| dd| dd| dd | d d| d d |d}t j |sPd|d|_zt|Wnt|YnXtd||_zt|Wnt|YnX|szt}|||dWn8zt|Wnt|YnXtYnXt|tdkrRt|j|jt|jgtj}t|j|jgtj}|j|ddddddf<|j|ddddf<z0t|||ddddd|dddd|f Wn8zt|Wnt|YnXtYnX|dkr|dddd|f|ddddf<nt ||jk|dddd|f|jkB} t ||jk|dddd|f|jk@} |j|| <|dddd|f} || | | || <|tjdd}|tjdd}q`d} d} d} d|_!||||fS)z This is a function to read in hourly RAP forecasted precipitation data over the course of the accumulation duration period. NrrmrUrrnrorz/rap.rVz/rap.trpz z.awp130pgrbfrqrrrrrrszExpected RAP file: z: not found. Hourly RAP will not be used in disaggregation.zRAP file being used: rWrXrYrtruT)"r.rvr8r9r rr,rwrxr/rapDirrr1r2r3r4r5r6r7r r:rBrCrDrErrrFrGrHrrIr useRap) rrZdRap rapHrlyPrecipZ dRapCycleZrapStepr|ZrapFilerapHrlyTotalPreciprerfZrapTmprrrreadHourlyRapPrecipOs        (,, rc&Cs>|js^d|_zt|Wnt|YnXt|j|jgtj}|j |ddddf<t |j ddkr|j D]b}d|d|_zt |Wnt|YnX|t ||k}t ||k}|j ||<qvt ||j k||j k@}t |d}dt||_zt |Wnt|YnXt ||j k||j k@}t |d}dt||_zt |Wnt|YnX|jrt |dk}t |d}d t||_zt |Wnt|YnXt|j|jgtj}d|ddddf<|dkr d||<|jr:|dkr:d||<t |dk}t |d}d |_zt |Wnt|YnX|} |jr|jrt |dk|dk@| | j k@| d k@}t |ddkrHtd|jD]d}!| dddd|!f}"|dddd|!f}#|||#|| ||"|<|"| dddd|!f<qd ||<d tt |dd|_zt |Wnt|YnX|jr|t |dk| | j k@| d k@}t |ddkr}!| dddd|!f}"d|"|<| dddd|!f|"kqҐqnd}d}d}d}d}d}"d}#d}$d}%| S)z function that will combine/subdivide Stage IV / MRMS / HRRR / RAP where needed to produce a final MPE grid of precipitation values. z*NO HOURLY MPE FOUND, SETTING HOURLY TO NDVNrzOMITTING RFC: z FROM ACCUMULATED DISAGREGATIONzNum Missing Cells = zNum Good Cells = rzNum Flat Cells = z;Subdividing ST4/MRMS/HRRR/RAP to generate hourly MPE grids.r*rnzSUBDIVIDED WITH MRMS FOR: z CELLSzSUBDIVIDED WITH HRRR FOR: zSUBDIVIDED WITH RAP FOR: zSUBDIVIDED FLAT FOR: iz FROM MPE GENERATION)r`r4r5r6r7rrrFfloat32rrrfcsExcludeHrlyr:r rwflatLinerGrrirKr r,rzr rfcsMskOut)&r nwmGeoMeta st4GridMeta st4AccGridrcrbrdrLrjrk mrmsGridMetar{r} hrrrGridMetarr rapGridMetarr rfcMskGridZrfcTmpZ rfcValOmmitZ indRfcOmmitZ missingIndZ missingCountZindGoodZ goodDataCountZindFlatZ flatCountZ subdivideFlagrgZsubdivideCountZ hourlyOWPGridZhrTmpZ tempOWPGridZtempMRMSGCGridZ tempHRRRGridZ tempRAPGridrrr combinePrecips&             "r)numpyrr8r1rBr=r?r5rrTrhrlr~rrrrrr s")c ?jj