U g@sddlZddlZddlZddlmZddlZddlZddlZddl Z ddl Z ddZ GdddZ ddZ d d Zd d Zd dZdS)N)DatasetcCsBtj|js$d|jd|_ttj|jsHd|jd|_ttj|jsld|jd|_ttj|jsd|jd|_ttj|j sd|j d|_ttj |j sd|j d|_ttj |j sd|j d|_ttj |j s"d|j d|_ttj |jsHd|jd|_ttj |jsnd|jd|_ttj |jsd|jd|_t|jdkrd|_t|j|jdkrd|_t|jdkrd|_t|jd krd |_tt|j|j|_|jd kr$d |_t|jd krd|_tdS)z Generic function to do a routine check of static metadata for the MPE workflow that will be ran each time the program is initialized. This is sanity checking to ensure expected directories, files, etc, are present. zERROR: Expected directory: not found.zERROR: Expected file: z9ERROR: Number of hours to process must be greater than 0.rzMERROR: Number of hours going back must be divisible by accDuration specified.z@ERROR: Must have a non-negative accumulation duration specified.zPERROR: Currently, only accumulation durations of 0, 6, 12, and 24 are supported.z-ERROR: RQI threshold must be greater than 0.0g?z6ERROR: RQI threshold must be less than or equal to 1.0gN@z\ERROR: RQI Steps in Minutes must be an equal divisor of the accumulation period being used.z3ERROR: RQI Steps in minutes must be greater than 0.zYERROR: RQI Steps in Minutes must not be greater than the accumulation period being used.z6ERROR: RQI Steps in minutes must be a an even numbers.z?ERROR: Invalid HRRR forecast hour chosen. Only 1,2,3 available.g@@zTERROR: Please specify a minimum precip threshold for dividing between 0 and 1000 mm.N)ospathisdiroutDirerrMsg ExceptiontmpDirmrmsDirhrrrDirrapDirisfilegeoFile landMetaFile hrrrWghtPath rapWghtPath mrmsWghtPathhrapNwmWghtPath numHoursBack accDurationint nLoopSteps rqiThresh rqiStepMinrfcMsk hrrrFcstHrfloatthreshMM) statusMetar%=/lfs/h1/ops/prod/packages/nwm.v3.0.13/ush/mpe_refine/ioMod.pycheckStaticMetas        r'c@seZdZddZdS) geoConstantsc>Csd|_ddddddddddddd dd dd dd d dd d dd dd d d ddddddddddd d d ddddddddd d ddddddddddg>|_dddddddd d!d"d#d$d$d$d%d%d%d&d&d'd'd(d(d(d)d)d*d*d*d*d*d*d+d+d+d+d+d+d+d+d+d+d+d+d,d,d,d,d,d,d,d,d-d-d-d-d-d.d/d/d0d0g>|_d1S)2zt Initialize object to hold geospatial constants used for processing, regridding and output. TUSRQYWXV?=>@AB45678;<301/iiiiiiii!i"i#i$i%i&i'i;i<i=i>i?i@iAN)Z noDataValueCIColCIRow)selfr%r%r&__init__szgeoConstants.__init__N)__name__ __module__ __qualname__rGr%r%r%r&r(sr(c Cs|jdkrN|jd|jdd|jddtt|jdd}nR|jd |jd d |jdd |jddtt|jdd }tj |sd tt|jdd|jd|_ t |jdkrX|j d d|jddtt|jdd}zt||Wn$d|d||_ t YnXn4|j d d|jddtt|jdd }|j d d|jddtt|jdd}|j d d|jddtt|jdd}ztj |rt|Wnd||_ t YnXztj |rNt|Wnd||_ t YnXzt||Wn$d|d||_ t YnXz>t|d(}t|d}t||W5QRXW5QRXWnd||_ t YnXztj |rt|Wnd||_ t YnXzt|||Wnt YnXzt|Wnd||_ t YnXtj |sd|d|_ t |szt}|||dWnt YnXt|j|jgtj}|j|ddddf<zt|||dddd d!| Wnt YnXd"|_ztj |r`t|Wnd||_ t YnX||fS)#z Function to read in STAGE IV data that has been accumulated over the desired accumulation period. This is not to be confused with the hourly STAGE IV files that will be read in for modification by the OWP MPE process. rz/pcpanl.z%Y%m%dz /st4_conus.%Y%m%d%H.rzh.grb2/z%Yz/ST4.zh.gzzMISSING zH ACC ST4 FOR z st4_conus.zUnable to copy:  to: zST4.zh.grbzUnable to remove old file: z&Unable to remove temporary GRIB file: rbwbzUnable to unzip: z!Unable to remove temporary file: zExpected file: rZAPCPNZ APCP_surfaceZsurfacerzkg m**-2T) ncepWcossst4DirdEndAccstrftimestrrrzfillr r rr rrshutilcopyremovegzipopen copyfileobjgribMod2ZgribOneToGribTwoZgribMetareadMetanpemptynynxfloat64gribNdvZgetGribVariable haveAccData) gridMetar$Zst4PathZ st4PathTmp2Z st4PathTmpGzZ st4PathTmpZfTmpGzZfTmp st4AccGridr%r%r& getStage4Accs.                 $           rhcCszt|jd}Wnd|j|_tYnXt|j|jdjkrpdt|jdtt dj|_tt|j |jdjkrdt|j dtt dj|_t|jd jd krd t|jd jd |_tz|j dd d }Wnd|j|_tYnXz|j dd d }Wnd|j|_tYnXz|j dd d }Wnd|j|_tYnXz|j dd d }Wnd|j|_tYnXz |j dd d d d fj }Wnd|j|_tYnXz |j dd d d d f|_ Wnd|j|_tYnXz |j dd d d d f|_Wnd|j|_tYnXd|_||||||fS)z Function that will read in the NetCDF file containing a grid on HRAP where each pixel cell contains an integer value corresponding to a unique RFC region. rzUnable to open RFC mask file: xz#Stage IV GRIB data contains NX of: z Expected NX is: yz#Stage IV GRIB data contains NY of: z Expected NY is: ZRFC zRFC Mask file contains z when expected size is 12.Nz%Unable to obtain X coordinates from: z%Unable to obtain Y coordinates from: ZRFC_grid_valuez&Unable to obtain RFC_grid_value from: ZRFC_namez Unable to obtain RFC_name from: ZDataz/Unable to obtain the mask of RFC regions from: Zlatz4Unable to extract grid center latitude values from: Zlonz5Unable to extract grid center longitude values from: T)rr r rrrb dimensionssizerUidra variablesdataZgridCenterLatsZgridCenterLons haveRfcMsk)rfr$ZmskFilexCoordsyCoordsrfcValsrfcNames rfcMskGridr%r%r& getRfcMsk(sz                     rxcCs$tj|r d|d|_tzt|d}Wnd||_tYnXz|d|jdWnd||_tYnXz|d|jd Wnd ||_tYnXt|jd krz|d |jd Wnd||_tYnXt|jd kr6z| dddWnd||_tYnXt|jd krvz| dddWnd||_tYnXt|jd kr||j dddddf<t|jd krt d|jd D]4}|dddd|f|j d|ddddf<qz | Wnd||_tYnXdS)z1 Generic function to output NetCDF file. z Output file: z already existswzUnable to create output file: rkrz!Unable to create X dimension in: rjrz!Unable to create Y dimension in: rtimerz$Unable to create time dimension in: ZvarOutf8)rkrjz%Unable to create output variable in: rzrkrjNzUnable to close file: ) r r rr rrcreateDimensionshapelencreateVariablerprangeclose)r$ZinGridZoutFileZidOutZzTmpr%r%r& outputNetCDFys\             2  rc Cs |jd|jdd}|jd|jdd}tdddd}|j|}t|jdt|jd }ztj |rt |Wn>d ||_ zt |Wnt jYnXtYnXtj |r"zt||Wn>d ||_ zt |Wnt jYnXtYnXtj |rzt|d }WnBd |d|_ zt |Wnt jYnXtYnXt|dk|dk@} t| ddkrd|_ t|| d || <z&t||jddddddf<Wn>d||_ zt |Wnt jYnXtYnXz |Wn>d||_ zt |Wnt jYnXtYnXzt||WnFd |d||_ zt |Wnt jYnXtYnX n&zt|d}WnBd |d|_ zt |Wnt jYnXtYnXz|d|jWn>d||_ zt |Wnt jYnXtYnXz|d|jWn>d||_ zt |Wnt jYnXtYnXz|ddWn>d||_ zt |Wnt jYnXtYnXz|dddWn>d ||_ zt |Wnt jYnXtYnXz|dddWn>d!||_ zt |Wnt jYnXtYnXz|dddWn>d"||_ zt |Wnt jYnXtYnXz|d#d$Wn>d%||_ zt |Wnt jYnXtYnXz|jddd&d'd(Wn>d)||_ zt |Wnt jYnXtYnXz|j|jd#_Wn>d*||_ zt |Wnt jYnXtYnXz|j |jd#_!Wn>d+||_ zt |Wnt jYnXtYnXz|j"|jd#_#Wn>d,||_ zt |Wnt jYnXtYnXz|j$|jd#_%Wn>d-||_ zt |Wnt jYnXtYnXz|j&|jd#_'Wn>d.||_ zt |Wnt jYnXtYnXz|j(|jd#_)Wn>d/||_ zt |Wnt jYnXtYnXz|j*|jd#_+Wn>d0||_ zt |Wnt jYnXtYnXz|j,|jd#_-Wn>d1||_ zt |Wnt jYnXtYnXz|j.|jd#_/Wn>d2||_ zt |Wnt jYnXtYnXz|j0|jd#_1Wn>d3||_ zt |Wnt jYnXtYnXz|j2|jd#_3Wn>d4||_ zt |Wnt jYnXtYnXz |j4|_4Wn>d5||_ zt |Wnt jYnXtYnXz |j5|_5Wn>d6||_ zt |Wnt jYnXtYnXz8|j6|jd_7|j8|jd_7d7|jd_7d|jd_7Wn>d8||_ zt |Wnt jYnXtYnXz8|j9|jd_:|j;|jd_:d9|jd_:d:|jd_:Wn>d;||_ zt |Wnt jYnXtYnXz8|j<|jd_=|j>|jd_=d<|jd_=d=|jd_=Wn>d>||_ zt |Wnt jYnXtYnXz.|j?|jd_@|jA|jd_@|j?|jd_@Wn>d?||_ zt |Wnt jYnXtYnXt|dk|dk@} t| ddk rd|_ t|| d || <z&t||jddddddf<Wn>d||_ zt |Wnt jYnXtYnXz|jB|jddd<Wn>d@||_ zt |Wnt jYnXtYnXz|jC|jddd<Wn>dA||_ zt |Wnt jYnXtYnXz||jdd<Wn>dB||_ zt |Wnt jYnXtYnXz |Wn>dC||_ zt |Wnt jYnXtYnXzt||WnFd |d||_ zt |Wnt jYnXtYnXdS)Dz^ Function to output MPE grids on the 1km NWM grid to CF-compliant NetCDF files. rMrKz00.PRECIP_FORCING.nc.tmpz00.PRECIP_FORCING.ncirrig @zFailure to remove: zFailure to move: azFailure to open: z for editingriz5No nonzero precipitation values found in output grid.ZRAINRATENz6Failure to place MPE grid into variable RAINRATE for: zFailure to close: rNryz for MPE creation.rjz!Failre to create x dimension in: rkz"Failure to create y dimension in: rzz%Failure to create time dimension in: r{z!Failure to create x variable in: z!Failure to create y variable in: z$Failure to create time variable in: Zcrscz#Failure to create crs variable in: r|r)) fill_valuez(Failure to create RAINRATE variable in: zCFailure to place Proj attribute: CoordinateTranformType into file: z;Failure to place Proj attribute: transform_name into file: z>Failure to place Proj attribute: grid_mapping_name into file: z;Failure to place Proj attribute: CoordinateAxes into file: z;Failure to place Proj attribute: esri_pe_string into file: z>Failure to place Proj attribute: standard_parallel into file: zJFailure to place Proj attribute: longitude_of_central_meridian into file: zJFailure to place Proj attribute: latitude_of_projection_origin into file: z:Failure to place Proj attribute: false_easting into file: z;Failure to place Proj attribute: false_northing into file: z9Failure to place Proj attribute: earth_radius into file: z2Failure to place Proj attribute: proj4 into file: z4Failure to place global attribute conventions into: Zprecipitation_ratez0Failure to place standard name attributes into: zPrecipitation Ratez$Valid date of MPE precipitation ratez,Failure to place long name attributes into: zmm s-1zMinutes since EPOCHz(Failure to place units attributes into: z-Failure to place resolution attributes into: zFailure to place x array into: zFailure to place y array into: z"Failure to place time value into: zFailure to close NetCDF file: )DrdCurrrTr datetimerdayssecondsr r rrYr errModlogErr errOutScreenrrWmoverr_whererflipudrprr}rbrarZ transformTypeZ_CoordinateTransformTypeZ transformNameZtransform_nameZ gridMappingZgrid_mapping_nameZ coordAxisZ_CoordinateAxesZ esriStringZesri_pe_stringZ stdParallelZstandard_parallelZlon0Zlongitude_of_central_meridianZlat0Zlatitude_of_projection_originZ falseEastZ false_eastingZ falseNorthZfalse_northingZradiusZ earth_radiusZproj4Z conventionsZxStdNameZ standard_nameZyStdNameZ xLongNameZ long_nameZ yLongNameZxUnitsunitsZyUnitsZxRes resolutionZyResrsrt) r$ nwmGeoMetaZoutGridZ outPathTmpZoutPathZdEpochdtZnumMinSinceEpochZidTmpZindValidr%r%r&outputMpeNetCDFs          &                                                                                          &                 r)r r]numpyr_ZnetCDF4rrWrZsysrrr'r(rhrxrrr%r%r%r& s vQ;