satingest v4.7.3 RELEASED MAR 07 2024 v4.7.? IMPLEMENTED MAR 21 2024 PRELUDE The following products are updated: S-NPP/NOAA-20/NOAA-21 OMPS NP S-NPP/NOAA-21 VIIRS SST (on March 27, 2024, no filename changes) See OSPO note:https://www.ospo.noaa.gov/data/messages/2024/03/MSG_20240304_1730.html FILE CHANGES M docs/Release_Notes.txt M fix/idsb_ingest.source.html M fix/npp_omps.ompsnp_v8 M fix/poes_sst.viirs_clear_sky M fix/poes_sst.viirs_probclear_sky M versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/Release_Notes.txt Update contents to reflect this release FIX CHANGES fix/idsb_ingest.source.html Update file names in satingest table: https://www2.nco.ncep.noaa.gov/idsb/ingest/ NOTE: Data Flow should update subscriptions table: http://www2.pmb.ncep.noaa.gov/wiki/index.php/PDA_Subscription_Information fix/npp_omps.ompsnp_v8 Update product filename version number fix/poes_sst.viirs_clear_sky,fix/poes_sst.viirs_probclear_sky Cleaned old product filename VERSIONS CHANGES versions/VERSION Update version to reflect this release CHANGES TO OUTPUT AND FILE SIZES No new tanks are generated. BIG BROTHER INFORMATION START monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/OMPS-NP_v4r4_*.bufr STOP monitoring the following filename patterns: $COMROOT/logs/bb/ingest/OMPS-NP_v4r2_*.bufr CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Satingest works together with module bufr-tranjb (currently using v1.0.5) It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE:1.At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. 2.fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. 3.Directory /TRIGGERS is only used by EMC for testing 4.Directory /ecf is not part of the released code, b/c it is used only by NCO. It is added by the SPAs during implementation, see $PACKAGEROOT/satingest.v*.*.* PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.5 to be installed in order to run successfully. Module bufr-tranjb is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions B. If there is a NCO parallel run of satingest jobs, they must be stopped BEFORE the implementation, so that the prod run can pull the files from PDA OPS (and not run out of allowed PDA pulls per data type) C. NESDIS usually updates their polar instrument data products during "back orbits". That means they usually do it while the JPSS satellites are observing the other side of the CONUS. This is specific for JPSS to minimize the data flow impact. For example,usually: Date/Time of Initial Impact: 16:00 UTC Date/Time of Expected End: 18:30 UTC Tip: Watch for OSPO notice Updates, they provide Initial and End of Impact times. DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)Verify that subscribtions exists for the following products on PDA OPS: Short Name: OMPS_NP_V8_BUFR Full Name: OMPS-NP_v4r4_[npp,j01,n21]_s???????????????_e???????????????_c???????????????.bufr (version change only) Dest Dir: PDAFileLinks/NDE/OMPS Note: These data will appear on PDA OPS on 21th March 2024 4)Run JISATINGEST for the following satingest tasks (before and on implementation day) and verify the generation of the listed tanks: poes_sst - inspect tank b012/xx023, b012/xx024 npp_omps - inspect tank b008/xx017 5)Follow the PRE-IMPLEMENTATION TESTING instructioins above. DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.7.2 RELEASED FEB 20 2024 v4.7.2 IMPLEMENTED MAR 06 2024 PRELUDE The following products are updated: GMI-1CR radiances, S-NPP/NOAA-20/NOAA-21 VIIRS AMVs S-NPP/NOAA-20/NOAA-21 CrIS radiances PDA destination directories are corrected MODIS AMVs were turned off FILE CHANGES M docs/Release_Notes.txt M fix/CMakeLists.txt M fix/idsb_ingest.source.html M fix/npp_cris_431.cris_fsr_431 M fix/npp_omps.ompsnp_v8 M fix/satwnd D fix/satwnd.modir D fix/satwnd.modwv M fix/satwnd.nb_viirs M fix/ssmi.gmi_tmbr M versions/VERSION M versions/build.ver M versions/run.ver (M-modified; A-added; D-deleted) DOCS CHANGES docs/Release_Notes.txt Update contents to reflect this release FIX CHANGES fix/CMakeLists.txt Remove MODIS AMVs related fix files fix/idsb_ingest.source.html Update file names in satingest table: https://www2.nco.ncep.noaa.gov/idsb/ingest/ NOTE: Data Flow should update subscriptions table: http://www2.pmb.ncep.noaa.gov/wiki/index.php/PDA_Subscription_Information fix/npp_cris_431.cris_fsr_431 Correct destination directory fix/npp_omps.ompsnp_v8 Update product filename fix/satwnd Remove MODIS processing families fix/satwnd.nb_viirs Update product filename fix/ssmi.gmi_tmbr Update product filename VERSIONS CHANGES versions/VERSION Update version to reflect this release versions/build.ver,run.ver Update version of moduled bufr_tranjb/1.0.5 CHANGES TO OUTPUT AND FILE SIZES No new tanks are generated. BIG BROTHER INFORMATION START monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/NB-PAMV_v3r2_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-PAMV_v3r2_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-PAMV_v3r2_n21_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/GMI-1CR_v7r1_gpm_s???????????????_e???????????????_c???????????????.bufr STOP monitoring the following filename patterns: $COMROOT/logs/bb/ingest/NB-PAMV_v2r1_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-PAMV_v2r1_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/GMI-1CR_v7r0_gpm_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/satwnd.bufrcdft.aqua.D?????.T??:??:??Z $COMROOT/logs/bb/ingest/satwnd.bufrcdft.terra.D?????.T??:??:??Z $COMROOT/logs/bb/ingest/satwnd.bufrwvap.aqua.D?????.T??:??:??Z CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Satingest works together with module bufr-tranjb (currently using v1.0.4) It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS iBUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE:1.At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. 2.fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. 3.Directory /TRIGGERS is only used by EMC for testing 4.Directory /ecf is not part of the released code, b/c it is used only by NCO. It is added by the SPAs during implementation, see $PACKAGEROOT/satingest.v*.*.* PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.5 to be installed in order to run successfully. Module bufr-tranjb is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions B. If there is a NCO parallel run of satingest jobs, they must be stopped BEFORE the implementation, so that the prod run can pull the files from PDA OPS (and not run out of allowed PDA pulls per data type) C. NESDIS usually updates their polar instrument data products during "back orbits". That means they usually do it while the JPSS satellites are observing the other side of the CONUS. This is specific for JPSS to minimize the data flow impact. For example,usually: Date/Time of Initial Impact: 16:00 UTC Date/Time of Expected End: 18:30 UTC DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)Verify that subscribtions exists for the following products on PDA OPS: Short Name: VIIRS_WINDS_NB_BUFR Full Name: NB-PAMV_v3r2_[npp,j01,n21]_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/VIIRS/ Note: These data will appear on PDA OPS on 6th March 2024 Short Name: GPM_GMI_L1C-R_BUFR Full Name: GMI/GMI-1CR_v7r1_gpm_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/PULL/GMI Note: These data is already on PDA OPS, at the time of writing these RN 2)For the following subs, correct the Destination Directories Short name: CrIS_C0431_BUFR Instrument: j01 (choose only one satellite instrument) RegEx: NUCAPS-C0431_v\dr\d_(j01|npp|n21)_s\d{15}_e\d{15}_c\d{15}\.bufr Dest Dir: PDAFileLinks/NDE/NUCAPS/C0431/J01 Short name: CrIS_C0431_BUFR Instrument: n21 (choose only one satellite instrument) RegEx: NUCAPS-C0431_v\dr\d_(j01|npp|n21)_s\d{15}_e\d{15}_c\d{15}\.bufr Dest Dir: PDAFileLinks/NDE/NUCAPS/C0431/N21 3)Make the following sub: Short name: CrIS_C0431_BUFR Instrument: npp (choose only this satellite instrument!!!) RegEx: NUCAPS-C0431_v\dr\d_(j01|npp|n21)_s\d{15}_e\d{15}_c\d{15}\.bufr Dest Dir: PDAFileLinks/NDE/NUCAPS/C0431/SNPP Note: These data is already on PDA OPS, at the time of writing these RN 4)Run JISATINGEST for the following satingest tasks (before and on implementation day) and verify the generation of the listed tanks: satwnd - inspect tank b005/xx091 ssmi.gmi_tmbr - inspect tank b021/xx246 npp_cris_431 - inspect tank b021/xx206 5)Follow the PRE-IMPLEMENTATION TESTING instructioins above. DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.7.1 RELEASED JAN 16 2024 v4.7.? IMPLEMENTED JAN 31 2024 PRELUDE The following products are updated: S-NPP/NOAA-20/NOAA-21 CrIS radiances are augmented with VIIRS radiance cluster (VRC) Metop-1/3 IASI radiance filenames are updated Metop-1/3 GOME Ozone product is updated CIMSS AMV tank b005xx099 SAID assignemnt bug fixed FILE CHANGES M docs/Release_Notes.txt M fix/airs.iasi_efov M fix/bufrtab.008 M fix/hrsatwind.cimss M fix/idsb_ingest.source.html M fix/npp_cris_2211.cris_fsr_2211 M fix/npp_cris_431.cris_fsr_431 M fix/ozone_orbit.gome M sorc/bufr_trancimssamv.fd/trancimssamv.f90 M versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/Release_Notes.txt Update contents to reflect this release FIX CHANGES fix/airs.iasi_efov Introduce new filenames for METOP-1/3 IASI: NUCAPS-GMAO-616_v3r1_m0?_s???????????????_e???????????????_c???????????????.bufr fix/bufrtab.008 Update bufrtab.008 to match METOP-1/3 GOME Ozone product's BUFR table (b008/xx012 is now the same as b008/xx018) fix/hrsatwind.cimss Add omitted GOES-18 filenames for CIMSS Huricane/TC winds fix/idsb_ingest.source.html Update file names in satingest table: https://www2.nco.ncep.noaa.gov/idsb/ingest/ NOTE: Data Flow should update subscriptions table: http://www2.pmb.ncep.noaa.gov/wiki/index.php/PDA_Subscription_Information fix/npp_cris_2211.cris_fsr_2211 Update product filename version and destination directories (notice the new PDA subscriptions in PRE-IMPLEMENTATION TESTING REQUIREMENTS) fix/npp_cris_431.cris_fsr_431 Update product filename version and destination directories (notice the new PDA subscriptions in PRE-IMPLEMENTATION TESTING REQUIREMENTS) fix/ozone_orbit.gome Introduce new filenames for METOP-1/3 GOME Ozone: V8TOZ-EDR_v1r0_m0?_s???????????????_e???????????????_c???????????????.bufr SORC CHANGES sorc/bufr_trancimssamv.fd/trancimssamv.f90 Fix a big in SAID assignment VERSIONS CHANGES versions/VERSION Update version to reflect this release CHANGES TO OUTPUT AND FILE SIZES No new tanks are generated. The CrIS data tank, b021/xx206, will increase in size by ~5% as a number of mnemonics may be populated for some observations(only when cloud and/or cluster info exists) BIG BROTHER INFORMATION Start monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/NUCAPS-C0431_v4r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v4r0_n21_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v3r1_m01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v3r1_m03_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/V8TOZ-EDR_v1r0_m01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/V8TOZ-EDR_v1r0_m03_s???????????????_e???????????????_c???????????????.bufr Stop monitoring the following filename patterns: $COMROOT/logs/bb/ingest/NUCAPS-C0431_v2r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v3r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v3r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v3r0_n21_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C2211_v3r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C2211_v3r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C2211_v3r0_n21_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v3r0_m0?_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v2r0_m0?_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NPR_IARD.M0?_D?????_S??????_E??????_G???_C0616_U6 $COMROOT/logs/bb/ingest/GOME_M01_8TO3_??????????????Z_??????????????Z.BUFR CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Satingest works together with module bufr-tranjb (currently using v1.0.4) It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE:1.At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. 2.fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. 3.Directory /TRIGGERS is only used by EMC for testing 4.Directory /ecf is not part of the released code, b/c it is used only by NCO. It is added by the SPAs during implementation, see $PACKAGEROOT/satingest.v*.*.* PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.4 to be installed in order to run successfully. Module bufr-tranjb is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions B. If there is a NCO parallel run of satingest jobs, they must be stopped BEFORE the implementation, so that the prod run can pull the files from PDA OPS (and not run out of allowed PDA pulls per data type) DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)IMPORTANT On implementation day (Jan 31,2024), or sooner, stop the ingest of all satingest generated b008 tanks (i.e. b008/xx011,xx012,xx013,xx017,xx018,xx019) and use the apxdx tool to add the updated fix/bufrtab.008 table (from v4.7.1) to each satingest tank. /apps/ops/prod/nco/intel/19.1.3.304/bufr_util.v1.1.0/exec/apxd /apps/ops/prod/nco/intel/19.1.3.304/bufr_util.v1.1.0/ush/run_apxdx.sh Run apxdx tool with $APXDX_TYPE=satingest. Resume ingest for the tanks in question. NESDIS' tentative schedule is to start the CrIS and IASI product promotion ~10:00AM EST. The process will take approximately a couple of hours. Therefore the switch to satingest v4.7.1 could start ~11AM EST. 2)Make the following subscribtion(s) on PDA OPS a day or two before implementation day: A)For the following subs, first create the Destination Directories, then make the subs, making sure that only one satellite instrument is selected. This reduces num of files per dir. Short name: CrIS_C0431_BUFR Instrument: j01 (choose only one satellite instrument) RegEx: NUCAPS-C0431_v\dr\d_(j01|npp|n21)_s\d{15}_e\d{15}_c\d{15}\.bufr Dest Dir: PULL/NDE/NUCAPS/C0431/J01 (should have been: PDAFileLinks/NDE/NUCAPS/C0431/J01 Correct it in v4.7.2 !!!) Short name: CrIS_C0431_BUFR Instrument: n21 (choose only one satellite instrument) RegEx: NUCAPS-C0431_v\dr\d_(j01|npp|n21)_s\d{15}_e\d{15}_c\d{15}\.bufr Dest Dir: PULL/NDE/NUCAPS/C0431/N21 (should have been: PDAFileLinks/NDE/NUCAPS/C0431/N21 Correct it in v4.7.2 !!!) NOTE: After creating these new subscription, and you see the data flowing in on implementation day, Jan31 ~10am-Noon EST, please remove the old subs for CrIS_C0431_BUFR (so that files are not pulled multiple times) B)For the following subs, the Destination Directories already exists. Short name: GOME-TO-L2_BUFR RegEx: V8TOZ-EDR_v\dr\d_m(01|03)_s\d{15}_e\d{15}_c\d{15}.bufr Dest Dir: PDAFileLinks/PULL/OZONE Short name: HEAP_IASI_GMAO_BUFR RegEx: NUCAPS-GMAO-616_v\dr\d_(m01|m03)_s\d{15}_e\d{15}_c\d{15}\.bufr Dest Dir: PDAFileLinks/PULL/IASI 3)Run JISATINGEST for the following satingest tasks (before and on implementation day) and verify the generation of the listed tanks: npp_cris_431 - inspect tank b021/xx206 airs - inspect tank b021/xx241 hrsatwind - inspect tank b005/xx099 ozone_orbit - inspect tank b008/xx012 4)Follow the PRE-IMPLEMENTATION TESTING instructioins above. DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest [--v4.6.2--]v4.7.0 RELEASED [--JUN 08,--]AUG 14 2023 ATMS+CRIS [--v4.6.2--]v4.7.0 IMPLEMENTED [--JUN ??,--]SEP 20 2023 PRELUDE NOAA-21 CrIS data was declaired operational in early 2023. This implementation makes NOAA-21 CrIS data availble to GFS/GSI. NOAA-21 ATMS data was declaired operational in early 2023. NOAA_21 ATMS processing coefficient table was updated on May 12,2023. This implementation makes NOAA-21 ATMS data availble to GFS/GSI. Obsolete fix files, used by now retired data sets, were removed. Activates the use of EFIRE and cloud generated OMPS TC FILE CHANGES M docs/Release_Notes.txt M fix/aerosol.af_viirs D fix/aerosol.burned_area D fix/aerosol.burning_emissions D fix/aerosol.modis_aod M fix/npp_omps.ompstc_v8 D fix/snowore.gribcopy_usaf_nh D fix/snowore.gribcopy_usaf_sh M fix/idsb_ingest.source.html M fix/CMakeLists.txt M fix/npp_atms.atms M fix/npp_cris_431 M fix/npp_cris_431.cris_fsr_431 M fix/snowore M jobs/JISATINGEST M sorc/CMakeLists.txt A sorc/bufr_readmp.fd A sorc/bufr_readmp.fd/CMakeLists.txt A sorc/bufr_readmp.fd/temp_readmp.f90 M ush/ingest_copy.sh M ush/ingest_process_orbits_subscript.sh M versions/build.ver M versions/run.ver M versions/VERSION (M-modified; A-added; D-deleted) Only in $PACKAGEROOT/satingest.v4.6.1 (not part of the satingest.v4.6.1 S/W package that EMC released) fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi versions/run.ver.prod DOCS CHANGES docs/Release_Notes.txt Update contents to reflect this release FIX CHANGES fix/CMakeLists.txt Remove the names for the deleted fix files Delete fix files for data no longer ingested by satingest: fix/aerosol.burned_area : burned_area_*_*_*_*km.grib2.gz fix/aerosol.burning_emissions : GBBEPx*.nc fix/aerosol.modis_aod : Terra_AOS_BUFR.* Aqua_AOD_BUFR.* fix/snowore.gribcopy_usaf_nh : NPR.SNWN.SP.D*.S1200.MESH16 fix/snowore.gribcopy_usaf_sh : NPR.SNWS.SP.D*.S1200.MESH16 fix/aerosol.af_viirs Introduced new filename for active fire products from NCCF EFIRE-VIIRSI_v1r2_???_s???????????????_e???????????????_c???????????????.txt fix/idsb_ingest.source.html Update filenames that change in this release fix/npp_atms.atms Introduce filename for NOAA-21 ATMS data PDAFileLinks/NDE/ATMS/ATMS_v1r0_n21_s???????????????_e???????????????_c???????????????.bufr fix/npp_cris_431 Introduce iflag_CRIS fix/npp_cris_431.cris_fsr_431 Export iflag_CRIS, introduce filename for NOAA-21 CrIS data PDAFileLinks/NDE/NUCAPS/NUCAPS-C0431_v3r0_n21_s???????????????_e???????????????_c???????????????.bufr fix/npp_omps.ompstc_v8 Introduce filename for NCCF generated OMPS TC product OMPS-TC_v4r3_???_s???????????????_e???????????????_c???????????????.bufr fix/snowore Turn off snowore.gribcopy_usaf_nh and snowore.gribcopy_usaf_sh fix/idsb_ingest.source.html Removed obsolete data filenames and contact info JOBS CHANGES jobs/JISATINGEST Change PS4 format to print scripts names in log file SORC CHANGES sorc/CMakeLists.txt Add directory bufr_readmp.fd to list of directories sorc/bufr_readmp.fd Create directory bufr_readmp.fd sorc/bufr_readmp.fd/CMakeLists.txt Introduce source code temp_readmp.f90 sorc/bufr_readmp.fd/temp_readmp.f90 Add code to "proof read" incoming NESDIS CrIS data USH CHANGES ush/ingest_copy.sh Add functionality to process EFIRE data filenames ush/ingest_process_orbits_subscript.sh Add "proof reading" of incoming NESDIS CrIS data, a.k.a "JACK WOOLLEN CHECK" VERSIONS CHANGES versions/build.ver versions/run.ver Update bufr_tranjb_ver to v1.0.4 versions/VERSION Update version to reflect this release CHANGES TO OUTPUT AND FILE SIZES No new tanks are generated. The ATMS data tank, b021/xx203, will increase in size by ~50% as we add a 3rd ATMS instrument to the ingest. The CrIS data tank, b021/xx206, will increase in size by ~50% as we add a 3rd CrIS instrument to the ingest. The ATMS and CrIS tank sizes will return to normal when the S-NPP ATMS and CrIS instrument is switched off (i.e. no more *_npp_* files). NESDIS hasn't provided a date for this. BIG BROTHER INFORMATION Start monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/ATMS_v1r0_n21_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v3r0_n21_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/EFIRE-VIIRSI_v1r2_???_s???????????????_e???????????????_c???????????????.txt $COMROOT/logs/bb/ingest/OMPS-TC_v4r3_???_s???????????????_e???????????????_c???????????????.bufr CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Satingest works together with module bufr-tranjb (THIS RELEASE REQUIERS v1.0.4) It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE:1.At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. 2.fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. 3.Directory /TRIGGERS is only used by EMC for testing 4.Directory /ecf is not part of the released code, b/c it is used only by NCO. It is added by the SPAs during implementation, see $PACKAGEROOT/satingest.v*.*.* PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.3 or bufr-tranjb.v1.0.4 to be installed in order to run successfully. Module bufr-tranjb.v1.0.* is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions NOTE: Separate CDF will be submitted for bufr-tranjb.v1.0.4, at the same time as satingest.v4.7.0 B. Run JISATINGEST for the following satingest tasks (on implementation day) and verify the generation of the listed tanks: npp_atms - generates tanks b021/xx203 npp_cris_431 - generates tanks b021/xx206 C. If there is a NCO parallel run running, it must be stopped BEFORE the implementation, so that the prod run can pull the files from PDA OPS (and not run out of allowed PDA pulls per data type) DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)IMPORTANT: This upgrade must be implemented --after-- the GFS/GSI upgrade adding NOAA-21 ATMS calibr. coeff. to GFS/GSI Contact Andrew.Collard@noaa.gov for a specific date. 2)Confirm that NCO has an ACTIVE subscribtion to the following product(s) on PDA OPS a day or two prior to implementation day Short name: ATMS_BUFR RegEx: ATMS_v1r0_n21_s???????????????_e???????????????_c???????????????.bufr Dest Dir: NDE/ATMS NOTE:One and the same Short name is used for ATMS_v1r0_npp_* ATMS_v1r0_j01_* and ATMS_v1r0_n21_* filenames Short name: CrIS_C0431_BUFR RegEx: NUCAPS-C0431_v3r0_n21_s???????????????_e???????????????_c???????????????.bufr Dest Dir: NDE/NUCAPS NOTE:One and the same Short name is used for NUCAPS-C0431_v3r_npp_* NUCAPS-C0431_v3r_j01_* and NUCAPS-C0431_v3r_n21_* filenames 3) This satingest v4.7.0 requires that module bufr-trankb v1.0.4 is installed 4)Follow the PRE-IMPLEMENTATION TESTING instructioins above. Implementation could be done at any hour b/c the new data will be monitored at first. DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.6.1 RELEASED MAY 15, 2023 v4.6.1 IMPLEMENTED MAY 22, 2023 PRELUDE OMI instrument longevity led to 6 digit orbit number in the filename (instead of 5). This implementation prepares satingest for the longer filenames. GBBEPx v3 products were retired on 1 May 2023. Satingest will no longer ingest this product. Data Flow will ingest an extended GBBEPx v4 data set into /dcom/YYYYMMDD/burned_emission (and to /dcom/YYYYMMDD/firewx) FILE CHANGES M docs/Release_Notes.txt M fix/aerosol M fix/ozone_orbit.omi M versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/Release_Notes.txt Update contents to reflect this release FIX CHANGES fix/aerosol Turn off processing family BUFRNING_EMMISSIONS. fix/ozone_orbit.omi Introduce a filename with 6 digits for the orbit number of OMI PDAFileLinks/PULL/OZONE/OMI-Aura_L2-OMTO3_????m????t????-o??????_v883-????m????t??????.he5 SORC CHANGES None USH CHANGES None versions/VERSION Update version to reflect this release CHANGES TO OUTPUT AND FILE SIZES The following new tanks, and their expected sizes, are generated: N/A BIG BROTHER INFORMATION Remove the following filename(s) from monitoring: $COMROOT/logs/bb/ingest/GBBEPx* $COMROOT/logs/bb/ingest/OMI-Aura_L2-OMTO3_????m????t????-o?????_v883-????m????t??????.he5 Start monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/OMI-Aura_L2-OMTO3_????m????t????-o??????_v883-????m????t??????.he5 CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Module bufr-tranjb v1.0.3 stays as is. It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE:1.At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. 2.fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. 3.Directory /TRIGGERS is only used by EMC for testing 4.Directory /ecf is not part of the released code, b/c it is used only by NCO. It is added by the SPAs during implementation, see $PACKAGEROOT/satingest.v*.*.* PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.3 to be installed in order to run successfully. Module bufr-tranjb.v1.0.3 is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions B. Run JISATINGEST for the following satingest tasks (on implementation day) and verify the generation of the listed tanks: ozone_orbit - generates tanks b008/xx013 DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)Confirm that NCO has an ACTIVE subscribtion to the following product(s) on PDA OPS a day or two prior to implementation day: Aura OMI Ozone from eosb1.nesdis.noaa.gov Short name: [WCOSS_]OMITO3 (the EMC subscription is EMC_OMITO3, thus I guess NCO/DF existing sub is WCOSS_OMITO3) RegEx: OMI-Aura_L2-OMTO3_????m????t????-o??????_v883-????m????t??????.he5 (NOTE: there are 6 digits after -o) Dest Dir: PULL/OZONE/ 2)Follow the PRE-IMPLEMENTATION TESTING instructioins above. Implementation could be done at any hour b/c the legacy products will be retired in late May 2023. DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.6.0 RELEASED MAR 17, 2023 v4.6.0 IMPLEMENTED APR 12, 2023 PRELUDE NESDIS implements GMASI-AutoSnow daily snow and ice products in NCCF. https://www.ospo.noaa.gov/data/messages/2023/02/MSG_20230221_2140.html Satellite data ingest webpage is updated https://www2.nco.ncep.noaa.gov/idsb/ingest/ FILE CHANGES M docs/Release_Notes.txt M fix/CMakeLists.txt M fix/aerosol.af_viirs M fix/idsb_ingest.source.html M fix/snowore A fix/snowore.gribcopy_autoice_nh A fix/snowore.gribcopy_autoice_sh A fix/snowore.gribcopy_autosnow_nh M fix/snowore.gribcopy_autosnow_sh M fix/tovs.hirs4 M ush/CMakeLists.txt A ush/ingest_copytotarget.sh M versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/Release_Notes.txt Update contents to reflect this release FIX CHANGES fix/CMakeLists.txt Add new fix files' names for autoice - snowore.gribcopy_autoice_nh, snowore.gribcopy_autoice_sh fix/aerosol.af_viirs Correct a typo in instrument name, _jo1_ is now _j01_ fix/idsb_ingest.source.html Correct filenames for NAVO SST data: mcsst_* Add AutoSnow products: GMASI-* fix/snowore Added three new processing familes: GRIBCOPY_AUTOSNOW_NH GRIBCOPY_AUTOICE_NH GRIBCOPY_AUTOICE_SH fix/snowore.gribcopy_autoice_nh Introduce new processing family for autoice NH and relevant filenames: PDAFileLinks/PULL/GRIB/GMASI-Ice-NH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 fix/snowore.gribcopy_autoice_sh Introduce new processing family for autoice SH and relevant filenames: PDAFileLinks/PULL/GRIB/GMASI-Ice-SH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 fix/snowore.gribcopy_autosnow_nh Introduce new processing family for autosnow NH and relevant filenames: PDAFileLinks/PULL/GRIB/GMASI-Snow-NH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 fix/snowore.gribcopy_autosnow_sh Use new resolution product filenames: PDAFileLinks/PULL/GRIB/GMASI-Snow-SH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 fix/tovs.hirs4 Remove commented lines, changed criticality to "NO" SORC CHANGES None USH CHANGES ush/ingest_copytotarget.sh Add logic to copy filenames like AutoSnow and AutoIce's GMASI* with timestamp formated as s???????????????_e???????????????_c??????????????? versions/VERSION Update version to v4.6.0 CHANGES TO OUTPUT AND FILE SIZES The following new tanks, and their expected sizes, are generated: N/A BIG BROTHER INFORMATION Remove the following filename(s) from monitoring: $COMROOT/logs/bb/ingest/SH4km_autosnow_????????.grb - on implementation day Start monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/GMASI-Ice-NH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 $COMROOT/logs/bb/ingest/GMASI-Ice-SH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 $COMROOT/logs/bb/ingest/GMASI-Snow-NH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 $COMROOT/logs/bb/ingest/GMASI-Snow-SH2km_v1r0_blend_s???????????????_e???????????????_c???????????????.grib2 CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Module bufr-tranjb v1.0.3 stays as is. It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE:1.At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. 2.fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. 3.Directory /TRIGGERS is only used by EMC for testing 4.Directory /ecf is not part of the released code, b/c it is used only by NCO. It is added by the SPAs during implementation, see $PACKAGEROOT/satingest.v*.*.* PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.3 to be installed in order to run successfully. Module bufr-tranjb.v1.0.3 is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions B. Run JISATINGEST for the following satingest tasks (on implementation day) and verify the generation of the listed tanks: snowore :: $DCOMROOT/YYYYMMDD/wgrbbul/GMASI-Snow-NH2km*,$DCOMROOT/YYYYMMDD/wgrbbul/GMASI-Snow-SH2km* $DCOMROOT/YYYYMMDD/wgrbbul/GMASI-Ice-NH2km*, $DCOMROOT/YYYYMMDD/wgrbbul/GMASI-Ice-SH2km* Once a day, at ~05:00UTC DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)Subscribe to the following product(s) on PDA OPS a day or two prior to implementation day Short name: GMASI_BLEND_ICE_NH_GRIB RegEx: GMASI-Ice-NH2km_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}\.grib2 Dest Dir: PULL/GRIB Short name: GMASI_BLEND_ICE_SH_GRIB RegEx: GMASI-Ice-SH2km_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}\.grib2 Dest Dir: PULL/GRIB Short name: GMASI_BLEND_SNOW_NH_GRIB RegEx: GMASI-Snow-NH2km_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}\.grib2 Dest Dir: PULL/GRIB Short name: GMASI_BLEND_SNOW_SH_GRIB RegEx: GMASI-Snow-SH2km_v\dr\d_blend_s\d{15}_e\d{15}_c\d{15}\.grib2 Dest Dir: PULL/GRIB 2)Follow the PRE-IMPLEMENTATION TESTING instructioins above. Implementation could be done at any hour b/c the legacy products will be retired in late May 2023. DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.5.0 RELEASED JAN 16, 2023 v4.5.0|.1|.2 IMPLEMENTED FEB 08, 2023/FEB 15, 2023/ MAR 01, 2023 PRELUDE Respond to several NESDIS product upgrades and resultng filename changes: - NDE release 2.0.35 delivers: = Active Fire I-band product (Feb 8, 2023) = Update OMPS V8Pro products and prepare for NOAA-21 (Feb 15, 2023) - Declare GPM GMI L1 BUFR Products Operational in NCCF --(Feb 15, 2023)-(Feb 22,2023)-NEW: Mar 01, 2023 Add a new processing family for ingesting high-resolution winds: (Feb 8, 2023) i)CONUS domain NESDIS generated AMVs; tanks b005/xx05** ii)Tropical Cyclon domain UWisconsin generated AMVs; tank b005/xx099 Postpone radsnd task by 1 minute to allow UW Total Sky data to arrive for HH:15 RTMA RU cycles FILE CHANGES M docs/README.md M docs/Release_Notes.txt M fix/CMakeLists.txt M fix/aerosol.af_viirs M fix/bufrtab.005 M fix/idsb_ingest.source.html A fix/hrsatwind A fix/hrsatwind.cimss A fix/hrsatwind.circd_dmw A fix/hrsatwind.cshrtwv_dmw A fix/hrsatwind.cvisb_dmw A fix/hrsatwind.cwvics_dmw A fix/hrsatwind.cwvict_dmw M fix/npp_omps.ompsnp_v8 (Please update on Feb 15, 2023) M fix/idsb_ingest.source.html M fix/snowore.gribcopy_usaf_nh M fix/snowore.gribcopy_usaf_sh M fix/ssmi.gmi_tmbr (Please update on Mar 01, 2023) M sorc/CMakeLists.txt A sorc/bufr_trancimssamv.fd/CMakeLists.txt A sorc/bufr_trancimssamv.fd/trancimssamv.f90 M ush/ingest_copy.sh M versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/README.md Update build instructions docs/Release_Notes.txt Update contents to refelct this release FIX CHANGES fix/aerosol Turn off processing of MODIS_AOD BURNED_AREA ??? fix/CMakeLists.txt Add fix filenames for high-res CONUS and CIMSS AMVs fix/aerosol.af_viirs Add NOAA-21 file name versions for old AF product and support for new AF-Iband_* product: AF_v1r2_n21_s???????????????_e???????????????_c???????????????.txt AF-Iband_v1r1_???_s???????????????_e???????????????_c???????????????.txt fix/bufrtab.005 Updat FXY value (with unique ones) for NC00505* (CONUS AMVs) and NC005081 (Metop-B/C AMVs) Add entry for NC005099 (CIMSS AMVs) fix/idsb_ingest.source.html Update filenames and criticality status to reflect the NCO DF subscriiptons fix/hrsatwind Add a processinig family for the high-res AMVs fix/hrsatwind.cimss Add a processinig group for the CIMSS high-res AMVs fix/hrsatwind.circd_dmw fix/hrsatwind.cshrtwv_dmw fix/hrsatwind.cvisb_dmw fix/hrsatwind.cwvics_dmw fix/hrsatwind.cwvict_dmw Add processinig groups for the CIMSS high-res IR, SWIR, VIS, WVCT, WVCA AMVs fix/npp_omps.ompsnp_v8 Update OMPS V8Pro product filenames Old: OMPS-NP_v4r1_???_s???????????????_e???????????????_c???????????????.bufr New: OMPS-NP_v4r2_???_s???????????????_e???????????????_c???????????????.bufr fix/poes_sst.navy_avhrr Removed Metop-A filename mcsst_mta_d fix/snowore.gribcopy_usaf_nh fix/snowore.gribcopy_usaf_sh Turn off PDA pulls for *MESH16, now pulled by DF fix/ssmi.gmi_tmbr (Please update on Feb 15, 2023) Update GMI file name Old: 1C-R.GPM.GMI.XCAL2016-C.????????-S??????-E??????.V07A.bufr New: GMI-1CR_v7r0_gpm_s???????????????_e???????????????_c???????????????.bufr SORC CHANGES sorc/CMakeLists.txt Add directory bufr_trancimssamv.fd for CIMSS AMVs sorc/bufr_trancimssamv.fd/CMakeLists.txt List files for cmake sorc/bufr_trancimssamv.fd/trancimssamv.f90 Add iingest code for CIMSS AMVs USH CHANGES ush/ingest_copy.sh Update logic to copy new AFI file names for fix/aerosol.af_viirs versions/VERSION Update version from v4.4.0 to v4.5.0 CHANGES TO OUTPUT AND FILE SIZES The following new tanks, and their expected sizes, are generated: b005/052 120M b005/053 25M b005/054 250M b005/055 25M b005/056 35M b005/099 200M-600M BIG BROTHER INFORMATION Remove the following filenames from monitoring once new names (see below for subscription instructions) are present $COMROOT/logs/bb/ingest/OMPS-NP_v2r1_???_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/1C-R.GPM.GMI.XCAL2016-C.????????-S??????-E??????.V07A.bufr Start monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/OMPS-NP_v4r2_???_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/GMI-1CR_v7r0_gpm_s???????????????_e???????????????_c???????????????.bufr CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs were needed during EMC's testing. No changes to memory usage for any jobs were needed during EMC's testing. SHARED SOFTWARE Module bufr-tranjb v1.0.3 stays as is. It is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INSTRUCTIONS See README.md or do as follows: A shell script to build and install satingest on NOAA platforms is provided:/ush/build.sh. Its usage on WCOSS is: INSTALL_PREFIX= ./ush/build.sh NOTE: At the moment, the following does not work: cd ush ./build.sh (or alternatvely: source build.sh) It will be fixed soon. NOTE: fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.3 to be installed in order to run successfully. Module bufr-tranjb.v1.0.3 is co-owned/co-developed with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libraries and required versions B. Run JISATINGEST for the following tasks (on implementation day) and verify the generation of the listed tanks: npp_omps : b008/xx017 (OMPS-NP) poes_sst : b012/xx012 (navy_avhrr) ssmi : b021/xx246 (gmi_tmbr) (Please update on Feb 15, 2023) hrsatwind : b005/xx052,xx053,xx054,xx055,xx056 (CONUS AMVs) and xx099 (CIMSS AMVs) aerosol : /lfs/h1/ops/prod/dcom/YYYYMMDD/af_viirs/AF-Iband_* files are pulled C. During the testing of satingest v4.5.0, please run JISATINGEST for radsnd.efclamzip with 2 minutes delay. This could be accomplished by delaying the processing of the entire radsnd family. Then run parallel RTMA obsproc job using the delayed radsnd.efclamzip tanks (b012/xx160). The goal is to test if delaying the ingest would resolve the generation of blank *efclam*dump_d files for RTMA Rapit Update HH:15 cycles. Please reach out to obsproc for clarifciation if needed. DISSEMINATION INFORMATION No change in dissemination methods. SPECIAL INSTRUCTIONS 1)Follow the PRE-IMPLEMENTATION TESTING instructioins above 2)Make sure to wait and update fix/ssmi.gmi_tmbr on Feb 15, 2023 (unlike everything else on Feb 8, 2023) 3)Subscribe to the following product(s) on PDA OPS a day or two prior to implementation day (Feb 8, 2023) Short name: OMPS_NP_V8_BUFR RegEx: OMPS-NP_v4r2_???_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/OMPS Short name: NCCF_GPM_GMI_L1C-R_BUFR RedEx: GMI-1CR_v\d{1}r\d{1}_gpm_s\d{15}_e\d{15}_c\d{15}.bufr Dest Dir: PDAFileLinks/PULL/GMI Short name: AF_Iband_VIIRS_ASCII_List RegEx: AF-Iband_v1r1_jo1_s???????????????_e???????????????_c???????????????.txt Dest Dir: PDAFileLinks/PULL/ASCII Also, if NCO DF is not subscrbed to these already, please do so for: Short name: NB_DMWC_C02_BUFR RegEx: NB-DMWC-M6C02_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short name: NB_DMWC_C07_BUFR RegEx: NB-DMWC-M6C07_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWC_C08_BUFR RegEx: NB-DMWC-M6C08_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWC_C14_BUFR RegEx: NB-DMWC-M6C14_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVC_C08_BUFR RegEx: NB-DMWVC-M6C08_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVC_C09_BUFR RegEx: NB-DMWVC-M6C09_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVC_C10_BUFR RegEx: NB-DMWVC-M6C10_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV DISCLAIMER: In case of Critical Weather Day, NESDIS's updates might be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implementation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.4.0 RELEASED NOV 29, 2022 v4.4.0 IMPLEMENTED DEC 14&15, 2022 v4.4.1 IMPLEMENTED JAN 04, 2023 PRELUDE NESDIS retired a number of NOAA-19, MODIS Terra and Aqua products on Oct.31,2022. The corresponding data filenames were removed from satingest configuration files (i.e. /fix files).[implement 14Dec2022] NESDIS NDE release 2.0.34 includes a BUFR toolkit upgrade to version 5.4. The impacted BUFR products filenames are refelected in the relevant /fix files.[implement 14Dec2022] NESDIS started generating Metop-B and -C ASCAT winds, they are introduced in this release.[implement 14Dec2022,v4.4.0] NESDIS CrIS and ATMS producst are now NCCF generated (on the cloud) and have updated filenames.[implement 15Dec2022, v4.4.0] Filenames for GOES-18 NESDIS ASR, CSR and AMV, UWisc Total Sky Cover, NASA LARC Cloud products, and VIIRS GVF products might be added to the config files, in preparation for these products promotion on Jan 4, 2022.[implement 4Jan2023, most likely v4.4.1] FILE CHANGES M docs/Release_Notes.txt M fix/aerosol M fix/aerosol.burned_area M fix/aerosol.dust M fix/aerosol.modis_aod M fix/aerosol.viirs_weekly_gvf (4Jan2023) M fix/CMakeLists.txt M fix/npp_cris_2211.cris_fsr_2211 M fix/npp_cris_431.cris_fsr_431 M fix/npp_omps.ompsnp_v8 M fix/npp_omps.ompstc_v8 M fix/ozone14 M fix/ozone14.ozone19_v8bufr M fix/ozone_orbit M fix/ozone_orbit.ozone_v8bufr M fix/qscatter.ascat_50km M fix/radsnd D fix/radsnd.goes16_asr_base A fix/radsnd.goes_asr_base M fix/radsnd.goes_csr_base M fix/radsnd.efclamzip (4Jan2023) M fix/radsnd.lgycld (4Jan2023) M fix/satwnd.ircd_dmw M fix/satwnd.shrtwv_dmw M fix/satwnd.visb_dmw M fix/satwnd.wvics_dmw M fix/satwnd.wvict_dmw M versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/Release_Notes.txt Update contents and renamed release_notes/ to docs/ FIX CHANGES fix/CMakeLists.txt Replace radsnd.goes16_asr_base (Deleted) with radsnd.goes_asr_base (Added) fix/aerosol Turn off (remove) MODIS_AOD and BURNED_AREA processing family fix/aerosol.burned_area Comment out burned_area_* filenames (equivalent to stop their ingest) fix/aerosol.dust Comment out MYDdust.aod_* filenames (equivalent to stop their ingest) fix/aerosol.modis_aod Comment out Terra_AOD_* and Aqua_AOD_* filenames (equivalent to stop their ingest) fix/aerosol.viirs_weekly_gvf (4Jan2023) Update version in filenames GVF-WKL-GBL_* fix/npp_cris_2211.cris_fsr_2211 Update version in filenames NUCAPS-C2211_* fix/npp_cris_431.cris_fsr_431 Update version in filenames NUCAPS-C0431_* fix/npp_omps.ompsnp_v8 Remove old OMPS-NP_v2r0* filenames fix/npp_omps.ompstc_v8 Remove filenames OMPS-TC_v2r0* fix/ozone14 Turn off (remove) OZONE19_V8bufr processing family fix/ozone14.ozone19_v8bufr Comment out PRD.OZONE.PMFV8.N19.BUFR filenames (equivalent to stop its ingest) fix/ozone_orbit Turn off (remove) OZONE_V8bufr processing family fix/ozone_orbit.ozone_v8bufr Comment out PRD.OZONE.PMFV8.N19.D?????.S????.E????.????? filename (equivalent to stop its ingest) fix/qscatter.ascat_50km Remove old ascat_* filename. Add new OSVW50kmASCAT_v1r1_m0*_* filenames fix/radsnd Remove processing family GOES16_ASR_BASE. Add procesing family GOES_ASR_BASE fix/radsnd.goes_asr_base (and on 4Jan2023) Added to accomodate all GOES-16/17/18 filenames for ASR fix/radsnd.goes_csr_base (and on 4Jan2023) Update version in filenames ABI-L2-CSRF-M6* and add filenames for GOES-18 fix/radsnd.efclamzip (4Jan2023) Add filename for GOES_18, No PDA sub needed, data on ftp fix/radsnd.lgycld (4Jan2023) Add filename for GOES_18, No PDA sub needed, data on ftp fix/satwnd.ircd_dmw (and on 4Jan2023) Add filename for GOES_18 fix/satwnd.shrtwv_dmw (and on 4Jan2023) Add filename for GOES_18 fix/satwnd.visb_dmw (and on 4Jan2023) Add filename for GOES_18 fix/satwnd.wvics_dmw (and on 4Jan2023) Add filename for GOES_18 fix/satwnd.wvict_dmw (and on 4Jan2023) Add filename for GOES_18 versions/VERSION Update version from v4.3.0 to v4.4.0 CHANGES TO OUTPUT AND FILE SIZES No new tanks are generated, output file sizes are compariable to satingest v4.3.0 BIG BROTHER INFORMATION Remove the following filenames from monitoring once new names (see below for subscription instructioins) are present $COMROOT/logs/bb/ingest/burned_area_????????_??_??_1km.grib2.gz $COMROOT/logs/bb/ingest/burned_area_????????_??_??_12km.grib2.gz $COMROOT/logs/bb/ingest/MYDdust.aod_conc.v6.3.4.????????.hr??.grib $COMROOT/logs/bb/ingest/Terra_AOD_BUFR.???????.???? $COMROOT/logs/bb/ingest/Aqua_AOD_BUFR.???????.???? $COMROOT/logs/bb/ingest/PRD.OZONE.PMFV8.N19.BUFR $COMROOT/logs/bb/ingest/PRD.OZONE.PMFV8.N19.D?????.S????.E????.?????? $COMROOT/logs/bb/ingest/ABI-L2-CSRF-M6_v2r1_g17_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-CSRF-M6_v2r1_g16_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ascat_????????_??????_metopb_?????_srv_o_250_ovw.l2_bufr $COMROOT/logs/bb/ingest/OMPS-NP_v2r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-NP_v2r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v2r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v2r0_j01_s???????????????_e???????????????_c???????????????.bufr Start monitoring the following new filename patterns: $COMROOT/logs/bb/ingest/OMPS-TC_v4r2_???_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-LP-EV-EDR_v2r51_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C2211_v3r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C2211_v3r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v3r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-C0431_v3r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OSVW50kmASCAT_v1r1_m01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OSVW50kmASCAT_v1r1_m03_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-CSRF-M6_v2r2_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-CSRF-M6_v2r2_g17_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-CSRF-M6_v2r2_g16_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-ASRF-M6_v2r2_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-ASRF-M6_v2r2_g17_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ABI-L2-ASRF-M6_v2r2_g16_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?F-M?C14_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?F-M?C07_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?F-M?C02_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?VF-M?C08_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?VF-M?C09_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?VF-M?C10_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NB-?M?F-M?C08_v1r0_g18_s???????????????_e???????????????_c???????????????.bufr CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs. No changes to memory usage for any jobs. SHARED SOFTWARE Module bufr-tranjb v1.0.3 stays as is. It is co-owned/co-devoped with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INNSTRUCTIONS See README.md NOTE: fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. PRE-IMPLEMENTATION TESTING REQUIREMENTS A. satingest needs module bufr-tranjb.v1.0.3 to be installed in order to run successfully. Module bufr-tranjb.v1.0.3 is co-owned/co-devoped with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libaries and required versions B. Run JISATINGEST for the following tasks (on implemenatation day) and verify the generation of the listed tanks: npp_cris_431 : b021/xx206 npp_atms : b021/xx203 npp_omps : b008/xx018 (OMPS-TC), b008/xx019 (OMPS-LP) qscatter : b012/xx122 radsnd : b021/xx045 (ASR), b021/xx046 (CSR), b012/xx150(NASA LARC Cloud Product), b012/xx160 (UWisc Total Sky Cover) satwnd : b005/xx030, b005/xx031, b005/xx032, b005/xx034, b005/xx039 C. Run JISATINGEST for the following tasks (at any time after code release): aerosol,ozone14,ozone_orbit There should be no more warnings for files not found/retrieved in the satingest logs for: aerosol.burned_area aerosol.dust aerosol.modis_aod ozone14.ozone19_v8bufr ozone_orbit.ozone_v8bufr DISSEMINATION INFORMATION No change in dissemination. SPECIAL INSTRUCTIONS Subscribe to the following product(s) on PDA OPS a day or two prior to implementation day (Dec 14, 2022) Short name: OMPS_TC_V8_BUFR RegEx: OMPS-TC_v4r2_???_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/OMPS Short name: OMPS_LP_EV_EDR_BUFR RegEx: OMPS-LP-EV-EDR_v2r51_npp_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/OMPS (no _j01_ product) Short name: ABI_L2_ASRF_BUFR RegEx: ABI-L2-ASRF-M6_v2r2_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/PULL/BIN Short name: ABI_L2_CSRF_BUFR RegEx: ABI-L2-CSRF-M6_v2r2_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/PULL/BIN Short name: NB_DMWC_C02_BUFR RegEx: NB-DMWC-M6C02_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short name: NB_DMWC_C07_BUFR RegEx: NB-DMWC-M6C07_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWC_C08_BUFR RegEx: NB-DMWC-M6C08_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWC_C14_BUFR RegEx: NB-DMWC-M6C14_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWF_C02_BUFR RegEx: NB-DMWF-M6C02_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWF_C07_BUFR RegEx: NB-DMWF-M6C07_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWF_C08_BUFR RegEx: NB-DMWF-M6C08_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWF_C14_BUFR RegEx: NB-DMWF-M6C14_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVC_C08_BUFR RegEx: NB-DMWVC-M6C08_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVC_C09_BUFR RegEx: NB-DMWVC-M6C09_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVC_C10_BUFR RegEx: NB-DMWVC-M6C10_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVF_C08_BUFR RegEx: NB-DMWVF-M6C08_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVF_C09_BUFR RegEx: NB-DMWVF-M6C09_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short Name: NB_DMWVF_C10_BUFR RegEx: NB-DMWVF-M6C10_v1r0_g1?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/GOESAMV Short name: EMC_SEAP_ASCAT_OSW_500_BUFR RegEx: OSVW50kmASCAT_v1r1_m0?_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/PULL/SCAT Short name: CrIS_C0431_BUFR RegEx: NUCAPS-C0431_v3r0_???_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/NUCAPS Short name: CrIS_C2211_BUFR RegEx: NUCAPS-C2211_v3r0_???_s???????????????_e???????????????_c???????????????.bufr Dest Dir: PDAFileLinks/NDE/NUCAPS DISCLAIMER: In case of Critical Weather Day, NESDIS's updates will be postponed. Obsproc will keep you informed of the new NESDIS update date, which will become the new implemenation date. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.3.0 RELEASED SEP 29, 2022 v4.3.0 IMPLEMENTED OCT 18, 2022 PRELUDE Top level documents (except CMakeLists.txt) were moved to relevant directories to meet EIB standards. A number of /fix files were updated with new filenames from NESDIS NDE release 2.0.33. Metop-2(A) references were removed from /fix files (instrument was decomissioned). FILE CHANGES M CMakeLists.txt D README.md A docs/README.md A docs/Release_Notes.txt M fix/aerosol M fix/aerosol.modis_fire M fix/airs.iasi_efov M fix/avhrr.avhrr_m1_m2_m3 M fix/idsb_ingest.source.html M fix/npp_omps.omps_lp M fix/ozone_orbit.gome M fix/qscatter.ascat_50km M fix/satwnd.avhrr M fix/tovs.amsua M fix/tovs.hirs4 M fix/tovs.mhs D release_notes/Release_Notes.txt D VERSION M versions/build.ver M versions/run.ver A versions/VERSION (M-modified; A-added; D-deleted) DOCS CHANGES docs/README.md Updated contents and moved from top level dir to docs/ docs/Release_Notes.txt Updated contents and renamed release_notes/ to docs/ FIX CHANGES fix/aerosol Turn off (remove) WF_ABBA processing family fix/aerosol.modis_fire Remove obsolete flenames MODIS_C6_Global_MCD14DL_NRT_???????_????????????.txt fix/airs.iasi_efov fix/avhrr.avhrr_m1_m2_m3 fix/ozone_orbit.gome fix/qscatter.ascat_50km fix/satwnd.avhrr fix/tovs.amsua fix/tovs.hirs4 fix/tovs.mhs fix/satwnd.avhrr Remove Metop-2(A) references (filenames with Metop-2(A)) fix/idsb_ingest.source.html Updated filenames fix/npp_omps.omps_lp Introduced new OMPS LP filename OMPS-LP-EV-EDR_v2r51_npp_s???????????????_e???????????????_c???????????????.bufr There is no j01 product Remove obsolete filennames: OMPS-LP-EV-EDR_v2r50_npp_s???????????????_e???????????????_c???????????????.bufr OMPS-LP-EV-EDR_v2r50_j01_s???????????????_e???????????????_c???????????????.bufr CHANGES TO OUTPUT AND FILE SIZES No new tanks are generated, output file sizes are compariable to v4.2.1. BIG BROTHER INFORMATION Monitor new filename patterns: $COMROOT/logs/bb/ingest/OMPS-LP-EV-EDR_v2r51_npp_s???????????????_e???????????????_c???????????????.bufr Remove the following filenames from monitoring once new names (above) are present $COMROOT/logs/bb/ingest/OMPS-LP-EV-EDR_v2r50_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-LP-EV-EDR_v2r50_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/VIIRS-ACSPO-SST-ProbClear_v2r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/VIIRS-ACSPO-SST-ProbClear_v2r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/VIIRS-ACSPO-SST-Clear_v2r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/VIIRS-ACSPO-SST-Clear_v2r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v2r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v2r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-NP_v2r0_j01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/ascat_????????_??????_metopa_?????_srv_o_250_ovw.l2_bufr CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs. No changes to memory usage for any jobs. SHARED SOFTWARE Module bufr-tranjb v1.0.3 needs to be installed in sync. It is co-owned/co-devoped with Decoders (emc.decoders-support@noaa.gov) LIBRARIES w3emc 2.9.1 bufr 11.7.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INNSTRUCTIONS See README.md NOTE: fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. PRE-IMPLEMENTATION TESTING REQUIREMENTS satingest needs module bufr-tranjb.v1.0.3 to be installed inn order to run. Module bufr-tranjb.v1.0.3 is co-owned/co-devoped with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libaries and required versions Run JISATINGEST for the following tasks: npp_omps.omps_lp (generating tank b008/xx019) DISSEMINATION INFORMATION No change in dissemination. SPECIAL INSTRUCTIONS A.On Oct 18, 2022 (or a day earlier), subscribe to the following product(s) on PDA OPS: Short name: OMPS_LP_EV_EDR_BUFR_TEST Long name: J2 OMPS LP Earth View EDR LongName_TEST Regular Expression OMPS-LP-EV-EDR_v_r___n21_s________________e________________c_______________.bufr Destination Dir: PDAFileLinks/NDE/OMPS Short name: ACSPO_VIIRS_Clear_BUFR_TEST (NCO may have this already, but let's be explicit about Dest Dir) Long name: ACSPO VIIRS_GHRSST_Clear_BUFR LongName JPSS2_TEST Regular Expression VIIRS-ACSPO-SST-Clear_v%_n21_%.bufr Destination Dir: PDAFileLinks/NDE/VIIRS_GHRSST Short name: ACSPO_VIIRS_ProbClear_BUFR_TEST (NCO may have this already, but let's be explicit about Dest Dir) Long name: ACSPO VIIRS_GHRSST_BUFR LongName JPSS2_TEST Regular Expression VIIRS-ACSPO-SST-ProbClear_v%_n21_%.bufr Destination Dir: PDAFileLinks/NDE/VIIRS_GHRSST DISCLAIMER: i)The appearance of _n21_ may be confusing b/c NOAA-21 doesn't exist yet. The names were taken from PDA I&T product desciptions, exactly as entered by NESDIS. The filenames on PDA I&T server, however, contain _npp_ and _j01_ as expected. ii)The _TEST part of the names will probably drop out when subscribing on PDA OPS. At the time of handing this code to NCO, these products are only on PDA I&T. It is NESDIS's discretion to use the same filename on OPS. B. Satingest v4.3.0 must be implemented in sync with bufr-tranjb v1.0.3 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.2.0 RELEASED JUL 21, 2022 v4.2.1 IMPLEMENTED JUL 28, 2022(ARFC, SPAA Simon) FILE CHANGES M CMakeLists.txt M VERSION M fix/CMakeLists.txt M fix/aerosol D fix/aerosol.hms_smoke_final D fix/aerosol.hms_smoke_prelim D fix/aerosol.hms_smoke_prelim_early M fix/airs.airs_efov M fix/bufrtab.005 M fix/npp_omps.ompstc_v8 M fix/satwnd A fix/satwnd.nb_avhrr D fix/swd.wvics_dmw A release_notes/CMakeLists.txt A release_notes/Release_Notes.txt M ush/ingest_sncvgrib.sh M ush/ingest_sncvgrib96.sh (M-modified; A-added; D-deleted) USH CHANGES ush/ingest_sncvgrib.sh: Preserve the ASCII format snow file and place in $DCOMROOT/YYYYMMDD/wgrbbul ush/ingest_sncvgrib96.sh: Preserve the ASCII format snow file (different resolution) and place in $DCOMROOT/YYYYMMDD/wgrbbul FIX CHANGES fix/airs.iasi_efov Introduce new IASI HEAP data filenames (Impl.of Metop-B/C HEAP IASI in NCCF due 27 July 2022): NUCAPS-GMAO-616_v3r0_m01_s???????????????_e???????????????_c???????????????.bufr NUCAPS-GMAO-616_v3r0_m03_s???????????????_e???????????????_c???????????????.bufr Old file names will be removed with next implementation: NUCAPS-GMAO-616_v2r0_m03_s???????????????_e???????????????_c???????????????.bufr NPR_IARD.M02_D?????_S??????_E??????_G???_C0616_U6 NPR_IARD.M01_D?????_S??????_E??????_G???_C0616_U6 fix/npp_omps.ompstc_v8 Introduce new OMPS-TC data filename (NDE update/release 2.0.32 due July 21, 2022) OMPS-TC_v4r2_npp_s???????????????_e???????????????_c???????????????.bufr OMPS-TC_v4r2_j01_s???????????????_e???????????????_c???????????????.bufr Old file names will be removed with next implementation: OMPS-TC_v4r0_npp_s???????????????_e???????????????_c???????????????.bufr OMPS-TC_v4r0_j01_s???????????????_e???????????????_c???????????????.bufr fix/bufrtab.005 Rename old mnemonic MUCE to MUNCEX Rename old mnemonic MUNCEX to MUNCXX fix/aerosol Remove logic pertaning to HMS Split smoke model ingest (not transitioned to WCOSS2), and deleted the related fix/aerosol.hms_smoke_* files fix/aerosol.hms_smoke_final, fix/aerosol.hms_smoke_prelim, fix/aerosol.hms_smoke_prelim_early Remove because associated HYSPLIT model did not transition to WCOSS2. fix/satwnd Introduce new processing family NB_AVHRR fix/satwnd.nb_avhrr Introduce ingest and processing steps for new family NB_AVHRR's files: NAMV-AVHRR-C04CD-FRAC-NH_v1r0_m01_s???????????????_e???????????????_c???????????????.bufr NAMV-AVHRR-C04CD-FRAC-NH_v1r0_m03_s???????????????_e???????????????_c???????????????.bufr NAMV-AVHRR-C04CD-FRAC-SH_v1r0_m01_s???????????????_e???????????????_c???????????????.bufr NAMV-AVHRR-C04CD-FRAC-SH_v1r0_m03_s???????????????_e???????????????_c???????????????.bufr JOBS CHANGES No changes SOURSE CHANGES No changes. CHANGES TO OUTPUT AND FILE SIZES Generate new tank b005/xx081 from Metop-B/C satellite winds. New b005/xx081 tank will partially replace b005/xx080 and will be larger. Old b005/xx080 tanks size: ~2-7MB. New b005/xx081 tank size: ~10-15MB BIG BROTHER INFORMATION Monitor new filename patterns: $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v3r0_m01_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v3r0_m03_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v4r2_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v4r2_j01_s???????????????_e???????????????_c???????????????.bufr Remove following from monitoring once new names are present and BB begins identifying them as missing: $COMROOT/logs/bb/ingest/NUCAPS-GMAO-616_v2r0_m03_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/NPR_IARD.M02_D?????_S??????_E??????_G???_C0616_U6 $COMROOT/logs/bb/ingest/NPR_IARD.M01_D?????_S??????_E??????_G???_C0616_U6 $COMROOT/logs/bb/ingest/OMPS-TC_v4r0_npp_s???????????????_e???????????????_c???????????????.bufr $COMROOT/logs/bb/ingest/OMPS-TC_v4r0_j01_s???????????????_e???????????????_c???????????????.bufr Remove the following from monitoring: $COMROOT/logs/bb/ingest/hmshysplit????????.prelim.txt $COMROOT/logs/bb/ingest/hmshysplit????????.txt $COMROOT/logs/bb/ingest/hms????????.prelim.txt $COMROOT/logs/bb/ingest/hms????????.txt Disable BB alerts for: SAPHIR METOP-A HMS HYSPLIT CHANGES IN COMPUTE RESOURCES No changes in requested resources in the job cards for any jobs. No changes to memory usage for any jobs. Shared Software --------------- Module bufr-tranjb v1.0.2 is co-owned/co-devoped with Decoders (emc.decoders-support@noaa.gov) LIBRRAIESLibraries --------- w3emc 2.9.1 bufr 11.6.0 bacio 2.4.1 hdf5 1.10.6 netcdf 4.7.4 libjpeg 9c BUILD INNSTRUCTIONS See README.md NOTE: fix/CMakeLists.txt mentions the following files: fix/bufr_hirsrtcf_ibm.dat fix/bufr_lowtopog.dat fix/nesdis.lstags_transsmi These are binary files that belong to satingest, but are not on GitHub. They get downloaded from a server (emcrzdm) during the cmake install process and are used by satingest. PRE-IMPLEMENTATION TESTING REQUIREMENTS satingest needs module bufr-tranjb to be installed inn order to run. Module bufr-tranjb v1.0.2 is co-owned/co-devoped with Decoders (emc.decoders-support@noaa.gov) See versions/*.ver for libaries and required versions Run JISATINGEST for the following tasks: airs, npp_omps, satwnd DISSEMINATION INFORMATION No change in dissemination. Add tank b005/xx081 to archival on HPSS. SPECIAL INSTRUCTIONS On Jul 21, 2022 (or a day earlier), subscribe to the following product(s) on PDA OPS: OMPS-TC_v4r2_npp_s???????????????_e???????????????_c???????????????.bufr OMPS-TC_v4r2_j01_s???????????????_e???????????????_c???????????????.bufr On Jul 28, 2022 (or a day earlier), subscribe to the following product(s) on PDA OPS: NUCAPS-GMAO-616_v3r0_m01_s???????????????_e???????????????_c???????????????.bufr NUCAPS-GMAO-616_v3r0_m03_s???????????????_e???????????????_c???????????????.bufr On implementation day (Jul 21,2022), or sooner, stop the ingest of all satingest b005 tanks (i.e. b005/xx030,b005/xx031,b005/xx032,b005/xx034,b005/xx039,b005/xx070,b005/xx071,b005/xx072,b005/xx080,b005/xx081,b005/xx091) and use the apxdx tool to add the updated fix/bufrtab.005 table to each satingest tank. (/apps/ops/prod/nco/intel/19.1.3.304/bufr_util.v1.1.0/exec/apxdx) Run apxdx tool with $APXDX_TYPE=satingest. Resume ingest for the tanks in question. On implementation day (Jul 21,2022), or sooner, subscribe to the following Metop-B/C AMVs product(s) on PDA OPS: NAMV-AVHRR-C04CD-FRAC-NH_v1r0_m01_s???????????????_e???????????????_c???????????????.bufr NAMV-AVHRR-C04CD-FRAC-NH_v1r0_m03_s???????????????_e???????????????_c???????????????.bufr NAMV-AVHRR-C04CD-FRAC-SH_v1r0_m01_s???????????????_e???????????????_c???????????????.bufr NAMV-AVHRR-C04CD-FRAC-SH_v1r0_m03_s???????????????_e???????????????_c???????????????.bufr XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX satingest v4.1.0 RELEASED DEC 22, 2021 v4.1.3 IMPLEMENTED JUN 28, 2022 Transition from WCOSS Phase3 to WCOSS2 ====================================== https://docs.google.com/document/d/1kLpAJMp-xyVmQcmvJRPSeISzpXSB2I_ndcDjIIORXVE/edit?usp=sharing Where is the release tag on subversion/git/vlab? git clone https://github.com/NOAA-EMC/satingest git checkout release/satingest.v4.1.0 List of external software used (anything outside of your vertical structure), including compilers and version numbers for everything Software used must be a minimal list of modules/versions specified per job netcdf_ver=4.7.4 hdf5_ver=1.10.6 bacio_ver=2.4.1 w3emc_ver=2.9.1 bufr_ver=11.6.0 bufr_tranjb_ver=1.0.2 libjpeg_ver=9c grib_util_ver=1.2.2 gfs_ver=v16.2 List of all code/scripts modified with this release A CMakeLists.txt A README.md M Release_Notes.txt A VERSION A build.sh A fix/CMakeLists.txt M fix/airs M fix/ssmi.sdr A jobs/CMakeLists.txt M jobs/JISATINGEST A modulefiles/satingest_common.lua A modulefiles/satingest_hera.lua A modulefiles/satingest_orion.lua A modulefiles/satingest_wcoss2.lua A scripts/CMakeLists.txt D scripts/existday.sh.ecf D scripts/existore.sh.ecf A scripts/existday.sh A scripts/existore.sh A sorc/CMakeLists.txt A sorc/bufr_cycsplit.fd/CMakeLists.txt A sorc/bufr_tranamsua.fd/CMakeLists.txt A sorc/bufr_tranavhrr.fd/CMakeLists.txt A sorc/bufr_trangoescld.fd/CMakeLists.txt A sorc/bufr_trangoessst.fd/CMakeLists.txt A sorc/bufr_tranhirs3.fd/CMakeLists.txt A sorc/bufr_tranimgr.fd/CMakeLists.txt A sorc/bufr_tranmhs.fd/CMakeLists.txt A sorc/bufr_tranmls.fd/CMakeLists.txt M sorc/bufr_tranmls.fd/tranmls.f90 A sorc/bufr_tranmtypsbt.fd/CMakeLists.txt A sorc/bufr_tranomi.fd/CMakeLists.txt M sorc/bufr_tranomi.fd/tranomi.f90 A sorc/bufr_tranompstc.fd/CMakeLists.txt A sorc/bufr_tranpoessst.fd/CMakeLists.txt A sorc/bufr_tranpoessst_navo.fd/CMakeLists.txt A sorc/bufr_transaphir.fd/CMakeLists.txt A sorc/bufr_transatw.fd/CMakeLists.txt A sorc/bufr_transkycovr.fd/CMakeLists.txt A sorc/bufr_transsmi.fd/CMakeLists.txt A sorc/bufr_transsnd.fd/CMakeLists.txt A sorc/bufr_tranwindsat.fd/CMakeLists.txt M sorc/bufr_tranwindsat.fd/tranwindsat.f A sorc/snow_sno16grb.fd/CMakeLists.txt A sorc/snow_sno8grb.fd/CMakeLists.txt A sorc/snow_sno96grb.fd/CMakeLists.txt A TRIGGERS/ (2) A ush/CMakeLists.txt A ush/download_fix.sh M ush/ingest_check_lapsed_data.sh M ush/ingest_copy.sh M ush/ingest_process_days.sh M ush/ingest_process_onetype_newdays.sh M ush/ingest_process_onetype_neworbits.sh M ush/ingest_process_orbits.sh M ush/ingest_process_orbits_subscript.sh M ush/ingest_qmgr.sh M ush/ingest_query.sh M ush/ingest_script_atovs1b.sh M ush/ingest_script_mls.sh M ush/ingest_script_omi.sh M ush/ingest_sncvgrib.sh M ush/ingest_sncvgrib96.sh M ush/ingest_snodepgr.sh M ush/ingest_translate_orbits.sh M ush/ingest_transst_poes.sh M ush/ingest_transst_poes_navo.sh M ush/navo_poes_sst_bundle.sh A versions/build.ver A versions/run.ver ------------------ M-modified; A-added; D-deleted What changes were made to the above code/scripts to support the new architecture? -satingest now uses cmake to build and install the software package -build satingest repo with cmake -bufr_tranjb was moved out from underneath satingest and now is its own entity installed as a module -satingest software was modified to work with bufr_tranjb as a module (bufr_tranjb.v1.0.2) Were any other changes made that aren’t directly related to the transition? Obsolete data types were removed from processing:MEGHA-TROPIQUES SAPHIR L1A2 Brightness Temperatures. Affected file: fix/airs Are there any changes to incoming data needs or outgoing products? No If output filenames are changing, list the current and new filename: N/A Compute resource information, for every job: See TRIGGERS/ directory Usage: bash -l /path/to/triggers/lsf.JISATINGEST.DEVELOP.WCOSS2 $package $task > /path/to/output/log/file/lsf.JISATINGEST.DEVELOP.WCOSS2.${task}$$ 2>&1 For example: bash -l /path/to/triggers/lsf.JISATINGEST.DEVELOP.WCOSS2 satingest npp_omps > /path/to/output/log/file/lsf.JISATINGEST.DEVELOP.WCOSS2.npp_omps$$ 2>&1 Total tasks Tasks per node Serial or parallel or mpmd Memory requirements (exclusive or shared) List any other PBS resource information requirements or MPI tunables *** Providing PBS and/or ecFlow submission scripts as part of release tag is preferred; if they are provided then resource information is not needed in the release notes. Runtime changes compared to current production (/gpfs/dell1/nco/ops/com/logs/runtime/daily/* for current stats) WCOSS2 runtimes: grep "resources_used.walltime =" $OPSROOT/output/YYYYMMDD/satingest_${task}* Disk space required per day or per cycle; data retention on disk will remain the same unless otherwise requested. Same as WCOSS1 Dissemination will remain the same unless otherwise communicated to NCO and a PNS/SCN issued HPSS archive retention will remain the same unless approval granted by HPCRAC What are your up and downstream dependencies? NESDIS PDA subscriptions; decoders (decod_dcncld) Provide a pointer to your COMOUT, temp directory and pbspro output file that was used during testing: This is already running on dogwood by SPA group COMOUT=$DCOMROOT logs in $OPSROOT/output/YYYYMMDD ingest history = $DCOMROOT/ingest_hist WCOSS1 RN:https://github.com/ShelleyMelchior-NOAA/smelchior/tree/main/RNs END