Running prologue on parent mom node: nid001872... Job 1054213.dbqs01 nodelist: nid001872 HT is disabled: 0-127 Job 1054213.dbqs01 - Prologue complete. Execution time: 1 seconds Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash) Job 1054213.dbqs01 - Epilogue complete. Execution time: 1 seconds ================================================================== BEGIN - DEBUG INFO ================================================================== Job 1054213.dbqs01 - Post Job Mem Usage: nid001872: 2022-01-26.13 1 =========================================== nid001872: 2022-01-26.13 2 #### Node Memory Usage for nid001872 #### nid001872: 2022-01-26.13 3 ------------------ nid001872: 2022-01-26.13 4 Mem: total:527307116 used:9511332 free:514215344 shared:2125700 cache:3580440 avail:513559940 nid001872: 2022-01-26.13 5 4.0K /dev/shm nid001872: 2022-01-26.13 6 42M /tmp nid001872: 2022-01-26.13 7 =========================================== nid001872: 2022-01-26.21 1 =========================================== nid001872: 2022-01-26.21 2 #### Node Memory Usage for nid001872 #### nid001872: 2022-01-26.21 3 ------------------ nid001872: 2022-01-26.21 4 Mem: total:527307116 used:9497280 free:514168636 shared:2125708 cache:3641200 avail:513544216 nid001872: 2022-01-26.21 5 4.0K /dev/shm nid001872: 2022-01-26.21 6 42M /tmp nid001872: 2022-01-26.21 7 =========================================== ------------------------------------------------------------------ Job 1054213.dbqs01 - /var/log/messages for job duration: nid001872: 2022-01-26A--:--:--.------+--:-- nid001872 #### nid001872 - Job 1054213.dbqs01 Runtime Data from /var/log/messages nid001872: 2022-01-26T19:27:13.741987+00:00 nid001872 prologue: MARK: Job 1054213.dbqs01 Start nid001872: 2022-01-26T19:27:13.742884+00:00 nid001872 prologue: Job 1054213.dbqs01 nodelist: nid001872 nid001872: 2022-01-26T19:27:13.743861+00:00 nid001872 prologue: Job 1054213.dbqs01 - Checking existing ASLR settings... nid001872: 2022-01-26T19:27:13.747360+00:00 nid001872 prologue: Job 1054213.dbqs01 - Checking palsd open file count... nid001872: 2022-01-26T19:27:13.833778+00:00 nid001872 prologue: Job 1054213.dbqs01 - Recording pre-job HSN counters... nid001872: 2022-01-26T19:27:13.896139+00:00 nid001872 prologue: Job 1054213.dbqs01 - Recording pre-job memory usage... nid001872: 2022-01-26T19:27:13.923516+00:00 nid001872 prologue: Job 1054213.dbqs01 - Killing any stray user processes... nid001872: 2022-01-26T19:27:13.998949+00:00 nid001872 prologue: Job 1054213.dbqs01 - Enabling turboboost... nid001872: 2022-01-26T19:27:14.001850+00:00 nid001872 prologue: Job 1054213.dbqs01 - Verifying post-boot workarounds... nid001872: 2022-01-26T19:27:14.141200+00:00 nid001872 prologue: Job 1054213.dbqs01 - Warchk Complete nid001872: 2022-01-26T19:27:14.143126+00:00 nid001872 prologue: Job 1054213.dbqs01 - Prologue complete. Execution time: 1 seconds nid001872: 2022-01-26T19:27:21.398302+00:00 nid001872 epilogue: Job 1054213.dbqs01 complete, running post-job actions. nid001872: 2022-01-26T19:27:21.408439+00:00 nid001872 epilogue: Job 1054213.dbqs01 - Recording post-job HSN counters... nid001872: 2022-01-26T19:27:21.468452+00:00 nid001872 epilogue: Job 1054213.dbqs01 - Recording post-job memory usage... nid001872: 2022-01-26T19:27:21.492507+00:00 nid001872 epilogue: Job 1054213.dbqs01 - Clearing /tmp... nid001872: 2022-01-26T19:27:21.536624+00:00 nid001872 epilogue: Job 1054213.dbqs01 - Clearing shared memory... nid001872: 2022-01-26T19:27:21.545720+00:00 nid001872 epilogue: Job 1054213.dbqs01 - Clearing memory cache... nid001872: 2022-01-26T19:27:21.841697+00:00 nid001872 kernel: [7515323.752594] drop_caches (202184): drop_caches: 3 nid001872: 2022-01-26T19:27:21.962096+00:00 nid001872 epilogue: Job 1054213.dbqs01 - Releasing Lustre Locks... nid001872: 2022-01-26T19:27:21.988005+00:00 nid001872 epilogue: MARK: Job 1054213.dbqs01 Complete. ------------------------------------------------------------------ Job 1054213.dbqs01 - dmesg output for job duration: nid001872: 2022-01-26A--:--:--.------+--:-- nid001872 #### nid001872 - Job 1054213.dbqs01 Runtime Data from dmesg nid001872: [Wed Jan 26 19:21:02 2022] MARK: Job 1054213.dbqs01 Start nid001872: [Wed Jan 26 19:21:10 2022] MARK: Job 1054213.dbqs01 Complete. nid001872: [Wed Jan 26 19:21:10 2022] drop_caches (202184): drop_caches: 3 ------------------------------------------------------------------ Job 1054213.dbqs01 - Pre/Post job diff on HSN (MLX) Counters: nid001872: 2022-01-26A--:--:--.------+--:-- nid001872 #### nid001872 - Job 1054213.dbqs01 HSN0 MLX Counter Post-Job Difference nid001872: multicast: 74220828 | multicast: 74220850 nid001872: port_rcv_data: 581543476100073 | port_rcv_data: 581543476111145 nid001872: port_rcv_packets: 908226485954 | port_rcv_packets: 908226486043 nid001872: port_xmit_data: 621671026543282 | port_xmit_data: 621671026552651 nid001872: port_xmit_packets: 916539230200 | port_xmit_packets: 916539230288 nid001872: rx_bytes: 627311691806 | rx_bytes: 627313166339 nid001872: rx_packets: 201957965 | rx_packets: 201963954 nid001872: rx_write_requests: 1876397596 | rx_write_requests: 1876397599 nid001872: tx_bytes: 128430875323 | tx_bytes: 128432383460 nid001872: tx_packets: 88889419 | tx_packets: 88899468 nid001872: unicast_rcv_packets: 908226485954 | unicast_rcv_packets: 908226486043 nid001872: unicast_xmit_packets: 916539230200 | unicast_xmit_packets: 916539230288 ------------------------------------------------------------------ Job 1054213.dbqs01 - Job summary: Job Id: 1054213.dbqs01 Job_Name = SREF Job_Owner = Jun.Du@dlogin04.dogwood.wcoss2.ncep.noaa.gov resources_used.cpupercent = 0 resources_used.cput = 00:00:00 resources_used.mem = 0kb resources_used.ncpus = 28 resources_used.vmem = 0kb resources_used.walltime = 00:00:00 job_state = R queue = dev server = dbqs01 Account_Name = SREF-DEV Checkpoint = u ctime = Wed Jan 26 19:27:12 2022 Error_Path = dlogin04.dogwood.wcoss2.ncep.noaa.gov:/lfs/h2/emc/lam/noscrub/ Jun.Du/sref.v7_cray/run/SREF.e1054213 exec_host = nid001872/0*28 exec_vnode = (nid001872:ncpus=28:mem=20971520kb) Hold_Types = n Join_Path = n Keep_Files = oed Mail_Points = a mtime = Wed Jan 26 19:27:14 2022 Output_Path = dlogin04.dogwood.wcoss2.ncep.noaa.gov:/lfs/h2/emc/lam/noscrub /Jun.Du/sref.v7_cray/run/SREF.o1054213 Priority = 0 qtime = Wed Jan 26 19:27:12 2022 Rerunable = True Resource_List.alvl = 2 Resource_List.aslr = True Resource_List.debug = True Resource_List.dfs = False Resource_List.hyper = False Resource_List.mem = 20gb Resource_List.ncpus = 28 Resource_List.nodect = 1 Resource_List.place = free Resource_List.select = 1:ncpus=28:mem=20GB Resource_List.turbo = True Resource_List.walltime = 01:00:00 schedselect = 1:ncpus=28:mem=20GB:prepost=False stime = Wed Jan 26 19:27:13 2022 session_id = 201599 jobdir = /u/Jun.Du substate = 42 Variable_List = PBS_O_HOME=/u/Jun.Du,PBS_O_LANG=en_US.UTF-8, PBS_O_LOGNAME=Jun.Du, PBS_O_PATH=/apps/spack/grads/2.2.1/cce/11.0.4/6nvi5fhtdfnb6v6qngbvdplx ta4jofvt/bin:/apps/spack/cmake/3.20.2/intel/19.1.3.304/utnbptm3hrf7gppz tidueu4jogfgemut/bin:/apps/spack/git/2.29.0/gcc/10.2.0/rkn6xaqbrdlim3u4 it2l6ljqszg62s25/bin:/usrx/local/nceplibs/dev/lib/pygrib2/bin:/apps/spa ck/python/3.8.6/intel/19.1.3.304/pjn2nzkjvqgmjw4hmyz43v5x4jbxjzpk/bin:/ apps/spack/libpng/1.6.37/intel/19.1.3.304/4ohkronuhlyherusoszzrmur5ewvl wzh/bin:/apps/spack/imagemagick/7.0.8-7/cce/11.0.1/fyjvsbwngyzlsiluc4ud bnxkhlbwkzc3/bin:/apps/ops/prod/libs/intel/19.1.3.304/grib_util/1.2.2/b in:/apps/ops/prod/nco/core/prod_util.v2.0.9/ush:/apps/spack/libjpeg/9c/ intel/19.1.3.304/jkr3isi257ktoouprwaxcn4twtye747z/bin:/apps/spack/subve rsion/1.14.0/gcc/10.2.0/bpzu25qwc4qywkfrjhi2vw64fqwt5y36/bin:/apps/ops/ test/nco/core/rocoto.v1.3.4/bin:/apps/ops/prod/nco/intel/19.1.3.304/gem pak.v7.14.0/nawips/os/linux5.3.18_x86_64/bin:/apps/ops/prod/libs/intel/ 19.1.3.304/wgrib2/2.0.8/bin:/opt/cray/pe/mpich/8.1.7/ofi/intel/19.0/bin :/pe/intel/compilers_and_libraries_2020.4.304/linux/bin/intel64:/pe/int el/compilers_and_libraries_2020.4.304/linux/bin:/pe/intel/compilers_and _libraries_2020.4.304/linux/mpi/intel64/bin:/pe/intel/debugger_2020/gdb /intel64/bin:/opt/cray/pe/craype/2.7.8/bin:/apps/spack/rsync/3.2.2/gcc/ 10.2.0/irtdtlqeqapryosluwypn42xq5ut7rzu/bin:/opt/cray/libfabric/1.11.0. 4.91/bin:/opt/clmgr/sbin:/opt/clmgr/bin:/opt/sgi/sbin:/opt/sgi/bin:/usr /local/bin:/usr/bin:/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/pbs/bi n:/sbin:/u/Jun.Du/bin:/lfs/h2/emc/lam/noscrub/Jun.Du/dbrowse:/usrx/loca l/emc_rocoto/1.2.3/bin:/opt/cray/pe/bin:/apps/ops/prod/nco/intel/19.1.3 .304/gempak.v7.14.0/nawips/bin:/apps/ops/prod/nco/intel/19.1.3.304/gemp ak.v7.14.0/nawips/scripts/ez:/apps/ops/prod/nco/intel/19.1.3.304/gempak .v7.14.0/nawips/scripts/nawips:/apps/ops/prod/nco/intel/19.1.3.304/gemp ak.v7.14.0/nawips/scripts/prnt:/apps/ops/prod/nco/intel/19.1.3.304/gemp ak.v7.14.0/nawips/scripts/decoder,PBS_O_MAIL=/var/spool/mail/Jun.Du, PBS_O_SHELL=/bin/bash, PBS_O_WORKDIR=/lfs/h2/emc/lam/noscrub/Jun.Du/sref.v7_cray/run, PBS_O_SYSTEM=Linux,PBS_O_QUEUE=dev, PBS_O_HOST=dlogin04.dogwood.wcoss2.ncep.noaa.gov euser = Jun.Du egroup = emc hashname = 1054213.dbqs01 queue_rank = 1643225233001 queue_type = E comment = Job run at Wed Jan 26 at 19:27 on (nid001872:ncpus=28:mem=2097152 0kb) etime = Wed Jan 26 19:27:12 2022 umask = 22 run_count = 1 eligible_time = 00:00:00 accrue_type = 3 Submit_arguments = SREF_POST_ARW_ctl.qsub project = SREF-DEV run_version = 1 Submit_Host = dlogin04.dogwood.wcoss2.ncep.noaa.gov ------------------------------------------------------------------ Job 1054213.dbqs01 - PBS tracejob output (for parent mom node only): Job: 1054213.dbqs01 01/26/2022 19:27:13 M running prologue 01/26/2022 19:27:14 M Started, pid = 201599 01/26/2022 19:27:21 M task 00000001 terminated 01/26/2022 19:27:21 M Terminated 01/26/2022 19:27:21 M task 00000001 cput=00:00:06 01/26/2022 19:27:21 M kill_job 01/26/2022 19:27:21 M nid001872 cput=00:00:06 mem=0kb 01/26/2022 19:27:21 M running epilogue ------------------------------------------------------------------ To see full PBS log data, run: /sfs/admin/scripts/tracejob.sh 1054213.dbqs01 ================================================================== END - DEBUG INFO ==================================================================