How to verify cause of system malfunction, probably motherboard

Issues related to hardware problems
Post Reply
CZa
Posts: 8
Joined: 2020/05/15 10:18:05

How to verify cause of system malfunction, probably motherboard

Post by CZa » 2020/05/15 11:21:59

Hello everyone,

I have got a Supermicro SYS-6028R-TRT server running CentOS 7. A few weeks ago, the system started acting up: it started to freeze unexpectedly after a couple of days working normally. When that happened, it could not be reset through IPMI and it had to be power cycled, going back to normal after that for a few more days. It did not leave any information on the logs regarding the cause of the reset.

After a few cycles of the behaviour I described above, the system stopped booting up. It stated to get stuck in the boot process forever, periodically showing unintelligible (to me) kernel messages including call stacks on the screen. I contacted Supermicro support and they advised I updated the BIOS and IPMI, which I did with no success. Trying to decipher the messages on the screen I even recorded them using a serial console, to no avail. I could start in emergency mode but not in rescue mode. I looked for information on what specific services are activated in rescue mode in order to try and activate them manually one by one to see which caused the "crash", but I could not find that information.

--------------------------------------

Although I suspected a hardware problem from the beginning, I decided to make a fresh CentOS 7 install on a new hard drive to rule out a problem in the old installation.

On the new install, the system boots up to a terminal session without a problem. From there, I ran stress-ng to analyse its robustness, but the system "crashes" after a few minutes, logging me out of session. The advantage here is that the system stays up and I can still access all the logs. Nevertheless, I cannot extract any valuable information pointing at the root cause of the problem.

The output of journalctl -a is in the attached file. In that file, a "crash" takes place. The system boots up and I log in on tty1 at 05:29:29 and run stress-ng:

Code: Select all

May 14 05:29:29 localhost.localdomain login[911]: ROOT LOGIN ON tty1
May 14 05:29:37 localhost.localdomain stress-ng[1881]: invoked with 'stress-ng -a 0' by user 0
At 05:43:06, the system had already crashed and logged me out, so I logged back in and that's the end of the file. Taking a look at the lines before the login at the end of the file (after the crash), I can see that services are being killed and restarted, but I cannot make sense of what started that behaviour. Please see an extract fo the attached file below:
May 14 05:32:27 localhost.localdomain kernel: INFO: task stress-ng-io:4270 blocked for more than 120 seconds.
May 14 05:32:27 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:32:27 localhost.localdomain kernel: stress-ng-io D ffff8bf21d3df080 0 4270 1881 0x00000080
May 14 05:32:27 localhost.localdomain kernel: Call Trace:
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc60c>] ? __queue_work+0x13c/0x3f0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc9ba>] ? __queue_delayed_work+0xaa/0x1a0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebcd70>] ? try_to_grab_pending+0x100/0x160
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb207e09c>] sync_inodes_sb+0xbc/0x3d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585922>] ? __schedule+0x402/0x840
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d99>] sync_inodes_one_sb+0x19/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb20514e3>] iterate_supers+0xc3/0x120
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2083074>] sys_sync+0x44/0xb0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2592ed2>] system_call_fastpath+0x25/0x2a
May 14 05:32:27 localhost.localdomain kernel: INFO: task stress-ng-iomix:4580 blocked for more than 120 seconds.
May 14 05:32:27 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:32:27 localhost.localdomain kernel: stress-ng-iomix D ffff8bea1da51900 0 4580 1949 0x00000080
May 14 05:32:27 localhost.localdomain kernel: Call Trace:
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb207e09c>] sync_inodes_sb+0xbc/0x3d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585922>] ? __schedule+0x402/0x840
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d99>] sync_inodes_one_sb+0x19/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb20514e3>] iterate_supers+0xc3/0x120
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2083074>] sys_sync+0x44/0xb0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2592ed2>] system_call_fastpath+0x25/0x2a
May 14 05:32:27 localhost.localdomain kernel: INFO: task stress-ng-iomix:5626 blocked for more than 120 seconds.
May 14 05:32:27 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:32:27 localhost.localdomain kernel: stress-ng-iomix D ffff8bea06e812c0 0 5626 4273 0x00000080
May 14 05:32:27 localhost.localdomain kernel: Call Trace:
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc60c>] ? __queue_work+0x13c/0x3f0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1edb6d0>] ? try_to_wake_up+0x190/0x390
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc9ba>] ? __queue_delayed_work+0xaa/0x1a0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebcd70>] ? try_to_grab_pending+0x100/0x160
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb207e09c>] sync_inodes_sb+0xbc/0x3d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc9ba>] ? __queue_delayed_work+0xaa/0x1a0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebcd70>] ? try_to_grab_pending+0x100/0x160
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d99>] sync_inodes_one_sb+0x19/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb20514e3>] iterate_supers+0xc3/0x120
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2083074>] sys_sync+0x44/0xb0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2592ed2>] system_call_fastpath+0x25/0x2a
May 14 05:32:27 localhost.localdomain kernel: INFO: task stress-ng-io:11920 blocked for more than 120 seconds.
May 14 05:32:27 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:32:27 localhost.localdomain kernel: stress-ng-io D ffff8bf1d37bf6c0 0 11920 1881 0x00000080
May 14 05:32:27 localhost.localdomain kernel: Call Trace:
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc60c>] ? __queue_work+0x13c/0x3f0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebc9ba>] ? __queue_delayed_work+0xaa/0x1a0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1ebcd70>] ? try_to_grab_pending+0x100/0x160
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb207e09c>] sync_inodes_sb+0xbc/0x3d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585922>] ? __schedule+0x402/0x840
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d99>] sync_inodes_one_sb+0x19/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb20514e3>] iterate_supers+0xc3/0x120
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2083074>] sys_sync+0x44/0xb0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2592ed2>] system_call_fastpath+0x25/0x2a
May 14 05:32:27 localhost.localdomain kernel: INFO: task stress-ng-iomix:15207 blocked for more than 120 seconds.
May 14 05:32:27 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:32:27 localhost.localdomain kernel: stress-ng-iomix D ffff8bf1b88d1070 0 15207 11923 0x00000080
May 14 05:32:27 localhost.localdomain kernel: Call Trace:
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1e2b621>] ? __switch_to+0x151/0x580
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb207e09c>] sync_inodes_sb+0xbc/0x3d0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2585922>] ? __schedule+0x402/0x840
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d80>] ? generic_write_sync+0x70/0x70
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2082d99>] sync_inodes_one_sb+0x19/0x20
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb20514e3>] iterate_supers+0xc3/0x120
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2083074>] sys_sync+0x44/0xb0
May 14 05:32:27 localhost.localdomain kernel: [<ffffffffb2592ed2>] system_call_fastpath+0x25/0x2a
May 14 05:33:40 localhost.localdomain chronyd[901]: Selected source 212.183.233.76
May 14 05:33:43 localhost.localdomain stress-ng[2037]: memory (MB): total 64154.54, free 37585.26, shared 0.00, buffer 2.08, swap 32192.00, free swap 32192.00
May 14 05:33:43 localhost.localdomain stress-ng[6705]: memory (MB): total 64154.54, free 37466.51, shared 0.00, buffer 2.08, swap 32192.00, free swap 32192.00
May 14 05:33:44 localhost.localdomain stress-ng[18782]: memory (MB): total 64154.54, free 37424.55, shared 0.00, buffer 2.08, swap 32192.00, free swap 32192.00
May 14 05:33:44 localhost.localdomain stress-ng[18718]: memory (MB): total 64154.54, free 37423.97, shared 0.00, buffer 2.08, swap 32192.00, free swap 32192.00
May 14 05:34:12 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:34:12 localhost.localdomain kernel: VFS: file-max limit 6510517 reached
May 14 05:34:12 localhost.localdomain kernel: VFS: file-max limit 6510517 reached
May 14 05:34:12 localhost.localdomain kernel: VFS: file-max limit 6510517 reached
May 14 05:34:22 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:34:22 localhost.localdomain kernel: VFS: file-max limit 6510517 reached
May 14 05:34:25 localhost.localdomain kernel: INFO: task kworker/4:0:31 blocked for more than 120 seconds.
May 14 05:34:25 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:34:25 localhost.localdomain kernel: kworker/4:0 D ffff8bf09f06abc0 0 31 2 0x00000000
May 14 05:34:25 localhost.localdomain kernel: Workqueue: xfs-sync/dm-0 xfs_log_worker [xfs]
May 14 05:34:25 localhost.localdomain kernel: Call Trace:
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1fe114e>] ? clear_bdi_congested+0x3e/0x70
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2152092>] ? __freed_request+0x92/0xa0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebee4a>] flush_work+0x10a/0x1b0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebbb60>] ? move_linked_works+0x90/0x90
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066e6ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ee539c>] ? dequeue_entity+0x11c/0x5c0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ee2e89>] ? pick_next_entity+0xa9/0x190
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ee103c>] ? set_next_entity+0x3c/0xe0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066c8a6>] ? xfs_log_worker+0x36/0x100 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066c61b>] xfs_log_force+0x8b/0x2e0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066c8a6>] xfs_log_worker+0x36/0x100 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebe6bf>] process_one_work+0x17f/0x440
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebf7d6>] worker_thread+0x126/0x3c0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec6691>] kthread+0xd1/0xe0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec65c0>] ? insert_kthread_work+0x40/0x40
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2592d37>] ret_from_fork_nospec_begin+0x21/0x21
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec65c0>] ? insert_kthread_work+0x40/0x40
May 14 05:34:25 localhost.localdomain kernel: INFO: task kworker/u25:8:407 blocked for more than 120 seconds.
May 14 05:34:25 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:34:25 localhost.localdomain kernel: kworker/u25:8 D ffff8be93d836a40 0 407 2 0x00000000
May 14 05:34:25 localhost.localdomain kernel: Workqueue: writeback bdi_writeback_workfn (flush-253:0)
May 14 05:34:25 localhost.localdomain kernel: Call Trace:
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb22e9c0e>] ? scsi_init_io+0x4e/0x200
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc06693a3>] xlog_grant_head_wait+0xb3/0x1a0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0669524>] xlog_grant_head_check+0x94/0x100 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066d15c>] xfs_log_reserve+0xcc/0x1d0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0667067>] xfs_trans_reserve+0x187/0x1f0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066848e>] xfs_trans_alloc+0xbe/0x140 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0657e22>] xfs_iomap_write_allocate+0x1f2/0x380 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc063b47b>] ? xfs_iext_lookup_extent+0x12b/0x160 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0641506>] xfs_map_blocks+0x1a6/0x220 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0642544>] xfs_do_writepage+0x174/0x550 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1fca15c>] write_cache_pages+0x21c/0x470
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc06423d0>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc064239b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1fcb1b1>] do_writepages+0x21/0x50
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb207cbd0>] __writeback_single_inode+0x40/0x260
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec7745>] ? wake_up_bit+0x25/0x30
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb207d764>] writeback_sb_inodes+0x1c4/0x430
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb207da6f>] __writeback_inodes_wb+0x9f/0xd0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb207df53>] wb_writeback+0x263/0x2f0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb207e995>] bdi_writeback_workfn+0x115/0x460
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2585922>] ? __schedule+0x402/0x840
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebe6bf>] process_one_work+0x17f/0x440
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebf7d6>] worker_thread+0x126/0x3c0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec6691>] kthread+0xd1/0xe0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec65c0>] ? insert_kthread_work+0x40/0x40
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2592d37>] ret_from_fork_nospec_begin+0x21/0x21
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec65c0>] ? insert_kthread_work+0x40/0x40
May 14 05:34:25 localhost.localdomain kernel: INFO: task xfsaild/dm-0:531 blocked for more than 120 seconds.
May 14 05:34:25 localhost.localdomain kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 14 05:34:25 localhost.localdomain kernel: xfsaild/dm-0 D ffff8bf2537fddc0 0 531 2 0x00000000
May 14 05:34:25 localhost.localdomain kernel: Call Trace:
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2585d89>] schedule+0x29/0x70
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2583891>] schedule_timeout+0x221/0x2d0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1edb6d0>] ? try_to_wake_up+0x190/0x390
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb258613d>] wait_for_completion+0xfd/0x140
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1edb990>] ? wake_up_state+0x20/0x20
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebee4a>] flush_work+0x10a/0x1b0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ebbb60>] ? move_linked_works+0x90/0x90
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066e6ba>] xlog_cil_force_lsn+0x8a/0x210 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1eaea02>] ? del_timer_sync+0x52/0x60
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb25837e0>] ? schedule_timeout+0x170/0x2d0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0678be0>] ? xfsaild+0x180/0x760 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc066c61b>] xfs_log_force+0x8b/0x2e0 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0678a60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0678be0>] xfsaild+0x180/0x760 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffc0678a60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec6691>] kthread+0xd1/0xe0
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec65c0>] ? insert_kthread_work+0x40/0x40
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb2592d37>] ret_from_fork_nospec_begin+0x21/0x21
May 14 05:34:25 localhost.localdomain kernel: [<ffffffffb1ec65c0>] ? insert_kthread_work+0x40/0x40
May 14 05:34:32 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:34:42 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:34:46 localhost.localdomain kernel: VFS: file-max limit 6510517 reached
May 14 05:34:52 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:35:02 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:35:12 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:35:22 localhost.localdomain /usr/sbin/irqbalance[856]: WARNING cant open /proc/stat. balacing is broken
May 14 05:37:34 localhost.localdomain kernel: perf: interrupt took too long (2511 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
May 14 05:38:23 localhost.localdomain systemd-journal[32635]: Runtime journal is using 16.0M (max allowed 3.1G, trying to leave 4.0G free of 31.3G available → current limit 3.1G).
May 14 05:38:23 localhost.localdomain kernel: smpboot: CPU 8 is now offline
May 14 05:38:23 localhost.localdomain kernel: audit: netlink_unicast sending to audit_pid=831 returned error: -111
May 14 05:38:23 localhost.localdomain kernel: audit: audit_lost=1 audit_rate_limit=0 audit_backlog_limit=8192
May 14 05:38:23 localhost.localdomain kernel: audit: audit_pid=831 reset
May 14 05:38:23 localhost.localdomain kernel: type=1326 audit(1589449044.192:462): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=29153 comm="stress-ng-secco" sig=31 arch=c000003e syscall=1 compat=0 ip=0x7f77dc07d6e0 code=0x0
May 14 05:38:23 localhost.localdomain kernel: type=1701 audit(1589449044.192:463): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=29153 comm="stress-ng-secco" reason="memory violation" sig=31
May 14 05:38:23 localhost.localdomain kernel: type=1326 audit(1589449046.686:464): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=6249 comm="stress-ng-secco" sig=31 arch=c000003e syscall=1 compat=0 ip=0x7f77dc07d6e0 code=0x0
May 14 05:38:23 localhost.localdomain kernel: type=1701 audit(1589449046.686:465): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=6249 comm="stress-ng-secco" reason="memory violation" sig=31
May 14 05:38:23 localhost.localdomain kernel: type=1326 audit(1589449049.764:466): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=30451 comm="stress-ng-secco" sig=31 arch=c000003e syscall=1 compat=0 ip=0x7f77dc07d6e0 code=0x0
May 14 05:38:23 localhost.localdomain kernel: type=1701 audit(1589449049.764:467): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=30451 comm="stress-ng-secco" reason="memory violation" sig=31
May 14 05:38:23 localhost.localdomain kernel: type=1326 audit(1589449051.556:468): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=20741 comm="stress-ng-secco" sig=31 arch=c000003e syscall=1 compat=0 ip=0x7f77dc07d6e0 code=0x0
May 14 05:38:23 localhost.localdomain kernel: type=1701 audit(1589449051.556:469): auid=0 uid=0 gid=0 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=20741 comm="stress-ng-secco" reason="memory violation" sig=31
May 14 05:38:23 localhost.localdomain systemd-journal[32635]: Journal started
May 14 05:38:23 localhost.localdomain rsyslogd[32637]: [origin software="rsyslogd" swVersion="8.24.0-52.el7" x-pid="32637" x-info="http://www.rsyslog.com"] start
May 14 05:38:18 localhost.localdomain systemd[1]: systemd-journald.service: main process exited, code=killed, status=9/KILL
May 14 05:38:18 localhost.localdomain systemd[1]: Unit systemd-journald.service entered failed state.
May 14 05:38:23 localhost.localdomain systemd-udevd[32631]: starting version 219
May 14 05:38:23 localhost.localdomain dbus[32638]: [system] Successfully activated service 'org.freedesktop.systemd1'
May 14 05:38:23 localhost.localdomain systemd-logind[32634]: New seat seat0.
May 14 05:38:23 localhost.localdomain systemd[1]: Starting Flush Journal to Persistent Storage...
May 14 05:38:23 localhost.localdomain systemd[1]: Started Login Service.
May 14 05:38:23 localhost.localdomain systemd-logind[32634]: Watching system buttons on /dev/input/event1 (Power Button)
May 14 05:38:23 localhost.localdomain systemd-logind[32634]: Watching system buttons on /dev/input/event0 (Power Button)
May 14 05:38:23 localhost.localdomain systemd-logind[32634]: New session 2 of user root.
May 14 05:38:23 localhost.localdomain systemd[1]: Started Flush Journal to Persistent Storage.
May 14 05:38:34 localhost.localdomain NetworkManager[32632]: <info> [1589449114.2192] NetworkManager (version 1.18.4-3.el7) is starting... (after a restart)
May 14 05:38:34 localhost.localdomain kernel: audit_printk_skb: 471 callbacks suppressed
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=filter family=2 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=filter family=10 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=raw family=2 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=security family=2 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=mangle family=2 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=nat family=2 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=raw family=10 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=security family=10 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=mangle family=10 entries=0
May 14 05:38:34 localhost.localdomain kernel: type=1325 audit(1589448659.257:550): table=nat family=10 entries=0
May 14 05:38:34 localhost.localdomain NetworkManager[32632]: <info> [1589449114.2202] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 10-slaves-order.conf)
May 14 05:38:34 localhost.localdomain NetworkManager[32632]: <info> [1589449114.2340] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
May 14 05:38:34 localhost.localdomain NetworkManager[32632]: <info> [1589449114.2349] manager[0x55db3c709040]: monitoring kernel firmware directory '/lib/firmware'.
May 14 05:38:34 localhost.localdomain dbus[32638]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service'
May 14 05:38:34 localhost.localdomain systemd[1]: Started Network Manager.
May 14 05:38:34 localhost.localdomain systemd[1]: Starting Hostname Service...
May 14 05:38:34 localhost.localdomain systemd[1]: Starting Network Manager Wait Online...
May 14 05:38:39 localhost.localdomain kernel: audit_printk_skb: 2457 callbacks suppressed
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=filter family=2 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=filter family=10 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=raw family=2 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=security family=2 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=mangle family=2 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=nat family=2 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=raw family=10 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=security family=10 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=mangle family=10 entries=0
May 14 05:38:39 localhost.localdomain kernel: type=1325 audit(1589448675.162:620): table=nat family=10 entries=0
May 14 05:38:44 localhost.localdomain kernel: audit_printk_skb: 2562 callbacks suppressed
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=filter family=2 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=filter family=10 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=raw family=2 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=security family=2 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=mangle family=2 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=nat family=2 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=raw family=10 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=security family=10 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=mangle family=10 entries=0
May 14 05:38:44 localhost.localdomain kernel: type=1325 audit(1589448682.576:692): table=nat family=10 entries=0
May 14 05:38:48 localhost.localdomain systemd[1]: crond.service holdoff time over, scheduling restart.
May 14 05:38:48 localhost.localdomain systemd[1]: Stopped Command Scheduler.
May 14 05:38:48 localhost.localdomain systemd[1]: Started Command Scheduler.
May 14 05:38:48 localhost.localdomain crond[377]: (CRON) INFO (RANDOM_DELAY will be scaled with factor 56% if used.)
May 14 05:38:48 localhost.localdomain crond[377]: (CRON) INFO (running with inotify support)
May 14 05:38:48 localhost.localdomain crond[377]: (CRON) INFO (@reboot jobs will be run at computer's startup.)
May 14 05:38:49 localhost.localdomain kernel: audit_printk_skb: 2283 callbacks suppressed
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=filter family=2 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=filter family=10 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=raw family=2 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=security family=2 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=mangle family=2 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=nat family=2 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=raw family=10 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=security family=10 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=mangle family=10 entries=0
May 14 05:38:49 localhost.localdomain kernel: type=1325 audit(1589448924.844:759): table=nat family=10 entries=0
May 14 05:38:55 localhost.localdomain kernel: audit_printk_skb: 2562 callbacks suppressed
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=filter family=2 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=filter family=10 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=raw family=2 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=security family=2 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=mangle family=2 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=nat family=2 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=raw family=10 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=security family=10 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=mangle family=10 entries=0
May 14 05:38:55 localhost.localdomain kernel: type=1325 audit(1589449055.742:831): table=nat family=10 entries=0
May 14 05:38:55 localhost.localdomain systemd-logind[32634]: Removed session 2.
May 14 05:38:55 localhost.localdomain systemd[1]: Removed slice User Slice of root.
May 14 05:38:55 localhost.localdomain dbus[32638]: [system] Successfully activated service 'org.freedesktop.hostname1'
May 14 05:38:55 localhost.localdomain systemd[1]: Started Hostname Service.
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.5842] hostname: hostname: using hostnamed
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.5846] hostname: hostname changed from (none) to "localhost.localdomain"
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.5869] dns-mgr[0x55db3c6e8220]: init: dns=default,systemd-resolved rc-manager=file
May 14 05:38:55 localhost.localdomain dbus[32638]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
May 14 05:38:55 localhost.localdomain systemd[1]: Starting Network Manager Script Dispatcher Service...
May 14 05:38:55 localhost.localdomain dbus[32638]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 14 05:38:55 localhost.localdomain systemd[1]: Started Network Manager Script Dispatcher Service.
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6096] settings: Loaded settings plugin: SettingsPluginIfcfg ("/usr/lib64/NetworkManager/1.18.4-3.el7/libnm-settings-plugin-ifcfg-rh.so")
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6099] settings: Loaded settings plugin: NMSIbftPlugin ("/usr/lib64/NetworkManager/1.18.4-3.el7/libnm-settings-plugin-ibft.so")
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6099] settings: Loaded settings plugin: NMSKeyfilePlugin (internal)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6123] ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-eno1 (c79556c4-42b6-4a5f-8cf9-f96566efe1f7,"eno1")
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6135] ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-eno2 (b3bb6804-24b0-4140-8101-fb0dadd3552c,"eno2")
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6146] ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-ens1f0 (28a7c187-0730-49b4-bb8a-1fb77dc3cf54,"ens1f0")
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6157] ifcfg-rh: new connection /etc/sysconfig/network-scripts/ifcfg-ens1f1 (24247351-253c-429e-bf95-63d0879bcc0f,"ens1f1")
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6203] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6205] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6207] manager: Networking is enabled by state file
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6211] dhcp-init: Using DHCP client 'dhclient'
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:1 'hostname': new request (3 scripts)
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:1 'hostname': start running ordered scripts...
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6233] Loaded device plugin: NMTeamFactory (/usr/lib64/NetworkManager/1.18.4-3.el7/libnm-device-plugin-team.so)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6251] device (lo): carrier: link connected
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6256] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6276] manager: (ens1f0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6293] device (ens1f0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
May 14 05:38:55 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): ens1f0: link is not ready
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:2 'connectivity-change': new request (3 scripts)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6351] manager: (ens1f1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6365] device (ens1f1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
May 14 05:38:55 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): ens1f1: link is not ready
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6407] device (eno1): carrier: link connected
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6411] manager: (eno1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/4)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6433] manager: (eno1): assume: will attempt to assume matching connection 'eno1' (c79556c4-42b6-4a5f-8cf9-f96566efe1f7) (indicated)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6434] device (eno1): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6452] device (eno1): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6470] device (eno1): Activation: starting connection 'eno1' (c79556c4-42b6-4a5f-8cf9-f96566efe1f7)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6480] manager: (eno2): new Ethernet device (/org/freedesktop/NetworkManager/Devices/5)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6495] device (eno2): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
May 14 05:38:55 localhost.localdomain kernel: IPv6: ADDRCONF(NETDEV_UP): eno2: link is not ready
May 14 05:38:55 localhost.localdomain dbus[32638]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkit.service'
May 14 05:38:55 localhost.localdomain systemd[1]: Starting Authorization Manager...
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6578] device (eno1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6586] device (eno1): state change: prepare -> config (reason 'none', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6600] device (eno1): state change: config -> ip-config (reason 'none', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6614] dhcp4 (eno1): activation: beginning transaction (timeout in 45 seconds)
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:2 'connectivity-change': start running ordered scripts...
May 14 05:38:55 localhost.localdomain polkitd[433]: Started polkitd version 0.112
May 14 05:38:55 localhost.localdomain polkitd[433]: Loading rules from directory /etc/polkit-1/rules.d
May 14 05:38:55 localhost.localdomain polkitd[433]: Loading rules from directory /usr/share/polkit-1/rules.d
May 14 05:38:55 localhost.localdomain polkitd[433]: Finished loading, compiling and executing 2 rules
May 14 05:38:55 localhost.localdomain dbus[32638]: [system] Successfully activated service 'org.freedesktop.PolicyKit1'
May 14 05:38:55 localhost.localdomain polkitd[433]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
May 14 05:38:55 localhost.localdomain systemd[1]: Started Authorization Manager.
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.6967] dhcp4 (eno1): dhclient started with pid 455
May 14 05:38:55 localhost.localdomain dhclient[455]: DHCPREQUEST on eno1 to 255.255.255.255 port 67 (xid=0x1eb66551)
May 14 05:38:55 localhost.localdomain dhclient[455]: DHCPACK from 192.168.1.1 (xid=0x1eb66551)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8429] dhcp4 (eno1): address 192.168.1.143
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8429] dhcp4 (eno1): plen 24 (255.255.255.0)
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8429] dhcp4 (eno1): gateway 192.168.1.1
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8429] dhcp4 (eno1): lease time 5905
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8430] dhcp4 (eno1): nameserver '192.168.1.1'
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8430] dhcp4 (eno1): domain name 'tsclab.uniovi.es'
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8430] dhcp4 (eno1): state changed unknown -> bound
May 14 05:38:55 localhost.localdomain dhclient[455]: bound to 192.168.1.143 -- renewal in 2388 seconds.
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8450] device (eno1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8467] device (eno1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8472] device (eno1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'assume')
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8489] manager: NetworkManager state is now CONNECTED_LOCAL
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8778] manager: NetworkManager state is now CONNECTED_SITE
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8780] policy: set 'eno1' (eno1) as default for IPv4 routing and DNS
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8944] device (eno1): Activation: successful, device activated.
May 14 05:38:55 localhost.localdomain NetworkManager[32632]: <info> [1589449135.8957] manager: NetworkManager state is now CONNECTED_GLOBAL
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:3 'up' [eno1]: new request (3 scripts)
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:3 'up' [eno1]: start running ordered scripts...
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:4 'connectivity-change': new request (3 scripts)
May 14 05:38:55 localhost.localdomain nm-dispatcher[423]: req:4 'connectivity-change': start running ordered scripts...
May 14 05:39:01 localhost.localdomain NetworkManager[32632]: <info> [1589449141.6513] manager: startup complete
May 14 05:39:01 localhost.localdomain systemd[1]: Started Network Manager Wait Online.
May 14 05:39:01 localhost.localdomain kernel: audit_printk_skb: 339 callbacks suppressed
May 14 05:39:01 localhost.localdomain kernel: type=1130 audit(1589449141.662:844): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-wait-online comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 14 05:39:03 localhost.localdomain systemd[1]: sshd.service holdoff time over, scheduling restart.
May 14 05:39:03 localhost.localdomain systemd[1]: Stopped OpenSSH server daemon.
May 14 05:39:03 localhost.localdomain kernel: type=1130 audit(1589449143.499:845): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 14 05:39:03 localhost.localdomain kernel: type=1131 audit(1589449143.499:846): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 14 05:39:03 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
May 14 05:39:03 localhost.localdomain sshd[509]: Server listening on 0.0.0.0 port 22.
May 14 05:39:03 localhost.localdomain sshd[509]: Server listening on :: port 22.
May 14 05:39:03 localhost.localdomain kernel: type=1130 audit(1589449143.530:847): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=sshd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 14 05:39:03 localhost.localdomain systemd[1]: Started OpenSSH server daemon.
May 14 05:39:06 localhost.localdomain kernel: type=1131 audit(1589449146.206:848): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 14 05:39:25 localhost.localdomain kernel: type=1131 audit(1589449165.621:849): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 14 05:39:32 localhost.localdomain kernel: type=2404 audit(1589449172.197:850): pid=551 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:30:e7:0e:d9:52:fd:06:59:1e:14:b1:6e:37:0c:ff:6e:75:7c:0c:d3:a7:55:b8:5c:11:3f:79:c4:aa:09:e6:5c direction=? spid=551 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
May 14 05:39:32 localhost.localdomain kernel: type=2404 audit(1589449172.197:851): pid=551 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:67:47:fe:1d:6c:62:3e:d1:59:1e:09:cd:df:db:5e:3b:98:3d:67:b8:eb:cc:d3:8d:6b:57:fc:b0:60:54:26:b7 direction=? spid=551 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
May 14 05:39:32 localhost.localdomain kernel: type=2404 audit(1589449172.197:852): pid=551 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=SHA256:4c:2b:6b:5f:5f:a5:84:01:37:e9:72:20:1a:3f:07:86:33:2d:92:21:e8:71:c6:10:7b:4f:e9:d6:52:64:89:d8 direction=? spid=551 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
May 14 05:39:32 localhost.localdomain kernel: type=2407 audit(1589449172.200:853): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-server cipher=chacha20-poly1305@openssh.com ksize=512 mac=<implicit> pfs=curve25519-sha256 spid=551 suid=74 rport=63568 laddr=192.168.1.143 lport=22 exe="/usr/sbin/sshd" hostname=? addr=192.168.1.161 terminal=? res=success'
May 14 05:39:32 localhost.localdomain kernel: type=2407 audit(1589449172.201:854): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=start direction=from-client cipher=chacha20-poly1305@openssh.com ksize=512 mac=<implicit> pfs=curve25519-sha256 spid=551 suid=74 rport=63568 laddr=192.168.1.143 lport=22 exe="/usr/sbin/sshd" hostname=? addr=192.168.1.161 terminal=? res=success'
May 14 05:39:32 localhost.localdomain kernel: type=1100 audit(1589449172.275:855): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=pubkey acct="root" exe="/usr/sbin/sshd" hostname=? addr=192.168.1.161 terminal=ssh res=failed'
May 14 05:39:34 localhost.localdomain kernel: type=1100 audit(1589449174.544:856): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_unix acct="root" exe="/usr/sbin/sshd" hostname=192.168.1.161 addr=192.168.1.161 terminal=ssh res=success'
May 14 05:39:34 localhost.localdomain sshd[550]: Accepted password for root from 192.168.1.161 port 63568 ssh2
May 14 05:39:34 localhost.localdomain kernel: type=1101 audit(1589449174.552:857): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="root" exe="/usr/sbin/sshd" hostname=192.168.1.161 addr=192.168.1.161 terminal=ssh res=success'
May 14 05:39:34 localhost.localdomain kernel: type=2404 audit(1589449174.553:858): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=session fp=? direction=both spid=551 suid=74 rport=63568 laddr=192.168.1.143 lport=22 exe="/usr/sbin/sshd" hostname=? addr=192.168.1.161 terminal=? res=success'
May 14 05:39:34 localhost.localdomain kernel: type=1100 audit(1589449174.555:859): pid=550 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=success acct="root" exe="/usr/sbin/sshd" hostname=? addr=192.168.1.161 terminal=ssh res=success'
May 14 05:39:34 localhost.localdomain systemd[1]: Created slice User Slice of root.
May 14 05:39:34 localhost.localdomain systemd-logind[32634]: New session 3 of user root.
May 14 05:39:34 localhost.localdomain systemd[1]: Started Session 3 of user root.
May 14 05:39:34 localhost.localdomain sshd[550]: pam_unix(sshd:session): session opened for user root by (uid=0)
May 14 05:43:05 localhost.localdomain kernel: audit_printk_skb: 30 callbacks suppressed
May 14 05:43:05 localhost.localdomain kernel: type=1100 audit(1589449385.743:870): pid=32636 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=pam_securetty,pam_unix acct="root" exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:05 localhost.localdomain kernel: type=1101 audit(1589449385.751:871): pid=32636 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="root" exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:05 localhost.localdomain kernel: type=1103 audit(1589449385.751:872): pid=32636 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_securetty,pam_unix acct="root" exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:05 localhost.localdomain kernel: type=1006 audit(1589449385.752:873): pid=32636 uid=0 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 tty=tty1 old-ses=4294967295 ses=4 res=1
May 14 05:43:05 localhost.localdomain kernel: type=2300 audit(1589449385.973:874): pid=32636 uid=0 auid=0 ses=4 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='pam: default-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 selected-context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:05 localhost.localdomain systemd-logind[32634]: New session 4 of user root.
May 14 05:43:05 localhost.localdomain systemd[1]: Started Session 4 of user root.
May 14 05:43:06 localhost.localdomain login[32636]: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
May 14 05:43:06 localhost.localdomain kernel: type=1105 audit(1589449386.001:875): pid=32636 uid=0 auid=0 ses=4 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_console,pam_selinux,pam_namespace,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="root" exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:06 localhost.localdomain kernel: type=1110 audit(1589449386.001:876): pid=32636 uid=0 auid=0 ses=4 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_securetty,pam_unix acct="root" exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:06 localhost.localdomain kernel: type=1112 audit(1589449386.002:877): pid=32636 uid=0 auid=0 ses=4 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=login id=0 exe="/usr/bin/login" hostname=localhost.localdomain addr=? terminal=tty1 res=success'
May 14 05:43:06 localhost.localdomain login[32636]: ROOT LOGIN ON tty1
Any comments or suggestions? I can run further tests if necessary. The support agent at Supermicro suspects there is a hardware issue with the motherboard, but I would like to verify that is the case because I fear I might buy a new motherboard only to discover the problem is somewhere else.

Any help will be much appreciated.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: How to verify cause of system malfunction, probably motherboard

Post by TrevorH » 2020/05/15 13:18:56

The stacktraces you posted here are pointing to a response time problem with your hard disk I/O but that maybe a side-effect of running this 'stress-ng' thing and may be what it does "normally". Never used it or even heard of it before.

I would start by downloading and running memtest86+ (or the free version of the commercial memtest86) and boot that and leave it running for 24h or more. See what that does. I'd also visually inspect the system to make sure that everything that is meant to spin is spinning freely and without problems - most especially the CPU fans.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

CZa
Posts: 8
Joined: 2020/05/15 10:18:05

Re: How to verify cause of system malfunction, probably motherboard

Post by CZa » 2020/05/16 14:46:00

Thanks for your reply TrevorH. I did run memtest86 (the version included in the CentOS installation ISO) successfully for a few hours, although not nearly 24, and the fans are running well. I will run memtest86 for more than 24 hours to see what happens.

On the other hand, please find below the messages printed on the screen during boot up with the old install.
[ 16.710897] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[ 16.749665] systemd[1]: Device dev-disk-by\x2dlabel-CentOS\x5cx207\x5cx20x86_64.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/host11/target11:0:0/11:0:0:0/block/sdb and /sys/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/host11/target11:0:0/11:0:0:0/block/sdb/sdb1
[ 16.778509] systemd[1]: Device dev-disk-by\x2duuid-2020\x2d04\x2d22\x2d00\x2d54\x2d00\x2d00.device appeared twice with different sysfs paths /sys/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/host11/target11:0:0/11:0:0:0/block/sdb and /sys/devices/pci0000:00/0000:00:14.0/usb1/1-10/1-10:1.0/host11/target11:0:0/11:0:0:0/block/sdb/sdb1
[ 17.043667] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx Queue count = 12
[ 17.112341] ixgbe 0000:03:00.1: PCI Express bandwidth of 32GT/s available
[ 17.119145] ixgbe 0000:03:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[ 17.166432] ixgbe 0000:03:00.1: MAC: 3, PHY: 0, PBA No: 030C00-000
[ 17.172625] ixgbe 0000:03:00.1: 00:25:90:f9:d5:4f
[ 17.422041] ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
[[0m[31m* [0m] (1 of 4) qqq (8s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...er\x2dtmp.device (9s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...er\x2dtmp.device (9s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (10s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (10s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (11s / 1min 30s)
[K[ [31m*[1;31m*[0m] (3 of 4) A start job is running for...5e536113.device (11s / 1min 30s)
[K[ [31m*[0m] (3 of 4) A start job is running for...5e536113.device (12s / 1min 30s)
[K[ [31m*[1;31m*[0m] (3 of 4) A start job is running for...5e536113.device (12s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (4 of 4) A start job is running for...vice Initialization (13s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (13s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (14s / 3min)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (14s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (15s / 1min 30s)
[K[[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (15s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (16s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (16s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (17s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (17s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (3 of 4) A start job is running for...5e536113.device (18s / 1min 30s)
[K[ [31m*[1;31m*[0m] (3 of 4) A start job is running for...5e536113.device (18s / 1min 30s)
[K[ [31m*[0m] (4 of 4) A start job is running for...vice Initialization (19s / 3min)
[K[ [31m*[1;31m*[0m] (4 of 4) A start job is running for...vice Initialization (19s / 3min)
[K[ [31m*[1;31m*[0m[31m*[0m] (4 of 4) A start job is running for...vice Initialization (20s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (20s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (21s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (21s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (22s / 1min 30s)
[K[[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (22s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (23s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (23s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (24s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (24s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (4 of 4) A start job is running for...vice Initialization (25s / 3min)
[K[ [31m*[1;31m*[0m] (4 of 4) A start job is running for...vice Initialization (25s / 3min)
[K[ [31m*[0m] (4 of 4) A start job is running for...vice Initialization (26s / 3min)
[K[ [31m*[1;31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (26s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (27s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (27s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (28s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (28s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (29s / 1min 30s)
[K[[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (29s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (30s / 1min 30s)[ 41.444372] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:4]
[ 41.450633] Modules linked in: lrw ixgbe igb ast(+) syscopyarea sysfillrect sysimgblt gf128mul glue_helper ablk_helper ttm drm_kms_helper drm iTCO_wdt usb_storage cryptd i2c_i801 iTCO_vendor_support lpc_ich mfd_core mei_me mei sg ptp pcspkr sb_edac edac_core ipmi_ssif i2c_algo_bit pps_core mdio i2c_core ipmi_si ipmi_msghandler ioatdma dca shpchp acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mxm_wmi crct10dif_pclmul crct10dif_common ahci crc32c_intel libahci libata wmi dm_mirror dm_region_hash dm_log dm_mod
[ 41.503758] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 3.10.0-327.36.1.el7.x86_64 #1
[ 41.511577] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 41.519053] Workqueue: events work_for_cpu_fn
[ 41.523423] task: ffff8808544ea280 ti: ffff880854568000 task.ti: ffff880854568000
[ 41.530891] RIP: 0010:[<ffffffff81309862>] [<ffffffff81309862>] ioread32+0x42/0x50
[ 41.538563] RSP: 0000:ffff88085456bd10 EFLAGS: 00000292
[ 41.543865] RAX: 00000000ffffffff RBX: 0000000000000000 RCX: 0000000000000000
[ 41.550990] RDX: ffffc90016a10000 RSI: ffffc90016a10000 RDI: ffffc90016a10000
[ 41.558114] RBP: ffff88085456bd40 R08: 0000000000000082 R09: 0000000000017382
[ 41.565239] R10: 0000000000000000 R11: ffff88085456ba1e R12: ffff88084fe1f800
[ 41.572361] R13: ffff88084fe1d000 R14: ffff88084fe1f800 R15: 0000000000000246
[ 41.579487] FS: 0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000
[ 41.587562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 41.593301] CR2: 00007f230d046118 CR3: 000000000194a000 CR4: 00000000001407f0
[ 41.600425] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 41.607548] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 41.614672] Stack:
[ 41.616683] ffffffffa03b5ac1 ffff88084fe1d000 0000000000000000 0000000000000000
[ 41.624133] ffffffffa03c0120 0000000000000000 ffff88085456bd68 ffffffffa045b82d
[ 41.631589] ffff88084fe1d000 ffff881052df8000 ffff881052df8098 ffff88085456bdb8
[ 41.639042] Call Trace:
[ 41.641494] [<ffffffffa03b5ac1>] ? ast_driver_load+0x591/0x860 [ast]
[ 41.647945] [<ffffffffa045b82d>] drm_dev_register+0xad/0x100 [drm]
[ 41.654220] [<ffffffffa045e6f9>] drm_get_pci_dev+0x89/0x210 [drm]
[ 41.660400] [<ffffffffa03b51c5>] ast_pci_probe+0x15/0x20 [ast]
[ 41.666321] [<ffffffff813280d5>] local_pci_probe+0x45/0xa0
[ 41.671889] [<ffffffff8109a1b4>] work_for_cpu_fn+0x14/0x20
[ 41.677455] [<ffffffff8109d69b>] process_one_work+0x17b/0x470
[ 41.683285] [<ffffffff8109e5e3>] worker_thread+0x293/0x400
[ 41.688849] [<ffffffff8109e350>] ? rescuer_thread+0x400/0x400
[ 41.694676] [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[ 41.699552] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[ 41.706073] [<ffffffff81646958>] ret_from_fork+0x58/0x90
[ 41.711471] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[ 41.717994] Code: 66 0f 1f 84 00 00 00 00 00 55 48 c7 c6 d7 83 89 81 48 89 e5 e8 f0 fe ff ff b8 ff ff ff ff 5d c3 66 0f 1f 84 00 00 00 00 00 8b 07 <c3> 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 81 fe ff ff 03 00

[K[[31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (30s / 1min 30s)[ 41.994196] AES CTR mode by8 optimization enabled
[ 42.002724] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[ 42.015990] alg: No test for crc32 (crc32-pclmul)
[ 42.035672] systemd-udevd[663]: '/lib/udev/rename_device' [863] terminated by signal 9 (Killed)

[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (31s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (31s / 3min)
[K[ [31m*[1;31m*[0m[31m*[0m] (4 of 4) A start job is running for...vice Initialization (32s / 3min)
[K[ [31m*[1;31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (32s / 1min 30s)
[K[ [31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (33s / 1min 30s)
[K[ [31m*[1;31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (33s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (34s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (34s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (35s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (35s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (36s / 1min 30s)
[K[[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (36s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (37s / 3min)
[K[[31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (37s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (38s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (38s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (39s / 1min 30s)
[K[ [31m*[1;31m*[0m] (1 of 4) A start job is running for...r\x2dtmp.device (39s / 1min 30s)
[K[ [31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (40s / 1min 30s)
[K[ [31m*[1;31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (40s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (41s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (41s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (42s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (42s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (43s / 3min)
[K[[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (43s / 3min)
[K[[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (44s / 3min)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (44s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (45s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (45s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (46s / 1min 30s)
[K[ [31m*[1;31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (46s / 1min 30s)
[K[ [31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (47s / 1min 30s)
[K[ [31m*[1;31m*[0m] (3 of 4) A start job is running for...5e536113.device (47s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (3 of 4) A start job is running for...5e536113.device (48s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (3 of 4) A start job is running for...5e536113.device (48s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (49s / 3min)
[K[[31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (49s / 3min)
[K[[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (50s / 3min)
[K[[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (50s / 1min 30s)
[K[[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (51s / 1min 30s)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (51s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (52s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 4) A start job is running for...\x2dswap.device (52s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (2 of 4) A start job is running for...\x2dswap.device (53s / 1min 30s)
[K[ [31m*[1;31m*[0m] (3 of 4) A start job is running for...5e536113.device (53s / 1min 30s)
[K[ [31m*[0m] (3 of 4) A start job is running for...5e536113.device (54s / 1min 30s)
[K[ [31m*[1;31m*[0m] (3 of 4) A start job is running for...5e536113.device (54s / 1min 30s)
[K[ [31m*[1;31m*[0m[31m*[0m] (4 of 4) A start job is running for...vice Initialization (55s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (55s / 3min)
[K[ [31m*[1;31m*[0m[31m* [0m] (4 of 4) A start job is running for...vice Initialization (56s / 3min)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 4) A start job is running for...r\x2dtmp.device (56s / 1min 30s)[ 68.002815] systemd[1]: Found device /dev/mapper/centos_roadrunner-swap.

[K[[32m OK [ 68.010439] systemd[1]: Activating swap /dev/mapper/centos_roadrunner-swap...
[0m] Found device /dev/mapper/centos_roadrunner-swap.

Activating swap /dev/mapper/centos_roadrunner-swap...

[ 68.062169] systemd[1]: Found device WDC_WD10EZRX-00A8LB0 2.
[[32m OK [0m[ 68.068547] systemd[1]: Mounting /boot...
] Found device WDC_WD10EZRX-00A8LB0 2.

Mounting /boot...

[ 68.104295] Adding 16777212k swap on /dev/mapper/centos_roadrunner-swap. Priority:-1 extents:1 across:16777212k FS
[[32m OK [0m[ 68.117914] systemd[1]: Activated swap /dev/mapper/centos_roadrunner-swap.
] Activated swap[ 68.126743] systemd[1]: Reached target Swap.
/dev/mapper/cen[ 68.131835] systemd[1]: Starting Swap.
tos_roadrunner-swap.

[[32m OK [0m] Reached target Swap.

[ 69.444338] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:0:4]
[ 69.450600] Modules linked in: kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw ixgbe igb ast(+) syscopyarea sysfillrect sysimgblt gf128mul glue_helper ablk_helper ttm drm_kms_helper drm iTCO_wdt usb_storage cryptd i2c_i801 iTCO_vendor_support lpc_ich mfd_core mei_me mei sg ptp pcspkr sb_edac edac_core ipmi_ssif i2c_algo_bit pps_core mdio i2c_core ipmi_si ipmi_msghandler ioatdma dca shpchp acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mxm_wmi crct10dif_pclmul crct10dif_common ahci crc32c_intel libahci libata wmi dm_mirror dm_region_hash dm_log dm_mod
[ 69.508927] CPU: 0 PID: 4 Comm: kworker/0:0 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 69.519169] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 69.526648] Workqueue: events work_for_cpu_fn
[ 69.531016] task: ffff8808544ea280 ti: ffff880854568000 task.ti: ffff880854568000
[ 69.538488] RIP: 0010:[<ffffffff81309862>] [<ffffffff81309862>] ioread32+0x42/0x50
[ 69.546157] RSP: 0000:ffff88085456bd10 EFLAGS: 00000292
[ 69.551462] RAX: 00000000ffffffff RBX: 0000000000000000 RCX: 0000000000000000
[ 69.558584] RDX: ffffc90016a10000 RSI: ffffc90016a10000 RDI: ffffc90016a10000
[ 69.565708] RBP: ffff88085456bd40 R08: 0000000000000082 R09: 0000000000017382
[ 69.572832] R10: 0000000000000000 R11: ffff88085456ba1e R12: ffff88084fe1f800
[ 69.579958] R13: ffff88084fe1d000 R14: ffff88084fe1f800 R15: 0000000000000246
[ 69.587080] FS: 0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000
[ 69.595158] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 69.600894] CR2: 00007f230d046118 CR3: 000000000194a000 CR4: 00000000001407f0
[ 69.608019] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 69.615142] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 69.622266] Stack:
[ 69.624276] ffffffffa03b5ac1 ffff88084fe1d000 0000000000000000 0000000000000000
[ 69.631729] ffffffffa03c0120 0000000000000000 ffff88085456bd68 ffffffffa045b82d
[ 69.639182] ffff88084fe1d000 ffff881052df8000 ffff881052df8098 ffff88085456bdb8
[ 69.646635] Call Trace:
[ 69.649091] [<ffffffffa03b5ac1>] ? ast_driver_load+0x591/0x860 [ast]
[ 69.655537] [<ffffffffa045b82d>] drm_dev_register+0xad/0x100 [drm]
[ 69.661807] [<ffffffffa045e6f9>] drm_get_pci_dev+0x89/0x210 [drm]
[ 69.667986] [<ffffffffa03b51c5>] ast_pci_probe+0x15/0x20 [ast]
[ 69.673908] [<ffffffff813280d5>] local_pci_probe+0x45/0xa0
[ 69.679475] [<ffffffff8109a1b4>] work_for_cpu_fn+0x14/0x20
[ 69.685041] [<ffffffff8109d69b>] process_one_work+0x17b/0x470
[ 69.690872] [<ffffffff8109e5e3>] worker_thread+0x293/0x400
[ 69.696434] [<ffffffff8109e350>] ? rescuer_thread+0x400/0x400
[ 69.702261] [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[ 69.707131] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[ 69.713653] [<ffffffff81646958>] ret_from_fork+0x58/0x90
[ 69.719047] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[ 69.725562] Code: 66 0f 1f 84 00 00 00 00 00 55 48 c7 c6 d7 83 89 81 48 89 e5 e8 f0 fe ff ff b8 ff ff ff ff 5d c3 66 0f 1f 84 00 00 00 00 00 8b 07 <c3> 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 81 fe ff ff 03 00
[[1;31m*[0m[31m* [0m] (3 of 3) A start job is running for... Initialization (1min 2s / 3min)
[K[[0m[31m* [0m] (3 of 3) A start job is running for... Initialization (1min 2s / 3min)
[K[[1;31m*[0m[31m* [0m] (1 of 3) A start job is running for /boot (1min 3s / 2min 26s)
[K[[31m*[1;31m*[0m[31m* [0m] (1 of 3) A start job is running for /boot (1min 3s / 2min 26s)
[K[ [31m*[1;31m*[0m[31m* [0m] (1 of 3) A start job is running for /boot (1min 4s / 2min 26s)
[K[ [31m*[1;31m*[0m[31m* [0m] (2 of 3) A start job is running for...dtmp.device (1min 4s / 1min 30s)[ 75.938327] INFO: rcu_sched self-detected stall on CPU[ 75.941352] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 3, t=60003 jiffies, g=1024, c=1023, q=0)
[ 75.941360] sending NMI to all CPUs:
[ 75.943464] NMI backtrace for cpu 0
[ 75.943465] CPU: 0 PID: 4 Comm: kworker/0:0 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 75.943468] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 75.943469] Workqueue: events work_for_cpu_fn
[ 75.943470] task: ffff8808544ea280 ti: ffff880854568000 task.ti: ffff880854568000
[ 75.943472] RIP: 0010:[<ffffffff8101c800>] [<ffffffff8101c800>] hw_breakpoint_pmu_read+0x10/0x10
[ 75.943473] RSP: 0000:ffff88085fc03bd0 EFLAGS: 00000046
[ 75.943473] RAX: 0000000000000000 RBX: 000000007db780d8 RCX: 0000000000000000
[ 75.943474] RDX: 000000007db782cc RSI: 00000000000003fd RDI: 0000000000000641
[ 75.943475] RBP: ffff88085fc03bf0 R08: ffffffff81a68f20 R09: 0000000000000000
[ 75.943475] R10: 0000000000000000 R11: ffff88085fc03b96 R12: 0000000000000641
[ 75.943476] R13: 0000000000000000 R14: ffffffff81cadc99 R15: 0000000000000039
[ 75.943476] FS: 0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000
[ 75.943477] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.943477] CR2: 00007f230d046118 CR3: 000000000194a000 CR4: 00000000001407f0
[ 75.943478] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 75.943479] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 75.943479] Stack:
[ 75.943480] ffffffff8130073a ffffffff81f1b4e0 00000000000026f5 0000000000000020
[ 75.943480] ffff88085fc03c00 ffffffff81300688 ffff88085fc03c28 ffffffff813d0dd0
[ 75.943481] ffffffff81f1b4e0 0000000000000065 ffffffff81f1b4e0 ffff88085fc03c48
[ 75.943481] Call Trace:
[ 75.943482] <IRQ> d [<ffffffff8130073a>] ? delay_tsc+0x4a/0x80
[ 75.943483] [<ffffffff81300688>] __const_udelay+0x28/0x30
[ 75.943483] [<ffffffff813d0dd0>] wait_for_xmitr+0x30/0xa0
[ 75.943484] [<ffffffff813d0f9c>] serial8250_console_putchar+0x1c/0x30
[ 75.943484] [<ffffffff813d0f80>] ? serial8250_console_write+0x140/0x140
[ 75.943485] [<ffffffff813cba3a>] uart_console_write+0x3a/0x80
[ 75.943486] [<ffffffff813d0eee>] serial8250_console_write+0xae/0x140
[ 75.943486] [<ffffffff8107c651>] call_console_drivers.constprop.15+0x91/0xf0
[ 75.943487] [<ffffffff8107de6e>] console_unlock+0x2ae/0x400
[ 75.943487] [<ffffffff8107e276>] vprintk_emit+0x2b6/0x530
[ 75.943488] [<ffffffff8108bf18>] ? __internal_add_timer+0xc8/0x130
[ 75.943488] [<ffffffff8107e719>] vprintk_default+0x29/0x40
[ 75.943489] [<ffffffff8162fd31>] printk+0x5e/0x75
[ 75.943489] [<ffffffff81126ac8>] rcu_check_callbacks+0x268/0x610
[ 75.943490] [<ffffffff810e08c0>] ? tick_sched_handle.isra.14+0x60/0x60
[ 75.943490] [<ffffffff8108e967>] update_process_times+0x47/0x80
[ 75.943491] [<ffffffff810e0885>] tick_sched_handle.isra.14+0x25/0x60
[ 75.943491] [<ffffffff810e0901>] tick_sched_timer+0x41/0x70
[ 75.943492] [<ffffffff810a9db2>] __hrtimer_run_queues+0xd2/0x260
[ 75.943492] [<ffffffff810aa350>] hrtimer_interrupt+0xb0/0x1e0
[ 75.943493] [<ffffffff8164831c>] ? call_softirq+0x1c/0x30
[ 75.943493] [<ffffffff810495c7>] local_apic_timer_interrupt+0x37/0x60
[ 75.943494] [<ffffffff81648f8f>] smp_apic_timer_interrupt+0x3f/0x60
[ 75.943494] [<ffffffff8164765d>] apic_timer_interrupt+0x6d/0x80
[ 75.943495] <EOI> d [<ffffffff81309862>] ? ioread32+0x42/0x50
[ 75.943495] [<ffffffffa03b5ac1>] ? ast_driver_load+0x591/0x860 [ast]
[ 75.943496] [<ffffffffa045b82d>] drm_dev_register+0xad/0x100 [drm]
[ 75.943496] [<ffffffffa045e6f9>] drm_get_pci_dev+0x89/0x210 [drm]
[ 75.943497] [<ffffffffa03b51c5>] ast_pci_probe+0x15/0x20 [ast]
[ 75.943497] [<ffffffff813280d5>] local_pci_probe+0x45/0xa0
[ 75.943498] [<ffffffff8109a1b4>] work_for_cpu_fn+0x14/0x20
[ 75.943498] [<ffffffff8109d69b>] process_one_work+0x17b/0x470
[ 75.943499] [<ffffffff8109e5e3>] worker_thread+0x293/0x400
[ 75.943499] [<ffffffff8109e350>] ? rescuer_thread+0x400/0x400
[ 75.943499] [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[ 75.943500] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[ 75.943500] [<ffffffff81646958>] ret_from_fork+0x58/0x90
[ 75.943501] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[ 75.943502] Code: 48 c7 43 f8 00 00 00 00 41 83 ec 01 75 e6 5b 41 5c 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 5d c3 0f 1f 44 00 00 <55> 48 89 e5 0f 31 89 c0 48 c1 e2 20 48 09 c2 48 89 d0 5d c3 66
[ 75.943502] NMI backtrace for cpu 1
[ 75.943503] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 75.943504] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 75.943504] task: ffff880854709700 ti: ffff880854738000 task.ti: ffff880854738000
[ 75.943505] RIP: 0010:[<ffffffff8135e1a7>] [<ffffffff8135e1a7>] intel_idle+0xd7/0x160
[ 75.943505] RSP: 0018:ffff88085473be10 EFLAGS: 00000046
[ 75.943506] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 75.943506] RDX: 0000000000000000 RSI: ffff88085473bfd8 RDI: 0000000000000001
[ 75.943507] RBP: ffff88085473be40 R08: 0000000000054f3e R09: 0000000000000018
[ 75.943507] R10: 0000000000002eba R11: 0000000000000001 R12: ffff88085473bfd8
[ 75.943508] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fee38
[ 75.943508] FS: 0000000000000000(0000) GS:ffff88085fc40000(0000) knlGS:0000000000000000
[ 75.943509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.943510] CR2: 00007fb45f383910 CR3: 000000000194a000 CR4: 00000000001407e0
[ 75.943510] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 75.943511] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 75.943512] Stack:
[ 75.943512] 000000015473be40 18c410b9370e4663 ffff88085fc60500 ffffffff819fecc0
[ 75.943513] 000000116243b57c 0000000000000004 ffff88085473be78 ffffffff814d4ab0
[ 75.943513] ffff88085fc60500 0000000000000004 0000000000000004 ffffffff819fecc0
[ 75.943514] Call Trace:
[ 75.943514] [<ffffffff814d4ab0>] cpuidle_enter_state+0x40/0xc0
[ 75.943515] [<ffffffff814d4c09>] cpuidle_idle_call+0xd9/0x210
[ 75.943516] [<ffffffff8101e4ee>] arch_cpu_idle+0xe/0x30
[ 75.943516] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[ 75.943517] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[ 75.943518] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d fa 0a
[ 75.943518] NMI backtrace for cpu 2
[ 75.943519] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 75.943519] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 75.943520] task: ffff88085470a280 ti: ffff88085473c000 task.ti: ffff88085473c000
[ 75.943520] RIP: 0010:[<ffffffff8135e1a7>] [<ffffffff8135e1a7>] intel_idle+0xd7/0x160
[ 75.943521] RSP: 0018:ffff88085473fe10 EFLAGS: 00000046
[ 75.943521] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 75.943522] RDX: 0000000000000000 RSI: ffff88085473ffd8 RDI: 0000000000000002
[ 75.943522] RBP: ffff88085473fe40 R08: 0000000000000b85 R09: ffff88085fc93738
[ 75.943523] R10: 7fffffffffffffff R11: 0000000000000000 R12: ffff88085473ffd8
[ 75.943523] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fee38
[ 75.943524] FS: 0000000000000000(0000) GS:ffff88085fc80000(0000) knlGS:0000000000000000
[ 75.943524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.943525] CR2: 00007fb45fb93f5e CR3: 000000000194a000 CR4: 00000000001407e0
[ 75.943526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 75.943526] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 75.943526] Stack:
[ 75.943527] 000000025473fe40 4cc7ada01b4a1bfc ffff88085fca0500 ffffffff819fecc0
[ 75.943528] 00000010ce71e752 0000000000000004 ffff88085473fe78 ffffffff814d4ab0
[ 75.943528] ffff88085fca0500 0000000000000004 0000000000000004 ffffffff819fecc0
[ 75.943528] Call Trace:
[ 75.943529] [<ffffffff814d4ab0>] cpuidle_enter_state+0x40/0xc0
[ 75.943529] [<ffffffff814d4c09>] cpuidle_idle_call+0xd9/0x210
[ 75.943530] [<ffffffff8101e4ee>] arch_cpu_idle+0xe/0x30
[ 75.943530] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[ 75.943531] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[ 75.943532] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d fa 0a
[ 75.943532] NMI backtrace for cpu 3
[ 75.943533] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 75.943533] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 75.943534] task: ffff88085470ae00 ti: ffff880854740000 task.ti: ffff880854740000
[ 75.943534] RIP: 0010:[<ffffffff8130072a>] [<ffffffff8130072a>] delay_tsc+0x3a/0x80
[ 75.943535] RSP: 0018:ffff88085fcc3d20 EFLAGS: 00000093
[ 75.943536] RAX: 0000000000000003 RBX: 000000007dbed970 RCX: 0000000000000034
[ 75.943536] RDX: 000000007dbed990 RSI: 0000000000000008 RDI: 0000000000027101
[ 75.943537] RBP: ffff88085fcc3d38 R08: ffffffff81a66820 R09: ffff88107ff9ba84
[ 75.943537] R10: 00000000000625a4 R11: 0000000000800000 R12: 0000000000027101
[ 75.943538] R13: 0000000000000003 R14: 0000000000000086 R15: 0000000000000002
[ 75.943539] FS: 0000000000000000(0000) GS:ffff88085fcc0000(0000) knlGS:0000000000000000
[ 75.943539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.943540] CR2: 00007fe673a0f038 CR3: 000000000194a000 CR4: 00000000001407e0
[ 75.943541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 75.943541] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 75.943542] Stack:
[ 75.943542] 00000000000003e9 0000000000001000 0000000000000400 ffff88085fcc3d48
[ 75.943543] ffffffff81300688 ffff88085fcc3d68 ffffffff8104964a 000000000000a022
[ 75.943543] ffffffff81a66820 ffff88085fcc3db0 ffffffff8104b0eb 000000085fcc3dd0
[ 75.943544] Call Trace:
[ 75.943544] <IRQ> d [<ffffffff81300688>] __const_udelay+0x28/0x30
[ 75.943545] [<ffffffff8104964a>] native_safe_apic_wait_icr_idle+0x2a/0x60
[ 75.943545] [<ffffffff8104b0eb>] default_send_IPI_mask_sequence_phys+0xeb/0x100
[ 75.943546] [<ffffffff810508ce>] physflat_send_IPI_mask+0xe/0x10
[ 75.943546] [<ffffffff8104b55d>] arch_trigger_all_cpu_backtrace+0x2bd/0x2d0
[ 75.943547] [<ffffffff81126e1d>] rcu_check_callbacks+0x5bd/0x610
[ 75.943547] [<ffffffff810e08c0>] ? tick_sched_handle.isra.14+0x60/0x60
[ 75.943548] [<ffffffff8108e967>] update_process_times+0x47/0x80
[ 75.943548] [<ffffffff810e0885>] tick_sched_handle.isra.14+0x25/0x60
[ 75.943549] [<ffffffff810e0901>] tick_sched_timer+0x41/0x70
[ 75.943549] [<ffffffff810a9db2>] __hrtimer_run_queues+0xd2/0x260
[ 75.943550] [<ffffffff810aa350>] hrtimer_interrupt+0xb0/0x1e0
[ 75.943550] [<ffffffff810495c7>] local_apic_timer_interrupt+0x37/0x60
[ 75.943551] [<ffffffff81648f8f>] smp_apic_timer_interrupt+0x3f/0x60
[ 75.943551] [<ffffffff8164765d>] apic_timer_interrupt+0x6d/0x80
[ 75.943552] <EOI> d [<ffffffff814d4ac2>] ? cpuidle_enter_state+0x52/0xc0
[ 75.943552] [<ffffffff814d4c09>] cpuidle_idle_call+0xd9/0x210
[ 75.943553] [<ffffffff8101e4ee>] arch_cpu_idle+0xe/0x30
[ 75.943553] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[ 75.943554] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[ 75.943555] Code: fc 53 65 44 8b 2c 25 1c a0 00 00 66 66 90 0f ae e8 e8 eb c0 d1 ff 66 90 89 c3 eb 14 0f 1f 44 00 00 f3 90 65 8b 04 25 1c a0 00 00 <41> 39 c5 75 1d 66 66 90 0f ae e8 e8 c6 c0 d1 ff 66 90 89 c2 29
[ 75.943555] NMI backtrace for cpu 4
[ 75.943556] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 75.943556] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 75.943557] task: ffff88085470b980 ti: ffff880854744000 task.ti: ffff880854744000
[ 75.943557] RIP: 0010:[<ffffffff8135e1a7>] [<ffffffff8135e1a7>] intel_idle+0xd7/0x160
[ 75.943558] RSP: 0018:ffff880854747e10 EFLAGS: 00000046
[ 75.943559] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 75.943559] RDX: 0000000000000000 RSI: ffff880854747fd8 RDI: 000000000194a000
[ 75.943560] RBP: ffff880854747e40 R08: 000000000003a4cf R09: 0000000000000018
[ 75.943560] R10: 0000000000085892 R11: 0000000000000001 R12: ffff880854747fd8
[ 75.943561] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fee38
[ 75.943561] FS: 0000000000000000(0000) GS:ffff88085fd00000(0000) knlGS:0000000000000000
[ 75.943562] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.943562] CR2: 00007fb45fb93f5e CR3: 000000000194a000 CR4: 00000000001407e0
[ 75.943563] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 75.943563] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 75.943564] Stack:
[ 75.943564] 0000000454747e40 b47832745093d629 ffff88085fd20500 ffffffff819fecc0
[ 75.943565] 000000113b46628c 0000000000000004 ffff880854747e78 ffffffff814d4ab0
[ 75.943565] ffff88085fd20500 0000000000000004 0000000000000004 ffffffff819fecc0
[ 75.943566] Call Trace:
[ 75.943566] [<ffffffff814d4ab0>] cpuidle_enter_state+0x40/0xc0
[ 75.943567] [<ffffffff814d4c09>] cpuidle_idle_call+0xd9/0x210
[ 75.943567] [<ffffffff8101e4ee>] arch_cpu_idle+0xe/0x30
[ 75.943568] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[ 75.943569] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[ 75.943570] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d fa 0a
[ 75.943570] NMI backtrace for cpu 5
[ 75.943571] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G L ------------ 3.10.0-327.36.1.el7.x86_64 #1
[ 75.943572] Hardware name: Supermicro SYS-6028R-TRT/X10DRI-T, BIOS 3.2 11/22/2019
[ 75.943573] task: ffff88085470c500 ti: ffff880854748000 task.ti: ffff880854748000
[ 75.943573] RIP: 0010:[<ffffffff8135e1a7>] [<ffffffff8135e1a7>] intel_idle+0xd7/0x160
[ 75.943574] RSP: 0018:ffff88085474be10 EFLAGS: 00000046
[ 75.943574] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 75.943575] RDX: 0000000000000000 RSI: ffff88085474bfd8 RDI: 0000000000000005
[ 75.943575] RBP: ffff88085474be40 R08: 0000000000000049 R09: 0000000000000018
[ 75.943576] R10: 00000000000da7ce R11: 0000000000000000 R12: ffff88085474bfd8
[ 75.943576] R13: 0000000000000004 R14: 0000000000000020 R15: ffffffff819fee38
[ 75.943577] FS: 0000000000000000(0000) GS:ffff88085fd40000(0000) knlGS:0000000000000000
[ 75.943577] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 75.943578] CR2: 00007fb45fb93f5e CR3: 000000000194a000 CR4: 00000000001407e0
[ 75.943578] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 75.943579] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 75.943579] Stack:
[ 75.943580] 000000055474be40 e01da4bedc0870bd ffff88085fd60500 ffffffff819fecc0
[ 75.943580] 00000010d0c46458 0000000000000004 ffff88085474be78 ffffffff814d4ab0
[ 75.943581] ffff88085fd60500 0000000000000004 0000000000000004 ffffffff819fecc0
[ 75.943581] Call Trace:
[ 75.943582] [<ffffffff814d4ab0>] cpuidle_enter_state+0x40/0xc0
[ 75.943582] [<ffffffff814d4c09>] cpuidle_idle_call+0xd9/0x210
[ 75.943583] [<ffffffff8101e4ee>] arch_cpu_idle+0xe/0x30
[ 75.943583] [<ffffffff810d64a5>] cpu_startup_entry+0x245/0x290
[ 75.943584] [<ffffffff8104768a>] start_secondary+0x1ba/0x230
[ 75.943585] Code: 31 d2 65 48 8b 34 25 b8 b7 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <65> 48 8b 04 25 b8 b7 00 00 f0 80 a0 3a c0 ff ff 7f 85 1d fa 0a
The file is very large but the process seems to start normally until the following line appears:

Code: Select all

A start job is running for...er\x2dtmp.device
Then the process gets stuck in a loop of errors/call traces that never ends. Do you have any idea what the problem might be in those call traces?

Like I said, the file is about 1.5 Mb but I could upload it somewhere if you would like to take a look.

Thanks again and best regards.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: How to verify cause of system malfunction, probably motherboard

Post by TrevorH » 2020/05/16 16:12:53

All those "A start job is running for...er\x2dtmp.device (9s / 1min 30s)" type messages are coming from systemd as it tries to mount the things listed in /etc/fstab. Either your hard disk has gone south or something else has happened to corrupt things. I'd start by powering off and checking all your cabling and making sure it's correct. I'd also boot a rescue media and run smartctl against any and all disks and see if any report hardware problems.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

CZa
Posts: 8
Joined: 2020/05/15 10:18:05

Re: How to verify cause of system malfunction, probably motherboard

Post by CZa » 2020/05/16 18:04:24

Thanks again for your prompt reply, TrevorH.

So everything seems to be pointing at hard drive problems, even with different hard drives, actually connected to different ports. I will do what you advise and get back with the results on Monday. Also, I am currently connecting the hard drives to the 8-way front panel of the server which enables their live connection/disconnection. I will also try to connect them directly to the motherboard to rule out potential problems with that front panel.

Nevertheless, it's weird, because I tried writing hundreds of GBs to the hard drive in the new installation and it worked flawlessly.

Well, like I said, I'll continue running tests.

Thanks again and best regards

CZa
Posts: 8
Joined: 2020/05/15 10:18:05

Re: How to verify cause of system malfunction, probably motherboard

Post by CZa » 2020/05/20 11:50:40

Hi again,

I ran smartctl on all the drives as well as memtest86 for 24 hours. No errors were reported. Since I can only make the new installation crash using stress-ng and that programme, well, stresses the system in many different ways simultaneously, it is difficult to draw conclusions.

Thus, I am thinking about going back to the previous approach. My old installation gets caught up in a loop of errors and does not boot up. I can boot to emergency mode but not to rescue mode. However, I am not sure what is the difference between those two modes in terms of what services are run in either of them. Would it be possible to start from the emergency mode and run services manually, one by one, to see which one produces the crash? What services should I run then?

Thanks and best regards.

CZa
Posts: 8
Joined: 2020/05/15 10:18:05

Re: How to verify cause of system malfunction, probably motherboard

Post by CZa » 2020/05/21 09:08:26

By the way, when it gets stuck in the loop showing errors and stack traces while trying to boot the old installation, the system does not reboot when pressing ctrl+alt+del. When pressed a few times, it says started show plymouth reboot screen but nothing happens. If I press it many times it says failed to start plymouth reboot screen.

Post Reply