CentOS 7.6 Getting kernel panic

General support questions
Post Reply
HxGao
Posts: 1
Joined: 2021/11/19 03:20:02

CentOS 7.6 Getting kernel panic

Post by HxGao » 2021/11/19 06:00:42

Hi,
Recently, I ran into a serious problem with kernel panics and hanging systems on CentOS Linux release 7.6.1810 (kernel: 3.10.0-957.el7.x86_64) .
Does anyone have any insight into kernel panics like this?
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284754 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284754 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e91396180 0 284754 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e91396180 0 284754 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc55bf4b>] ? queued_spin_lock_slowpath+0xb/0xf
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc55bf4b>] ? queued_spin_lock_slowpath+0xb/0xf
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbf0cb18>] ? get_futex_key+0x1c8/0x2c0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbf0cb18>] ? get_futex_key+0x1c8/0x2c0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284755 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284755 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e91395140 0 284755 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e91395140 0 284755 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284756 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284756 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e91390000 0 284756 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e91390000 0 284756 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284757 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284757 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df1040 0 284757 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df1040 0 284757 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284758 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284758 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df30c0 0 284758 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df30c0 0 284758 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbedb6e8>] ? __enqueue_entity+0x78/0x80
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbedb6e8>] ? __enqueue_entity+0x78/0x80
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5691fd>] rwsem_down_read_failed+0x10d/0x1a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5691fd>] rwsem_down_read_failed+0x10d/0x1a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed2ea0>] ? check_preempt_curr+0x80/0xa0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed2ea0>] ? check_preempt_curr+0x80/0xa0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c55d0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c55d0>] ? xfs_ilock_data_map_shared+0x30/0x40 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869a8>] call_rwsem_down_read_failed+0x18/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869a8>] call_rwsem_down_read_failed+0x18/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566e80>] down_read+0x20/0x40
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566e80>] down_read+0x20/0x40
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c555c>] xfs_ilock+0xdc/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c555c>] xfs_ilock+0xdc/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c55d0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c55d0>] xfs_ilock_data_map_shared+0x30/0x40 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ab9ca>] __xfs_get_blocks+0x9a/0x6d0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ab9ca>] __xfs_get_blocks+0x9a/0x6d0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbe798e2>] ? get_user_pages_fast+0x122/0x1a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbe798e2>] ? get_user_pages_fast+0x122/0x1a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace24>] xfs_get_blocks_direct+0x14/0x20 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace24>] xfs_get_blocks_direct+0x14/0x20 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc084286>] do_blockdev_direct_IO+0xf96/0x20a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc084286>] do_blockdev_direct_IO+0xf96/0x20a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace10>] ? xfs_setfilesize+0x80/0x80 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace10>] ? xfs_setfilesize+0x80/0x80 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0853e5>] __blockdev_direct_IO+0x55/0x60
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0853e5>] __blockdev_direct_IO+0x55/0x60
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace10>] ? xfs_setfilesize+0x80/0x80 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace10>] ? xfs_setfilesize+0x80/0x80 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8759>] xfs_file_dio_aio_write+0x179/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8759>] xfs_file_dio_aio_write+0x179/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace10>] ? xfs_setfilesize+0x80/0x80 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace10>] ? xfs_setfilesize+0x80/0x80 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08ace30>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284759 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284759 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df2080 0 284759 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df2080 0 284759 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284760 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284760 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df0000 0 284760 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df0000 0 284760 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284761 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284761 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df5140 0 284761 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03eb1df5140 0 284761 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284762 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284762 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e8dd82080 0 284762 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e8dd82080 0 284762 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbbed64f0>] ? try_to_wake_up+0x190/0x390
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284763 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: INFO: task mysqld:284763 blocked for more than 120 seconds.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e8dd85140 0 284763 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: mysqld D ffffa03e8dd85140 0 284763 1 0x00000000
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: Call Trace:
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc01be42>] ? kmem_cache_alloc+0x1c2/0x1f0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d3227>] ? kmem_zone_alloc+0x97/0x130 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc567bc9>] schedule+0x29/0x70
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc5694b5>] rwsem_down_write_failed+0x225/0x3a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08d38e4>] ? xlog_grant_head_check+0x54/0x100 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] ? xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc1869d7>] call_rwsem_down_write_failed+0x17/0x30
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc566ecd>] down_write+0x2d/0x3d
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c5544>] xfs_ilock+0xc4/0x120 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08c27dd>] xfs_vn_update_time+0xcd/0x150 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05df58>] update_time+0x28/0xd0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc05e0a0>] file_update_time+0xa0/0xf0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b83bd>] xfs_file_aio_write_checks+0x18d/0x200 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b86a8>] xfs_file_dio_aio_write+0xc8/0x340 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8ce2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffc08b8be0>] ? xfs_file_buffered_aio_write+0x2c0/0x2c0 [xfs]
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc0974b3>] do_io_submit+0x3e3/0x8a0
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc097980>] SyS_io_submit+0x10/0x20
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Nov 11 04:07:35 shard07-db01 kernel: [<ffffffffbc574ddb>] system_call_fastpath+0x22/0x27
Is this a known issue with this kernel. fixed in latest version?
Top
User avatar
TrevorH
Site Admin
Posts: 33220
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 7.6 Getting kernel panic

Post by TrevorH » 2021/11/19 14:57:38

It may or may not be a known issue (I think it is) but only the latest gets any support so the first thing we will ask you to do is: update to $latest and retry. And by update to latest, I mean everything not just the kernel. Run yum update.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Top
Post Reply

Return to “CentOS 7 - General Support”