Horrible performance HDIO_GET_IDENTITY failed /dev/sdb

General support questions
Post Reply
cwensink
Posts: 3
Joined: 2014/06/24 13:35:20

Horrible performance HDIO_GET_IDENTITY failed /dev/sdb

Post by cwensink » 2020/04/22 14:44:42

Hello Everyone,
Since rebooting my Centos 6.10 Openvz server "daisy" yesterday, I am getting horrible system performance. /var/log/messages is full of HDIO_GET_IDENTITY failed for /dev/sdb. The latest entries look like this:

Apr 22 08:51:32 daisy kernel: [141224.655699] CT: 1005: stopped
Apr 22 08:55:04 daisy ata_id[21513]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:00:05 daisy ata_id[21584]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:05:02 daisy ata_id[21644]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:10:01 daisy ata_id[22282]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:11:49 daisy kernel: [142441.721065] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:11:49 daisy kernel: [142441.721083] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:11:49 daisy kernel: [142441.721093] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:11:49 daisy kernel: [142441.721109] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:11:49 daisy kernel: [142441.721115] ffff88006654bcb8 0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:11:49 daisy kernel: [142441.721121] ffff88000004d238 ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:11:49 daisy kernel: [142441.721125] ffff88011a707000 ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:11:49 daisy kernel: [142441.721130] Call Trace:
Apr 22 09:11:49 daisy kernel: [142441.721139] [<ffffffff8114f130>] ? sync_page+0x0/0x50
Apr 22 09:11:49 daisy kernel: [142441.721144] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:11:49 daisy kernel: [142441.721149] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:11:49 daisy kernel: [142441.721155] [<ffffffff81067432>] ? check_preempt_curr+0x82/0xa0
Apr 22 09:11:49 daisy kernel: [142441.721159] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:11:49 daisy kernel: [142441.721162] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:11:49 daisy kernel: [142441.721167] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:11:49 daisy kernel: [142441.721172] [<ffffffff811f98d8>] sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:11:49 daisy kernel: [142441.721176] [<ffffffff8114fa6f>] ? filemap_fdatawait+0x2f/0x40
Apr 22 09:11:49 daisy kernel: [142441.721181] [<ffffffff81200f85>] __sync_filesystem+0x95/0xa0
Apr 22 09:11:49 daisy kernel: [142441.721184] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:11:49 daisy kernel: [142441.721188] [<ffffffff812016e5>] sys_sync+0x155/0x1a0
Apr 22 09:11:49 daisy kernel: [142441.721192] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:13:49 daisy kernel: [142561.721069] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:13:49 daisy kernel: [142561.721087] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:13:49 daisy kernel: [142561.721096] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:13:49 daisy kernel: [142561.721112] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:13:49 daisy kernel: [142561.721118] ffff88006654bcb8 0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:13:49 daisy kernel: [142561.721123] ffff88000004d238 ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:13:49 daisy kernel: [142561.721128] ffff88011a707000 ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:13:49 daisy kernel: [142561.721133] Call Trace:
Apr 22 09:13:49 daisy kernel: [142561.721142] [<ffffffff8114f130>] ? sync_page+0x0/0x50
Apr 22 09:13:49 daisy kernel: [142561.721148] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:13:49 daisy kernel: [142561.721153] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:13:49 daisy kernel: [142561.721158] [<ffffffff81067432>] ? check_preempt_curr+0x82/0xa0
Apr 22 09:13:49 daisy kernel: [142561.721162] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:13:49 daisy kernel: [142561.721166] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:13:49 daisy kernel: [142561.721170] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:13:49 daisy kernel: [142561.721176] [<ffffffff811f98d8>] sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:13:49 daisy kernel: [142561.721180] [<ffffffff8114fa6f>] ? filemap_fdatawait+0x2f/0x40
Apr 22 09:13:49 daisy kernel: [142561.721184] [<ffffffff81200f85>] __sync_filesystem+0x95/0xa0
Apr 22 09:13:49 daisy kernel: [142561.721188] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:13:49 daisy kernel: [142561.721192] [<ffffffff812016e5>] sys_sync+0x155/0x1a0
Apr 22 09:13:49 daisy kernel: [142561.721196] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:15:06 daisy ata_id[22299]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:15:49 daisy kernel: [142681.721085] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:15:49 daisy kernel: [142681.721104] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:15:49 daisy kernel: [142681.721113] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:15:49 daisy kernel: [142681.721129] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:15:49 daisy kernel: [142681.721136] ffff88006654bcb8 0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:15:49 daisy kernel: [142681.721141] ffff88000004d238 ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:15:49 daisy kernel: [142681.721146] ffff88011a707000 ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:15:49 daisy kernel: [142681.721150] Call Trace:
Apr 22 09:15:49 daisy kernel: [142681.721160] [<ffffffff8114f130>] ? sync_page+0x0/0x50
Apr 22 09:15:49 daisy kernel: [142681.721166] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:15:49 daisy kernel: [142681.721172] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:15:49 daisy kernel: [142681.721178] [<ffffffff81067432>] ? check_preempt_curr+0x82/0xa0
Apr 22 09:15:49 daisy kernel: [142681.721182] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:15:49 daisy kernel: [142681.721185] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:15:49 daisy kernel: [142681.721190] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:15:49 daisy kernel: [142681.721196] [<ffffffff811f98d8>] sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:15:49 daisy kernel: [142681.721200] [<ffffffff8114fa6f>] ? filemap_fdatawait+0x2f/0x40
Apr 22 09:15:49 daisy kernel: [142681.721204] [<ffffffff81200f85>] __sync_filesystem+0x95/0xa0
Apr 22 09:15:49 daisy kernel: [142681.721208] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:15:49 daisy kernel: [142681.721212] [<ffffffff812016e5>] sys_sync+0x155/0x1a0
Apr 22 09:15:49 daisy kernel: [142681.721217] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:17:49 daisy kernel: [142801.721064] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:17:49 daisy kernel: [142801.721082] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:17:49 daisy kernel: [142801.721091] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:17:49 daisy kernel: [142801.721107] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:17:49 daisy kernel: [142801.721114] ffff88006654bcb8 0000000000000086 ffffffff8114f130 ffff88002821fa40
Apr 22 09:17:49 daisy kernel: [142801.721119] ffff88000004d238 ffff88006654bd70 ffff88006654bc88 ffffea00016ab7c0
Apr 22 09:17:49 daisy kernel: [142801.721124] ffff88011a707000 ffff880028321168 000000000001b7ea 0000816be9b3faa2
Apr 22 09:17:49 daisy kernel: [142801.721128] Call Trace:
Apr 22 09:17:49 daisy kernel: [142801.721137] [<ffffffff8114f130>] ? sync_page+0x0/0x50
Apr 22 09:17:49 daisy kernel: [142801.721143] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:17:49 daisy kernel: [142801.721149] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:17:49 daisy kernel: [142801.721154] [<ffffffff81067432>] ? check_preempt_curr+0x82/0xa0
Apr 22 09:17:49 daisy kernel: [142801.721158] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:17:49 daisy kernel: [142801.721162] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:17:49 daisy kernel: [142801.721166] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:17:49 daisy kernel: [142801.721172] [<ffffffff811f98d8>] sync_inodes_sb_ub+0xa8/0x1d0
Apr 22 09:17:49 daisy kernel: [142801.721176] [<ffffffff8114fa6f>] ? filemap_fdatawait+0x2f/0x40
Apr 22 09:17:49 daisy kernel: [142801.721180] [<ffffffff81200f85>] __sync_filesystem+0x95/0xa0
Apr 22 09:17:49 daisy kernel: [142801.721184] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:17:49 daisy kernel: [142801.721188] [<ffffffff812016e5>] sys_sync+0x155/0x1a0
Apr 22 09:17:49 daisy kernel: [142801.721192] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:20:01 daisy ata_id[22405]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:21:49 daisy kernel: [143041.721494] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:21:49 daisy kernel: [143041.721512] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:21:49 daisy kernel: [143041.721522] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:21:49 daisy kernel: [143041.721691] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:21:49 daisy kernel: [143041.721697] ffff88006654bcc8 0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:21:49 daisy kernel: [143041.721702] ffff880028200000 000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:21:49 daisy kernel: [143041.721706] ffff88006654bc68 ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:21:49 daisy kernel: [143041.721711] Call Trace:
Apr 22 09:21:49 daisy kernel: [143041.721720] [<ffffffff810098af>] ? __switch_to+0x16f/0x470
Apr 22 09:21:49 daisy kernel: [143041.721726] [<ffffffff8107bbfe>] ? finish_task_switch+0xce/0x120
Apr 22 09:21:49 daisy kernel: [143041.721730] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:21:49 daisy kernel: [143041.721735] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:21:49 daisy kernel: [143041.721739] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:21:49 daisy kernel: [143041.721743] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:21:49 daisy kernel: [143041.721747] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:21:49 daisy kernel: [143041.721753] [<ffffffff811f9773>] writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:21:49 daisy kernel: [143041.721757] [<ffffffff811f9806>] writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:21:49 daisy kernel: [143041.721762] [<ffffffff81200f38>] __sync_filesystem+0x48/0xa0
Apr 22 09:21:49 daisy kernel: [143041.721765] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:21:49 daisy kernel: [143041.721769] [<ffffffff812016d8>] sys_sync+0x148/0x1a0
Apr 22 09:21:49 daisy kernel: [143041.721773] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:23:49 daisy kernel: [143161.721064] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:23:49 daisy kernel: [143161.721169] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:23:49 daisy kernel: [143161.721259] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:23:49 daisy kernel: [143161.721430] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:23:49 daisy kernel: [143161.721437] ffff88006654bcc8 0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:23:49 daisy kernel: [143161.721442] ffff880028200000 000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:23:49 daisy kernel: [143161.721447] ffff88006654bc68 ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:23:49 daisy kernel: [143161.721451] Call Trace:
Apr 22 09:23:49 daisy kernel: [143161.721460] [<ffffffff810098af>] ? __switch_to+0x16f/0x470
Apr 22 09:23:49 daisy kernel: [143161.721466] [<ffffffff8107bbfe>] ? finish_task_switch+0xce/0x120
Apr 22 09:23:49 daisy kernel: [143161.721470] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:23:49 daisy kernel: [143161.721475] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:23:49 daisy kernel: [143161.721479] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:23:49 daisy kernel: [143161.721483] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:23:49 daisy kernel: [143161.721487] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:23:49 daisy kernel: [143161.721493] [<ffffffff811f9773>] writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:23:49 daisy kernel: [143161.721498] [<ffffffff811f9806>] writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:23:49 daisy kernel: [143161.721502] [<ffffffff81200f38>] __sync_filesystem+0x48/0xa0
Apr 22 09:23:49 daisy kernel: [143161.721506] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:23:49 daisy kernel: [143161.721510] [<ffffffff812016d8>] sys_sync+0x148/0x1a0
Apr 22 09:23:49 daisy kernel: [143161.721514] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:25:02 daisy ata_id[22445]: HDIO_GET_IDENTITY failed for '/dev/sdb'
Apr 22 09:25:49 daisy kernel: [143281.721066] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:25:49 daisy kernel: [143281.721159] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:25:49 daisy kernel: [143281.721244] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:25:49 daisy kernel: [143281.721408] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:25:49 daisy kernel: [143281.721415] ffff88006654bcc8 0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:25:49 daisy kernel: [143281.721420] ffff880028200000 000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:25:49 daisy kernel: [143281.721424] ffff88006654bc68 ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:25:49 daisy kernel: [143281.721429] Call Trace:
Apr 22 09:25:49 daisy kernel: [143281.721438] [<ffffffff810098af>] ? __switch_to+0x16f/0x470
Apr 22 09:25:49 daisy kernel: [143281.721444] [<ffffffff8107bbfe>] ? finish_task_switch+0xce/0x120
Apr 22 09:25:49 daisy kernel: [143281.721448] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:25:49 daisy kernel: [143281.721453] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:25:49 daisy kernel: [143281.721457] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:25:49 daisy kernel: [143281.721461] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:25:49 daisy kernel: [143281.721465] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:25:49 daisy kernel: [143281.721471] [<ffffffff811f9773>] writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:25:49 daisy kernel: [143281.721476] [<ffffffff811f9806>] writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:25:49 daisy kernel: [143281.721480] [<ffffffff81200f38>] __sync_filesystem+0x48/0xa0
Apr 22 09:25:49 daisy kernel: [143281.721484] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:25:49 daisy kernel: [143281.721487] [<ffffffff812016d8>] sys_sync+0x148/0x1a0
Apr 22 09:25:49 daisy kernel: [143281.721492] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:27:49 daisy kernel: [143401.721072] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:27:49 daisy kernel: [143401.721165] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:27:49 daisy kernel: [143401.721253] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:27:49 daisy kernel: [143401.721421] hdparm D ffff88000c778300 0 22246 20845 0 0x00000080
Apr 22 09:27:49 daisy kernel: [143401.721427] ffff88006654bcc8 0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:27:49 daisy kernel: [143401.721432] ffff880028200000 000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:27:49 daisy kernel: [143401.721436] ffff88006654bc68 ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:27:49 daisy kernel: [143401.721441] Call Trace:
Apr 22 09:27:49 daisy kernel: [143401.721450] [<ffffffff810098af>] ? __switch_to+0x16f/0x470
Apr 22 09:27:49 daisy kernel: [143401.721456] [<ffffffff8107bbfe>] ? finish_task_switch+0xce/0x120
Apr 22 09:27:49 daisy kernel: [143401.721460] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:27:49 daisy kernel: [143401.721465] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:27:49 daisy kernel: [143401.721469] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:27:49 daisy kernel: [143401.721473] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:27:49 daisy kernel: [143401.721477] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:27:49 daisy kernel: [143401.721483] [<ffffffff811f9773>] writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:27:49 daisy kernel: [143401.721487] [<ffffffff811f9806>] writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:27:49 daisy kernel: [143401.721492] [<ffffffff81200f38>] __sync_filesystem+0x48/0xa0
Apr 22 09:27:49 daisy kernel: [143401.721495] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:27:49 daisy kernel: [143401.721499] [<ffffffff812016d8>] sys_sync+0x148/0x1a0
Apr 22 09:27:49 daisy kernel: [143401.721503] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:29:49 daisy kernel: [143521.721059] INFO: task hdparm:22246 blocked for more than 120 seconds.
Apr 22 09:29:49 daisy kernel: [143521.721158] Not tainted 2.6.32-042stab142.1 #1
Apr 22 09:29:49 daisy kernel: [143521.721245] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 09:29:49 daisy kernel: [143521.721415] hdparm D ffff88000c778300 0 22246 20845 0 0x00000084
Apr 22 09:29:49 daisy kernel: [143521.721421] ffff88006654bcc8 0000000000000086 ffff88006654bc58 ffffffff810098af
Apr 22 09:29:49 daisy kernel: [143521.721426] ffff880028200000 000000001a42f238 ffff8800110101c0 ffff88011a42f200
Apr 22 09:29:49 daisy kernel: [143521.721431] ffff88006654bc68 ffffffff8107bbfe ffff8800110101c0 0000000000000000
Apr 22 09:29:49 daisy kernel: [143521.721436] Call Trace:
Apr 22 09:29:49 daisy kernel: [143521.721445] [<ffffffff810098af>] ? __switch_to+0x16f/0x470
Apr 22 09:29:49 daisy kernel: [143521.721451] [<ffffffff8107bbfe>] ? finish_task_switch+0xce/0x120
Apr 22 09:29:49 daisy kernel: [143521.721455] [<ffffffff8107c851>] ? update_curr+0xe1/0x1f0
Apr 22 09:29:49 daisy kernel: [143521.721460] [<ffffffff81566c55>] schedule_timeout+0x215/0x2f0
Apr 22 09:29:49 daisy kernel: [143521.721465] [<ffffffff815669b4>] wait_for_completion+0xe4/0x120
Apr 22 09:29:49 daisy kernel: [143521.721469] [<ffffffff81071ce0>] ? default_wake_function+0x0/0x20
Apr 22 09:29:49 daisy kernel: [143521.721473] [<ffffffff815694db>] ? _spin_unlock_bh+0x1b/0x20
Apr 22 09:29:49 daisy kernel: [143521.721479] [<ffffffff811f9773>] writeback_inodes_sb_nr_ub+0x83/0xb0
Apr 22 09:29:49 daisy kernel: [143521.721483] [<ffffffff811f9806>] writeback_inodes_sb_ub+0x46/0x50
Apr 22 09:29:49 daisy kernel: [143521.721487] [<ffffffff81200f38>] __sync_filesystem+0x48/0xa0
Apr 22 09:29:49 daisy kernel: [143521.721491] [<ffffffff8120151d>] sync_filesystems+0x30d/0x350
Apr 22 09:29:49 daisy kernel: [143521.721495] [<ffffffff812016d8>] sys_sync+0x148/0x1a0
Apr 22 09:29:49 daisy kernel: [143521.721499] [<ffffffff81571424>] system_call_fastpath+0x22/0x3a
Apr 22 09:30:04 daisy ata_id[22489]: HDIO_GET_IDENTITY failed for '/dev/sdb'
------------------
I tried running hdparm -tT /dev/sda, but after waiting 5+ minutes for any command output I cancelled it.

I am rsyncing the data from this system over to another system now, clearly something is wrong, but I can't tell what.

The system is an older AMD Opteron 180 processor (dual core) 4 GB ram, RAID controller with RAID 5 set up with 4x 4TB Western Digital Drives.

I rebooted the system day before yesterday, and that's when the timeout messages started pouring into the log.

when I run tw_cli /c8 show, all four drives say they are ok
[root@daisy cron.daily]# tw_cli /c8 show

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u0 RAID-5 OK - - 256K 11175.8 Ri ON

VPort Status Unit Size Type Phy Encl-Slot Model
------------------------------------------------------------------------------
p0 OK u0 3.63 TB SATA 0 - WDC WD4005FZBX-00K5
p1 OK u0 3.63 TB SATA 1 - WDC WD4005FZBX-00K5
p2 OK u0 3.63 TB SATA 2 - WDC WD4005FZBX-00K5
p3 OK u0 3.63 TB SATA 3 - WDC WD4005FZBX-00K5

Logical Volumes appear active:
[root@daisy cron.daily]# lvscan
ACTIVE '/dev/vg_daisy/lv_root' [10.89 TiB] inherit
ACTIVE '/dev/vg_daisy/lv_swap' [3.88 GiB] inherit
ACTIVE '/dev/vg_daisy/lv_home' [20.00 GiB] inherit
[root@daisy cron.daily]#

[root@daisy cron.daily]# lvmdiskscan
/dev/ram0 [ 16.00 MiB]
/dev/root [ 10.89 TiB]
/dev/ram1 [ 16.00 MiB]
/dev/sda1 [ 2.82 TiB]
/dev/vg_daisy/lv_swap [ 3.88 GiB]
/dev/ram2 [ 16.00 MiB]
/dev/vg_daisy/lv_home [ 20.00 GiB]
/dev/ram3 [ 16.00 MiB]
/dev/sda3 [ 842.87 GiB]
/dev/ram4 [ 16.00 MiB]
/dev/ram5 [ 16.00 MiB]
/dev/ram6 [ 16.00 MiB]
/dev/ram7 [ 16.00 MiB]
/dev/ram8 [ 16.00 MiB]
/dev/ram9 [ 16.00 MiB]
/dev/ram10 [ 16.00 MiB]
/dev/ram11 [ 16.00 MiB]
/dev/ram12 [ 16.00 MiB]
/dev/ram13 [ 16.00 MiB]
/dev/ram14 [ 16.00 MiB]
/dev/ram15 [ 16.00 MiB]
/dev/sdb1 [ 1.82 TiB] LVM physical volume
/dev/sdc1 [ 500.00 MiB]
/dev/sdc2 [ 4.00 TiB] LVM physical volume
/dev/sdd1 [ 4.00 TiB] LVM physical volume
/dev/sde1 [ 2.91 TiB] LVM physical volume
3 disks
19 partitions
0 LVM physical volume whole disks
4 LVM physical volumes
[root@daisy cron.daily]#

grub.conf:
[root@daisy grub]# cat grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_daisy-lv_root
# initrd /initrd-[generic-]version.img
#boot=/dev/sdb
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title OpenVZ (2.6.32-042stab142.1)
root (hd0,0)
kernel /vmlinuz-2.6.32-042stab142.1 ro root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-042stab142.1.img
title OpenVZ (2.6.32-042stab141.3)
root (hd0,0)
kernel /vmlinuz-2.6.32-042stab141.3 ro root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-042stab141.3.img
title OpenVZ (2.6.32-042stab140.4)
root (hd0,0)
kernel /vmlinuz-2.6.32-042stab140.4 ro root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-042stab140.4.img
title OpenVZ (2.6.32-042stab140.1)
root (hd0,0)
kernel /vmlinuz-2.6.32-042stab140.1 ro root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-042stab140.1.img
title OpenVZ (2.6.32-042stab139.1)
root (hd0,0)
kernel /vmlinuz-2.6.32-042stab139.1 ro root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-042stab139.1.img
title CentOS 6 (2.6.32-754.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-754.el6.x86_64 ro root=/dev/mapper/vg_daisy-lv_root rd_NO_LUKS rd_LVM_LV=vg_daisy/lv_swap LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=vg_daisy/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
initrd /initramfs-2.6.32-754.el6.x86_64.img
Top is not showing anything out of the ordinary:
[root@daisy grub]#

top - 09:41:57 up 1 day, 16:04, 3 users, load average: 5.89, 5.83, 5.43
Tasks: 369 total, 1 running, 368 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 1.2%sy, 0.0%ni, 25.0%id, 73.5%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 3894628k total, 3861280k used, 33348k free, 95608k buffers
Swap: 4063228k total, 34888k used, 4028340k free, 3139272k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1266 root 20 0 0 0 0 D 1.0 0.0 12:27.75 flush-253:0
21041 1153 20 0 3188 1840 1012 D 0.7 0.0 0:00.72 imap
21599 97 20 0 5160 1940 1568 S 0.7 0.0 0:01.06 imap-login
22636 root 20 0 15272 1524 964 R 0.7 0.0 0:00.06 top
1977 root 20 0 2096 644 360 S 0.3 0.0 0:27.92 dovecot
22528 97 20 0 5160 2044 1672 S 0.3 0.1 0:00.35 imap-login
22578 1155 20 0 2904 1528 940 D 0.3 0.0 0:00.22 imap
1 root 20 0 19236 268 136 S 0.0 0.0 0:00.68 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.04 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:01.88 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.19 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1
9 root 20 0 0 0 0 S 0.0 0.0 0:03.17 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.20 watchdog/1
11 root 20 0 0 0 0 S 0.0 0.0 0:07.23 events/0
12 root 20 0 0 0 0 S 0.0 0.0 0:08.55 events/1
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/0
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/1
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/0
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_long/1
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events_power_ef
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
21 root 20 0 0 0 0 S 0.0 0.0 0:00.01 netns
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
24 root 20 0 0 0 0 S 0.0 0.0 0:00.29 sync_supers

:!:
This is a company production mail server, and I can't find the solution, I need help, as soon as someone is able, thank you!

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Horrible performance HDIO_GET_IDENTITY failed /dev/sdb

Post by TrevorH » 2020/04/22 15:02:03

We cannot support openvz systems here. They are not CentOS and do not use a CentOS kernel. At a guess that looks like either failing hardware or possibly very very busy hardware.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

cwensink
Posts: 3
Joined: 2014/06/24 13:35:20

Re: Horrible performance HDIO_GET_IDENTITY failed /dev/sdb

Post by cwensink » 2020/04/22 15:06:58

Thanks for the input trevor. Is there any way to tell if it's the motherboard, ram, raid controller card, etc?

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Horrible performance HDIO_GET_IDENTITY failed /dev/sdb

Post by TrevorH » 2020/04/22 15:29:06

Since everything there is talking about filesystems, I'd guess disks.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

cwensink
Posts: 3
Joined: 2014/06/24 13:35:20

Re: Horrible performance HDIO_GET_IDENTITY failed /dev/sdb

Post by cwensink » 2020/04/22 16:19:21

Update. I had an 8TB Western Digital Disk plugged into a USB port, in and I unplugged that, and in the last 15 minutes email performance seems to be better. The 3ware controller is reporting all disks internal to it are ok, and the smartctl test passed.

[root@daisy dev]# smartctl -H -d 3ware,0 /dev/twa0
smartctl 5.43 2016-09-28 r4347 [x86_64-linux-2.6.32-042stab142.1] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

-----------------------------------------------------------------------------------------
What other tests other than hdparm should I run to verify the issue is resolved?

Post Reply