Possible SATA related system slowdowns

[troyunverdruss]

Hi all, and thanks in advance for any assistance that you can provide. I wasn't sure if this was the right area to post in since I'm using CentOS, but since it is basically RH, I figured I'd be ok.

I've got a machine with CentOS 5 installed with all the latest updates/patches. This machine is set up to act as a storage server with a decent amount of redundancy on the OS drives as well as a RAID 5 solution for the main storage area. The storage is shared out to Windows via samba over a 1Gbps network link.

The problem that I'm seeing is that every so often, and with increasing frequency, the system just seems to lock up when under heavy network and disk load. Even the console gets "hung" for up to 30 seconds while the system recovers from the problems that are shown in dmesg (shown below), and then it is zippy again. Our technical needs require that this server stream the data consistently and these hangups are causing real pain in our workflow. I tried compiling the newest vanilla kernel (2.6.27) from kernel.org with the CentOS config (which I think is for 2.6.18-92.1.18.el5) but this didn't solve the problem.

I've searched all over and have found that other people are having this problem, but can't find a solution.

Whenever one of these long read/write delays happens, I see some specific output in dmesg like this (this is truncated because it just repeats forever):

[code]SCSI device sdf: 2930277168 512-byte hdwr sectors (1500302 MB)
sdf: Write Protect is off
sdf: Mode Sense: 00 3a 00 00
SCSI device sdf: drive cache: write back
ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
ata1.01: cmd ea/00:00:00:00:00/00:00:00:00:00/b0 tag 0
res 40/00:01:00:4f:c2/00:00:00:00:00/10 Emask 0x4 (timeout)
ata1.01: status: { DRDY }
ata1: port is slow to respond, please be patient (Status 0xd0)
ata1: device not ready (errno=-16), forcing hardreset
ata1: soft resetting link
ata1.00: configured for UDMA/133
ata1.01: configured for UDMA/33
ata1: EH complete
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sdb: 2930277168 512-byte hdwr sectors (1500302 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sdb: 2930277168 512-byte hdwr sectors (1500302 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
ata1.01: cmd ea/00:00:00:00:00/00:00:00:00:00/b0 tag 0
res 40/00:01:00:4f:c2/00:00:00:00:00/10 Emask 0x4 (timeout)
ata1.01: status: { DRDY }
ata1: port is slow to respond, please be patient (Status 0xd0)
ata1: device not ready (errno=-16), forcing hardreset
ata1: soft resetting link
ata1.00: configured for UDMA/133
ata1.01: configured for UDMA/33
ata1: EH complete
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sdb: 2930277168 512-byte hdwr sectors (1500302 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
SCSI device sda: 625142448 512-byte hdwr sectors (320073 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sdb: 2930277168 512-byte hdwr sectors (1500302 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
[/code]

I've also seen some posts that indicated it could be related to ata_piix but I wasn't able to prevent that module from loading to see if there was a potential performance improvement. Any tips on this as well would be nice (I tried adding it to the /etc/modprobe.d/blacklist file)

Any help that you can provide would be greatly appreciated and I can gladly provide more system specs if needed.

Troy

[NedSlider]

Can you give some more info please.

Presumably you think the problem is with the device attached to ata1? What device is attached to that - sda, sdb?

Can we see the chipset/driver info (post output from lspci and lsmod)

Please detail any RAID configurations.

[troyunverdruss]

Sorry for the slow response time, I thought I was supposed to be emailed when a response was posted. I guess I need to check my settings.

I think I've figured out that this is actually related to the HDDs in the RAID5 array. They are the 1.5TB drives that Seagate manufactured and there is a lot of info out there on this now (I was looking for a linux solution, but it was a HW problem).

As a workaround, I was able to (so far) eliminate the delays and freezes by disabling write-cache on the drives with this command:

[code]hdparm -W0 /dev/sdX[/code]

Here's a link to the Seagate FW upgrade thread: http://forums.seagate.com/stx/board/message?board.id=ata_drives&thread.id=2879&view=by_date_ascending&page=21

[NedSlider]

[quote]
troyunverdruss wrote:

I think I've figured out that this is actually related to the HDDs in the RAID5 array. They are the 1.5TB drives that Seagate manufactured and there is a lot of info out there on this now (I was looking for a linux solution, but it was a HW problem).
[/quote]

Ah yes, I'd read about that.

Thanks for posting back with the workaround. Is it possible to update the firmware on these drives?

[toracat]

[quote]
troyunverdruss wrote:
I thought I was supposed to be emailed when a response was posted. I guess I need to check my settings.
[/quote]
If you scroll down to somewhere near the bottom of the page, you will find a Notification method line. You need to change the option to "email". :-)

[bjlockie]

There was a firmware update from Seagate in Nov.

[sydenis]

[quote]but it was a HW problem[/quote]
I dont think that this is HW problem. I have the same trouble on two hdd Samsung Spinpoint SATA2
System is Phenom x4 on amd 770/sb600 chipset, 1 Gb lan
I had Centos 5.1 before and all work nice. Then I was update to Centos 5.2.
After this upgrade I get all thеse troubles...

[NedSlider]

[quote]
sydenis wrote:

I dont think that this is HW problem. I have the same trouble on two hdd Samsung Spinpoint SATA2
[/quote]

Then you have a [i]different[/i] issue and should start a new thread for it.

This thread relates to a known firmware bug affecting 1.5TB Seagate drives.