Buffer I/O error on device - CentOS

A 5 star hangout for overworked and underpaid system admins.
Post Reply
linconruam
Posts: 2
Joined: 2021/11/19 18:06:55

Buffer I/O error on device - CentOS

Post by linconruam » 2021/11/22 13:27:33

Hi,

We have an mail server configured on a virtual server (VM) running the Linux CentOS 5.9 system.
We identified that for about three weeks now a failure is occurring that is changing the access of the sdb1 partition to read-only.

We identified that the failure is happening during the virtual machine backup time, it may be related to the backup tool, but until last month this failure did not happen and we did not have any changes or updates in the linux server or backup server.

When the sdb1 partition fails as read-only, the solution so far is to restart the server to return to normal.
We can see in the log the information of "Buffer I/O error" in some random blocks of disk sdb1.

What is the recommendation in this case?
Run a disk check via fsck command?
Is there any other procedure?

Server name: vPadova
CentOS Version - 5.9

Accessing the log /var/log/messages:

Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: scsi: Device offlined - not ready after error recovery
Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: scsi: Device offlined - not ready after error recovery
Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: Unhandled error code
Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: SCSI error: return code = 0x07000000
Nov 21 14:41:37 vPADOVA kernel: Result: hostbyte=DID_OK driverbyte=DRIVER_HARD,SUGGEST_OK
Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:37 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 112173824
Nov 21 14:41:37 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:37 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 65012164
Nov 21 14:41:37 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:37 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:37 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 65020100
Nov 21 14:41:54 vPADOVA postfix/smtpd[23851]: auxpropfunc error invalid parameter supplied
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 65020101
Nov 21 14:41:54 vPADOVA postfix/smtpd[23851]: sql_select option missing
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA postfix/smtpd[23851]: auxpropfunc error no mechanism available
Nov 21 14:41:54 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 82313228
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 121634829
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 128450564
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 152043549
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 65011754
Nov 21 14:41:54 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:54 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:54 vPADOVA kernel: Buffer I/O error on device sdb1, logical block 152044577
Nov 21 14:41:55 vPADOVA kernel: lost page write due to I/O error on sdb1
Nov 21 14:41:55 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:55 vPADOVA last message repeated 43 times
Nov 21 14:41:55 vPADOVA kernel: EXT4-fs error (device sdb1): ext4_find_entry: reading directory #38912001 offset 0
Nov 21 14:41:55 vPADOVA kernel: sd 1:0:0:0: rejecting I/O to offline device
Nov 21 14:41:55 vPADOVA last message repeated 14 times

I appreciate if you can help me.

Thanks.
Lincon Ruam

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Buffer I/O error on device - CentOS

Post by TrevorH » 2021/11/22 14:46:19

Well, really, your best path is to install something completely different and much newer. CentOS 5.9 has been out of support since the release of 5.10 and all of CentOS 5 went EOL in 2017. That means that not only are you 2 point releases out of date, the entire distro has been unmaintained for 4 years and in that time there have been numerous high severity security vulnerabilities discovered that make it extremely unwise to use. I just looked up 5..9 and it dates from the start of 2013 so you are actually 8 years out of date and in that time there have been some real doozies of security problems. In addition, the openssl packages etc on CentOS 5 are too old to support any currently recommended security protocols. It doesn't support TLS 1.2 and that's currently the lowest recommended SSL version.

Having said all that, your VM is seeing "hardware" errors on its virtual disk. I would start by looking at the host and see if everything is running correctly there. There may also be certain parameters that need to be set if you are using VMWare in combination with ext3/4 filesystems though it's been such a long time since I heard about that that the details are gone.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

linconruam
Posts: 2
Joined: 2021/11/19 18:06:55

Re: Buffer I/O error on device - CentOS

Post by linconruam » 2021/11/22 15:03:41

TrevorH,

This virtual machine called vPadova is running on the CentOS 5.9 system and is a virtual machine, here at the company we use the Microsoft Hyper-V virtualization solution. This server is our email server - we use postfix.

I understand that upgrading to the latest version would be the ideal scenario, but I would like to know what we can do to fix the "Buffer I/O error on device sdb1" bug, I believe someone has seen this same situation.

Can I see more information about this failure in another system file? I got this information in the "messages" log (/var/log/messages).

Thanks.
Lincon Ruam

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Buffer I/O error on device - CentOS

Post by TrevorH » 2021/11/22 16:07:09

It's an I/O error on the "disk". That's reported to the operating system by the "disk": in this case, that's from Hyper-V. Look there first.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply