Urgent Attention : I/O error occurred while writing

[naresh.yakkati]

Dear All,

We are having I/O error issue on one of our critical system which is running on CentOS 7.6.

can someone please help to resolve the issue. Below are logs.

Thanks in advance.

2020-10-25T05:05:30+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='108', error='Connection refused (111)'
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 1717379272
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 2189117728
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 2189112656
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 2157661480
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 2157661312
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 396169384
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 396169264
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 364752744
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 364752488
2020-10-24T22:27:05+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 2723859264
2020-10-24T22:27:05+11:00 centos7 kernel: XFS (dm-2): metadata I/O error: block 0x12a1b7db8 ("xfs_trans_read_buf_map") error 5 numblks 8
2020-10-24T22:27:05+11:00 centos7 kernel: XFS (dm-2): metadata I/O error: block 0x1b2f984e0 ("xfs_trans_read_buf_map") error 5 numblks 16
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 617323033
2020-10-24T22:27:05+11:00 centos7 kernel: XFS (dm-2): metadata I/O error: block 0x1d88b0e20 ("xfs_trans_read_buf_map") error 5 numblks 16
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 676290340
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 676290341
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 676290342
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 676289706
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 672358309
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 672358288
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 672358289
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 672358290
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-24T22:27:05+11:00 centos7 kernel: Buffer I/O error on device dm-2, logical block 452171797
2020-10-24T22:27:05+11:00 centos7 kernel: lost page write due to I/O error on dm-2
2020-10-25T05:05:40+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='235', error='Connection refused (111)'
2020-10-25T05:05:50+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='268', error='Connection refused (111)'
2020-10-25T05:06:00+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='286', error='Connection refused (111)'
2020-10-26T05:03:22+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='323', error='Connection refused (111)'
2020-10-25T22:23:42+11:00 centos7 kernel: end_request: I/O error, dev sdc, sector 1437037656
2020-10-26T05:03:36+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='42', error='Connection refused (111)'
2020-10-26T05:03:46+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='219', error='Connection refused (111)'
2020-10-26T05:03:57+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='244', error='Connection refused (111)'
2020-10-27T05:03:38+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='285', error='Connection refused (111)'
2020-10-26T22:24:50+11:00 centos7 kernel: end_request: I/O error, dev sdd, sector 2023165944
2020-10-27T05:03:48+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='107', error='Connection refused (111)'
2020-10-27T05:03:58+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='124', error='Connection refused (111)'
2020-10-28T05:01:58+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='86', error='Connection refused (111)'
2020-10-28T05:02:08+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='101', error='Connection refused (111)'
2020-10-28T05:02:18+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='109', error='Connection refused (111)'
2020-10-28T05:02:28+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='29', error='Connection refused (111)'
2020-10-28T05:02:38+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='60', error='Connection refused (111)'
2020-10-28T05:02:48+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='117', error='Connection refused (111)'
2020-10-28T05:02:58+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='120', error='Connection refused (111)'
2020-10-28T05:03:47+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='34', error='Connection refused (111)'
2020-10-28T05:03:57+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='64', error='Connection refused (111)'
2020-10-28T05:04:35+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='90', error='Connection refused (111)'
2020-10-28T05:04:45+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='53', error='Connection refused (111)'
2020-10-28T05:04:55+11:00 centos7 syslog-ng[26239]: I/O error occurred while writing; fd='143', error='Connection refused (111)'
2020-10-29T09:18:46+11:00 centos7 syslog-ng[3201]: I/O error occurred while writing; fd='11', error='Connection refused (111)'
2020-10-29T09:18:56+11:00 centos7 syslog-ng[3201]: I/O error occurred while writing; fd='165', error='Connection refused (111)'

[TrevorH]

Is your /dev/sdc a physical hard disk in the machine? Or is it some sort of network attached device?

If it's a physical disk in the server then it's dying and needs to be replaced. If it's a network (iSCSI?) device then it looks like you have network connection problems... or the device is dying.

[naresh.yakkati]

Hi Trevor,

Thanks for your reply.

The server is a VM and running on ESXi host. yes it's a iSCSI device.

I can see the I/O errors on two disks sdc and sdd which are coming from same datastore.

is the below error related to the same I/O error or something related to syslog-ng service?

"syslog-ng[26239]: I/O error occurred while writing; fd='235', error='Connection refused (111)"

[TrevorH]

Well syslog-ng would get similar errors if you had network problems at the time.

[naresh.yakkati]

Hi Trevor.

Thank you again for your time on this.

can we check the disk health or network related problems with any commands or logs.