UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY

A 5 star hangout for overworked and underpaid system admins.
Post Reply
york0822
Posts: 2
Joined: 2022/03/07 11:36:03

UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY

Post by york0822 » 2022/03/07 13:26:35

Hi,

I was asked to maintain a small local (without internet connection) linux cluster built in 2015, but I do not have lots of experience of daily maintenance. The cluster contains management, I/O, storage, and computing nodes, all running CentOS 6. After replacing a bad disk of I/O node, the I/O node asks me to run "fsck" on boot:

Code: Select all

Checking filesystems
/dev/sda3 contains a file system with errors, check forced.
/dev/sda3: Resize inode not valid.

/dev/sda3: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY.
So I ran "fsck" but it ends up with a segmentation fault

Code: Select all

[root@io32 ~]# fsck
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sda3 contains a file system with errors, check forced.
Resize inode not valid. Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Inode 2490389 has an invalid extent
        (logical block 950272, invalid physical block 144233591734273, len 4294017025)
Clear<y>? yes
Inode 2490389, i_blocks is 8, should be 0.  Fix<y>? yes
Inode 2490394 has out of order extents
        (invalid logical block 2, physical block 9994759, len1)
Clear<y>? yes
Inode 2490394, i_blocks is 24, should be 16.  Fix<y>? yes
Inode 2490395, i_size is 201474993487872, should be 0. Fix<y>? yes
Warning... fsck.ext4 for device /dev/sda3 exited with signal 11.
[root@io32 ~]# fsck.ext4 -v /dev/sda3
...
Inode 2490395, i_size is 201474993487872, should be 0. Fix<y>? yes
Segmentation fault
Calling "df" only shows /dev/sda3:

Code: Select all

Filesystem   1K-blocks     Used Available  Use% Mounted on
/dev/sda3    162630140 11970676 142398304    8% /
and the storage (i.e., /backup) is not mounted, yet it is in /etc/fstab:

Code: Select all

UUID=...           /          ext4   defaults  1 1
UUID=...           /boot/efi  vfat   umask=0077,shortname=winnt  0 0
UUID=...           swap       swap   defaults 0 0
tmpfs              /dev/shm   tmpfs  defaults  0 0
devpts             /dev/pts   devpts gid=5,mode=620  0 0
sysfs              /sys       sysfs  defaults  0 0
proc               /proc      proc   defaults  0 0
/dev/mapper/mpathh /backup    xfs    defaults  0 0
/dev/pvf35         /pvfsDATA  gpfs  rw,mtime,atime,dev=pvf35,noauto 0 0
and lsblk:

Code: Select all

NAME            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda               8:0    0 222.2G  0 disk
|-sda1            8:1    0   200M  0 part
|-sda2            8:2    0  64.5G  0 part
|-sda3            8:3    0 157.6G  0 part  /
sdc               8:32   0   1.8T  0 disk
|-mpathb (dm-0) 253:0    0   1.8T  0 mpath
sdb               8:16   0   1.8T  0 disk
|-mpathc (dm-1) 253:1    0   1.8T  0 mpath
sdf               8:80   0  32.7T  0 disk
|-mpathh (dm-2) 253:2    0  32.7T  0 mpath
sde               8:64   0   2.8T  0 disk
|-mpathe (dm-3) 253:3    0   2.8T  0 mpath
sdd               8:48   0   2.7T  0 disk
|-mpathd (dm-4) 253:4    0   2.7T  0 mpath
sdh               8:112  0   2.8T  0 disk
|-mpathe (dm-3) 253:3    0   2.8T  0 mpath
sdg               8:96   0   2.8T  0 disk
|-mpathd (dm-4) 253:4    0   2.8T  0 mpath
sdi               8:128  0  32.7T  0 disk
|-mpathh (dm-2) 253:2    0  32.7T  0 mpath
sr0              11:0    1  1024M  0 rom
My questions are how to 1) mount /backup and 2) fix fsck on boot? I tried to directly call "mount /backup" and I can "ls" and "cd" to the /backup directory but it is still not shown in "df".

Any advice is greatly appreciated. Thank you!

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY

Post by TrevorH » 2022/03/07 14:05:17

I am sorry to tell you but CentOS 6 is dead. It went End Of Life more than 1 year ago and there is no support for it. You need to start a process ASAP to migrate off it to a newer, supported operating system. Since the CentOS 6 forums are now closed and readonly, I have moved your post to the CentOS Social forum.

The system is not mounting the /backup filesystem because it never gets that far in the boot process. You have an error on your root filesystem that fsck cannot fix and that is stopping it from reaching the point where it would process other filesystems. Since you don't say what kernel version it is running, I will sort of guess that it is massively out of date even compared to the last available version of CentOS 6 so you might try downloading the most recent CentOS 6 iso image from vault.centos.org and booting that in rescue mode (either by interrupting the boot process at the initial boot menu and appending the word 'rescue' to the end of the kernel command line or by selecting the Troubleshooting menu and using the Rescue option that is off that). Doing that should give you an option to drop to a command prompt. Do not ask it to mount your filesystems for you and then use that command prompt to attempt to run fsck and see if the bugt that causes it to crash has been fixed. I would not use a more recent CentOS version to boot to rescue in case the ext4 filesystem code has changed in newer versions and that might cause the repair to render the filesystem unusable by the old CentOS 6 code.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

york0822
Posts: 2
Joined: 2022/03/07 11:36:03

Re: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY

Post by york0822 » 2022/03/09 01:24:01

Thank you for the quick reply! I apologize to post this under CentOS 7 as I realized that CentOS 6 was dead and the forum no longer accepts new questions. I prepared a bootable USB of 6.10 and "fsck"ed in rescue mode, where the seg fault was still there. I ended up reinstalling the OS while disconnecting xfs storage node. It turns out nobody is using the gpfs, one less thing to worry about.

Post Reply