CentOS newbie errors with OS startup

[munasur12]

Hi,

I installed CentOS 7 which worked great for months. But after a recent restart the OS does not go beyond the a splash screen. The errors I get are that "unable to generate disk quotas" and "failed cannot start NFS server and services". Because I could not get tty, I mounted the system on a live CD and looked at the /etc/fstab. Below is the fstab:

Code: Select all

UUID=3bb90fa8-a996-47d4-ae52-27222355fed6 /                       xfs     defaults        0 0
UUID=e01786a9-e824-460f-9790-84e5711da0bc /boot                   xfs     defaults        0 0
UUID=5713-A4E0          /boot/efi               vfat    umask=0077,shortname=winnt 0 0
UUID=9a0786dd-c454-4197-9abb-e51685413464 swap                    swap    defaults        0 0

LABEL=DATA              /data           ext4    defaults,usrquota,grpquota      1 2
LABEL=HOME              /home           ext4    defaults,usrquota,grpquota      1 2
LABEL=APPS              /apps           ext4    defaults        1 2

As far as I can tell there is no NFS indicated. Any help would be appreciated.

[MartinR]

First off, a couple of tricks when dealing with problems like this:

• When you get to the splash screen, try CTRL/ALT/F2 to get a basic terminal.
• Delete rhgb and quiet from the boot command, typically linux16 in /boot/grub2/grub.cfg, but YMMV.

Next, notice that the error is "failed cannot start NFS server and services". You are not trying to mount a disk using NFS, instead the system is setting up the daemon so that you can share local disks to other machines. If you can get in with the basic terminal mentioned above you can then look in /var/log/messages for more details.

[TrevorH]

Also the error message(s) that are visible on the screen where it dumps you to emergency mode are often not the actual cause of the problem. Try using Shift-PgUp to go further back up the console output and check for previous things that went wrong that might be the cause of the errors that you see.

[munasur12]

Thanks Trevor and Martin,

I am not able to switch ttys. Therefore, I used an Ubuntu Live CD and mounted the CentOS system. When I use Shift-PgUp to go further back up the console I get the following output:

Code: Select all

[
[32m  OK  [0m] Started mlnx_interface_mgr - configure ib0.
[[32m  OK  [0m] Started Dump dmesg to /var/log/dmesg.
[[32m  OK  [0m] Started Load CPU microcode update.
[[32m  OK  [0m] Started Show Plymouth Boot Screen.
[[32m  OK  [0m] Started Forward Password Requests to Plymouth Directory Watch.
[[32m  OK  [0m] Reached target Paths.
[[32m  OK  [0m] Reached target Basic System.
%G%G%G[[32m  OK  [0m] Found device LOGICAL_VOLUME 4.
         Starting File System Check on /dev/disk/by-uuid/3bb90fa8-a996-47d4-ae52-27222355fed6...
[[32m  OK  [0m] Started File System Check on /dev/disk/by-uuid/3bb90fa8-a996-47d4-ae52-27222355fed6.
[[32m  OK  [0m] Started dracut initqueue hook.
         Mounting /sysroot...
[[32m  OK  [0m] Reached target Remote File Systems (Pre).
[[32m  OK  [0m] Reached target Remote File Systems.
[[32m  OK  [0m] Mounted /sysroot.
[[32m  OK  [0m] Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[[32m  OK  [0m] Started Reload Configuration from the Real Root.
[[32m  OK  [0m] Reached target Initrd File Systems.
[[32m  OK  [0m] Reached target Initrd Default Target.
         Starting dracut pre-pivot and cleanup hook...
[[32m  OK  [0m] Started dracut pre-pivot and cleanup hook.
         Starting Cleaning Up and Shutting Down Daemons...
[[32m  OK  [0m] Stopped target Timers.
         Starting Plymouth switch root service...
[[32m  OK  [0m] Stopped Cleaning Up and Shutting Down Daemons.
[[32m  OK  [0m] Stopped dracut pre-pivot and cleanup hook.
[[32m  OK  [0m] Stopped target Remote File Systems.
[[32m  OK  [0m] Stopped target Remote File Systems (Pre).
[[32m  OK  [0m] Stopped dracut initqueue hook.
[[32m  OK  [0m] Stopped target Initrd Default Target.
[[32m  OK  [0m] Stopped target Basic System.
[[32m  OK  [0m] Stopped target Paths.
[[32m  OK  [0m] Stopped target Slices.
[[32m  OK  [0m] Stopped target Sockets.
[[32m  OK  [0m] Stopped target System Initialization.
[[32m  OK  [0m] Stopped udev Coldplug all Devices.
[[32m  OK  [0m] Stopped target Swap.
         Stopping udev Kernel Device Manager...
[[32m  OK  [0m] Stopped target Local File Systems.
[[32m  OK  [0m] Stopped Apply Kernel Variables.
[[32m  OK  [0m] Stopped udev Kernel Device Manager.
[[32m  OK  [0m] Stopped Create Static Device Nodes in /dev.
[[32m  OK  [0m] Stopped Create list of required static device nodes for the current kernel.
[[32m  OK  [0m] Stopped dracut pre-udev hook.
[[32m  OK  [0m] Stopped dracut cmdline hook.
[[32m  OK  [0m] Closed udev Kernel Socket.
[[32m  OK  [0m] Closed udev Control Socket.
         Starting Cleanup udevd DB...
[[32m  OK  [0m] Started Plymouth switch root service.
[[32m  OK  [0m] Started Cleanup udevd DB.
[[32m  OK  [0m] Reached target Switch Root.
         Starting Switch Root...

Welcome to [0;31mCentOS Linux 7 (Core)[0m!

[[32m  OK  [0m] Stopped Switch Root.
[[32m  OK  [0m] Stopped Journal Service.
         Starting Journal Service...
[[32m  OK  [0m] Started Forward Password Requests to Wall Directory Watch.
[[32m  OK  [0m] Listening on Device-mapper event daemon FIFOs.
[[32m  OK  [0m] Listening on udev Kernel Socket.
[[32m  OK  [0m] Created slice User and Session Slice.
[[32m  OK  [0m] Created slice system-selinux\x2dpolicy\x2dmigrate\x2dlocal\x2dchanges.slice.
[[32m  OK  [0m] Created slice system-getty.slice.
[[32m  OK  [0m] Reached target Login Prompts.
[[32m  OK  [0m] Stopped target Switch Root.
[[32m  OK  [0m] Reached target RPC Port Mapper.
[[32m  OK  [0m] Listening on Delayed Shutdown Socket.
[[32m  OK  [0m] Reached target User and Group Name Lookups.
         Mounting POSIX Message Queue File System...
[[32m  OK  [0m] Stopped target Initrd Root File System.
         Mounting Cpuset /dev/cpuset mount...
[[32m  OK  [0m] Listening on LVM2 poll daemon socket.
[[32m  OK  [0m] Created slice Virtual Machine and Container Slice.
[[32m  OK  [0m] Listening on LVM2 metadata daemon socket.
         Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[[1;39m INFO [0m] dev-virtio\x2dports-com.redhat.spice.0.device is not active.
[[1;33mDEPEND[0m] Dependency failed for Activation socket for spice guest agent daemon.
         Starting Read and set NIS domainname from /etc/sysconfig/network...
[[32m  OK  [0m] Stopped File System Check on Root Device.
         Starting Collect Read-Ahead Data...
         Starting Enable cpuset memory spreading in top cpuset...
[[32m  OK  [0m] Stopped target Initrd File Systems.
         Mounting Huge Pages File System...
[[32m  OK  [0m] Listening on /dev/initctl Compatibility Named Pipe.
[[32m  OK  [0m] Listening on udev Control Socket.
[[32m  OK  [0m] Reached target Slices.
[[32m  OK  [0m] Reached target Host and Network Name Lookups.
[[32m  OK  [0m] Created slice system-systemd\x2dfsck.slice.
         Mounting NFSD configuration filesystem...
[[32m  OK  [0m] Set up automount Arbitrary Executable File Formats File System Automount Point.
         Starting Kernel Module supporting RPCSEC_GSS...
         Starting Replay Read-Ahead Data...
         Starting Create list of required static device nodes for the current kernel...
         Mounting Debug File System...
[[32m  OK  [0m] Mounted Cpuset /dev/cpuset mount.
[[32m  OK  [0m] Mounted POSIX Message Queue File System.
[[32m  OK  [0m] Mounted Huge Pages File System.
[[32m  OK  [0m] Mounted Debug File System.
[[32m  OK  [0m] Started Collect Read-Ahead Data.
[[32m  OK  [0m] Started Journal Service.
[[32m  OK  [0m] Started Replay Read-Ahead Data.
[[32m  OK  [0m] Started Read and set NIS domainname from /etc/sysconfig/network.
[[32m  OK  [0m] Started Enable cpuset memory spreading in top cpuset.
[[32m  OK  [0m] Started Create list of required static device nodes for the current kernel.
         Starting Load legacy module configuration...
         Starting Remount Root and Kernel File Systems...
         Starting Apply Kernel Variables...
         Starting Create Static Device Nodes in /dev...
         Starting Initialize cpuset cache /var/run/cpunodemap...
[[32m  OK  [0m] Started LVM2 metadata daemon.
[[32m  OK  [0m] Started Remount Root and Kernel File Systems.
         Starting udev Coldplug all Devices...
         Starting Configure read-only root support...
         Starting Flush Journal to Persistent Storage...
[[32m  OK  [0m] Started Apply Kernel Variables.
[[32m  OK  [0m] Started Initialize cpuset cache /var/run/cpunodemap.
[[32m  OK  [0m] Started Flush Journal to Persistent Storage.
[[32m  OK  [0m] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[[32m  OK  [0m] Started Load legacy module configuration.
[[32m  OK  [0m] Started Kernel Module supporting RPCSEC_GSS.
[[32m  OK  [0m] Started Configure read-only root support.
         Starting Load/Save Random Seed...
[[32m  OK  [0m] Started Load/Save Random Seed.
[[32m  OK  [0m] Started udev Coldplug all Devices.
[[32m  OK  [0m] Mounted NFSD configuration filesystem.
         Starting udev Wait for Complete Device Initialization...
[[32m  OK  [0m] Started udev Kernel Device Manager.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA1.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA4.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA5.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA6.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA2.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA3.
[[32m  OK  [0m] Found device LOGICAL_VOLUME 3.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA8.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA.
[[32m  OK  [0m] Found device LOGICAL_VOLUME DATA7.
[[32m  OK  [0m] Found device LOGICAL_VOLUME 2.
[[32m  OK  [0m] Found device LOGICAL_VOLUME EFI\x20System\x20Partition.
[[32m  OK  [0m] Found device LOGICAL_VOLUME APPS.
[[32m  OK  [0m] Found device LOGICAL_VOLUME HOME.
         Activating swap /dev/disk/by-uuid/9a0786dd-c454-4197-9abb-e51685413464...
[[32m  OK  [0m] Activated swap /dev/disk/by-uuid/9a0786dd-c454-4197-9abb-e51685413464.
[[32m  OK  [0m] Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[[32m  OK  [0m] Reached target Local File Systems (Pre).
         Starting File System Check on /dev/disk/by-label/DATA5...
         Starting File System Check on /dev/disk/by-label/DATA8...
         Starting File System Check on /dev/disk/by-label/DATA...
         Mounting /boot...
         Starting File System Check on /dev/disk/by-label/HOME...
         Starting File System Check on /dev/disk/by-label/DATA1...
         Starting File System Check on /dev/disk/by-label/DATA3...
         Starting File System Check on /dev/disk/by-label/DATA7...
         Starting File System Check on /dev/disk/by-label/APPS...
         Starting File System Check on /dev/disk/by-label/DATA4...
         Starting File System Check on /dev/disk/by-label/DATA2...
         Starting File System Check on /dev/disk/by-label/DATA6...
[[32m  OK  [0m] Reached target Swap.
[[32m  OK  [0m] Mounted /boot.
         Mounting /boot/efi...
[[32m  OK  [0m] Mounted /boot/efi.
[    7.534264] systemd-fsck[8043]: DATA8: clean, 10/366215168 files, 23351519/5859426304 blocks
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA8.
         Mounting /data8...
[    7.566662] systemd-fsck[8139]: APPS: clean, 429946/412219392 files, 102164370/6595501755 blocks
%G[    7.585157] systemd-fsck[8083]: HOME: clean, 1431147/503316480 files, 179486945/8053063680 blocks
%G[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/APPS.
[    7.640385] systemd-fsck[8114]: DATA3: clean, 10/915535872 files, 58336461/14648565760 blocks
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/HOME.
[    7.693647] systemd-fsck[8022]: DATA5: clean, 10/915535872 files, 58336461/14648565760 blocks
[    7.693874] systemd-fsck[8062]: DATA contains a file system with errors, check forced.
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA5.
[    7.694187] systemd-fsck[8103]: DATA1 contains a file system with errors, check forced.
[    7.694514] systemd-fsck[8150]: DATA4: clean, 10/915535872 files, 58336461/14648565760 blocks
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA3.
[    7.695125] systemd-fsck[8164]: DATA2: clean, 3575/915535872 files, 3324413702/14648565760 blocks
         Mounting /data3...
         Mounting /data5...
         Mounting /home...
         Mounting /apps...
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA4.
         Mounting /data4...
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA2.
         Mounting /data2...
[    7.734312] systemd-fsck[8179]: DATA6: clean, 10/915535872 files, 58336461/14648565760 blocks
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA6.
         Mounting /data6...
[    7.767893] systemd-fsck[8129]: DATA7: clean, 10/915535872 files, 58336461/14648565760 blocks
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA7.
         Mounting /data7...
[[32m  OK  [0m] Mounted /data8.
[[32m  OK  [0m] Mounted /apps.
[[32m  OK  [0m] Mounted /home.
[[32m  OK  [0m] Mounted /data2.
[[32m  OK  [0m] Mounted /data5.
[[32m  OK  [0m] Mounted /data4.
[[32m  OK  [0m] Mounted /data3.
[[32m  OK  [0m] Mounted /data6.
[[32m  OK  [0m] Mounted /data7.

DATA1: fsck 0.0% complete...

DATA1: fsck 4.7% complete...

DATA1: fsck 9.9% complete...

DATA1: fsck 15.0% complete...

DATA1: fsck 20.1% complete...

DATA1: fsck 25.2% complete...

DATA1: fsck 30.3% complete...

DATA1: fsck 35.4% complete...

DATA1: fsck 40.5% complete...

DATA1: fsck 45.5% complete...

DATA1: fsck 50.6% complete...

DATA1: fsck 55.7% complete...

DATA1: fsck 60.7% complete...
[   18.689467] systemd-fsck[8103]: DATA1: Inodes that were part of a corrupted orphan linked list found.
[   18.690383] systemd-fsck[8103]: DATA1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
[   18.691032] systemd-fsck[8103]: (i.e., without -a or -p options)

                               

DATA: fsck 8.8% complete...
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA1.
         Mounting /data1...
[[32m  OK  [0m] Started Stop Read-Ahead Data Collection 10s After Completed Startup.
[[32m  OK  [0m] Reached target Timers.
[[32m  OK  [0m] Reached target Login Prompts (Pre).
[[32m  OK  [0m] Reached target Paths.
[[32m  OK  [0m] Reached target Sockets.

DATA: fsck 8.8% complete...

DATA: fsck 8.8% complete...

DATA: fsck 8.8% complete...
[[32m  OK  [0m] Started udev Wait for Complete Device Initialization.
[[32m  OK  [0m] Mounted /data1.
         Starting Activation of DM RAID sets...
[[32m  OK  [0m] Started Activation of DM RAID sets.
[[32m  OK  [0m] Reached target Local Encrypted Volumes.

DATA: fsck 8.8% complete...

DATA: fsck 8.8% complete...

DATA: fsck 8.8% complete...

DATA: fsck 8.8% complete...

DATA: fsck 8.9% complete...

DATA: fsck 9.1% complete...

DATA: fsck 9.2% complete...

DATA: fsck 9.4% complete...

DATA: fsck 9.6% complete...

DATA: fsck 9.8% complete...

DATA: fsck 11.0% complete...

DATA: fsck 15.5% complete...

DATA: fsck 20.4% complete...

DATA: fsck 25.5% complete...

DATA: fsck 31.1% complete...

DATA: fsck 36.1% complete...

DATA: fsck 39.5% complete...

DATA: fsck 42.3% complete...

DATA: fsck 42.3% complete...
[   21.381038] systemd-fsck[8062]: DATA: Inodes that were part of a corrupted orphan linked list found.
[   21.381736] systemd-fsck[8062]: DATA: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
[   21.382316] systemd-fsck[8062]: (i.e., without -a or -p options)

                              
[[32m  OK  [0m] Started File System Check on /dev/disk/by-label/DATA.
         Mounting /data...
[[32m  OK  [0m] Mounted /data.
         Starting File System Quota Check...
[[32m  OK  [0m] Started File System Quota Check.
         Starting Enable File System Quotas...
[[1;31mFAILED[0m] Failed to start Enable File System Quotas.
See 'systemctl status quotaon.service' for details.
[[32m  OK  [0m] Reached target Local File Systems.
         Starting Tell Plymouth To Write Out Runtime Data...
         Starting openibd - configure Mellanox devices...
         Starting Preprocess NFS configuration...
         Starting Import network configuration from initramfs...
[[32m  OK  [0m] Started Tell Plymouth To Write Out Runtime Data.
[[32m  OK  [0m] Started Preprocess NFS configuration.
[[32m  OK  [0m] Started Import network configuration from initramfs.
         Starting Create Volatile Files and Directories...
[[32m  OK  [0m] Started Emergency Shell.
[[32m  OK  [0m] Reached target Emergency Mode.
[[32m  OK  [0m] Started Create Volatile Files and Directories.
         Mounting RPC Pipe File System...
         Starting Security Auditing Service...
[[32m  OK  [0m] Mounted RPC Pipe File System.
[[32m  OK  [0m] Reached target rpc_pipefs.target.
         Starting NFSv4 ID-name mapping service...
         Starting RPC security service for NFS client and server...
[[32m  OK  [0m] Started NFSv4 ID-name mapping service.
[[32m  OK  [0m] Started RPC security service for NFS client and server.
[[32m  OK  [0m] Reached target NFS client services.
[[32m  OK  [0m] Started Security Auditing Service.
         Starting Update UTMP about System Boot/Shutdown...
[[32m  OK  [0m] Started Update UTMP about System Boot/Shutdown.
         Starting Update UTMP about System Runlevel Changes...
[[32m  OK  [0m] Started Update UTMP about System Runlevel Changes.
         Starting Activation of DM RAID sets...
[[32m  OK  [0m] Stopped target Emergency Mode.
[[32m  OK  [0m] Created slice system-mlnx_interface_mgr.slice.
         Starting Tell Plymouth To Write Out Runtime Data...
[[32m  OK  [0m] Started Activation of DM RAID sets.
[[32m  OK  [0m] Started Tell Plymouth To Write Out Runtime Data.
[[32m  OK  [0m] Started openibd - configure Mellanox devices.
[[32m  OK  [0m] Reached target Network.
[[32m  OK  [0m] Reached target Network is Online.
         Starting NFS Mount Daemon...
         Starting Logout off all iSCSI sessions on shutdown...
[[32m  OK  [0m] Started Logout off all iSCSI sessions on shutdown.
[[32m  OK  [0m] Reached target Remote File Systems (Pre).
[[32m  OK  [0m] Reached target Remote File Systems.
         Starting Availability of block devices...
[[32m  OK  [0m] Started Availability of block devices.
[[32m  OK  [0m] Started NFS Mount Daemon.
         Starting NFS server and services...
[[1;31mFAILED[0m] Failed to start NFS server and services.
See 'systemctl status nfs-server.service' for details.
         Stopping NFS Mount Daemon...
         Stopping NFSv4 ID-name mapping service...
         Starting Notify NFS peers of a restart...
[[32m  OK  [0m] Stopped NFSv4 ID-name mapping service.
[[32m  OK  [0m] Stopped NFS Mount Daemon.
[[32m  OK  [0m] Started Notify NFS peers of a restart.
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
try again to boot into default mode.
[[32m  OK  [0m] Stopped Emergency Shell.
[[32m  OK  [0m] Reached target System Initialization.
[[32m  OK  [0m] Reached target Basic System.
         Starting Dump dmesg to /var/log/dmesg...
         Starting Load CPU microcode update...
[[32m  OK  [0m] Started mlnx_interface_mgr - configure ib0.
[[32m  OK  [0m] Started Dump dmesg to /var/log/dmesg.
[[32m  OK  [0m] Started Load CPU microcode update.

[TrevorH]

Your log there shows that it's running fsck against a lot of your filesystems and several of them are failing. You'll need to boot from rescue media and run fsck manually to fix the errors listed - they are apparently beyond the ability of the auto-repair to fix and need manual intervention.

[munasur12]

Thanks Trevor,

I tried fsck on one of the filesystem and I get the following output:

Code: Select all

fsck from util-linux 2.34
e2fsck 1.45.5 (07-Jan-2020)
fsck.ext2: Is a directory while trying to open /media/ubuntu/DATA

The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

Then I tried mke2fs to find the backup blocks and get the output below:

Code: Select all

mke2fs 1.45.5 (07-Jan-2020)
mke2fs: Device size reported to be zero.  Invalid partition specified, or
        partition table wasn't reread after running fdisk, due to
        a modified partition being busy and in use.  You may need to reboot
        to re-read your partition table.

[TrevorH]

Don't ever use mk[e2]fs on a filesystem that contains anything useful.

Looks like you're trying to run fsck against the mountpoint, not against the device it sits on.

[munasur12]

Thanks again TrevorH,

I was able to pull data off the drives and used the disks application in Ubuntu to repair the drives. I was able to save 100tb of data. Honestly, you guys are awesome!!