Centos 8 will not boot since kernel update

[jamesm75uk]

Today I foolishly pressed CTRL-C in the middle of a "yum update" (which I think may be the cause of breaking it) and then rebooted. Even more foolish I didn't take a new backup before running yum.

Now it wont boot up.

My provider, SimplyHosting.com, has kindly booted it into rescue mode using a Debian rescue image/disk and I can remotely SSH to it. Apparently it won't boot up in the previous kernel either as when I asked them if there were multiple kernel versions on the boot up screen the tech support enginer replied that he already tried the previous kernel version.

After I mounted the root filesystem I can see in the /mnt/var/log/boot.log the following text (weird characters included):

Code: Select all

         Mounting /boot/efi...^M
[^[[0;1;31mFAILED^[[0m] Failed to mount /boot/efi.^M
See 'systemctl status boot-efi.mount' for details.^M
[^[[0;1;33mDEPEND^[[0m] Dependency failed for Local File Systems.^M
         Starting Restore /run/initramfs on shutdown...^M
[^[[0;32m  OK  ^[[0m] Reached target Login Prompts.^M

What else can I do please?

[TrevorH]

Yum installs kernels in parallel so your old working kernel is still installed and should be bootable. All you need to do is select it from the grub menu at boot time. Failing that, you could probably use your debian chroot hack the grub.cfg file around to boot the older one by default and have it reboot into the working one automatically.

[jamesm75uk]

Hi Trevor,

Thanks for your response.

I think the previous kernel isnt bootable. This was my message exchange with SimplyHosting.com:

Code: Select all

James Moore2022-04-29 13:27
When you rebooted the server to put it into rescue mode did you notice if there was an option to boot into multiple versions of the kernel? I say this because the last things I did before it failed to reboot was run "yum update" and this included a kernel update.

If I can boot it into the previous version of the kernel then I could run some diagnostics in a CentOS environment which would be better than this Debian one.

Simply Hosting - Damon Gambrell2022-04-29 13:29
Hi

The previous kernels failed to boot. They went into maintenance mode.

Regards
Damon

Here is the output from fdisk -l , I was also able to mount /dev/mapper/vg01--vg-root to /mnt and mount /dev/sda1 to /mnt/boot

Code: Select all

root@debian:/# fdisk -l
Disk /dev/sda: 80 GiB, 85899345920 bytes, 167772160 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 175B807E-23C7-4DA9-B340-01EB5C1AD31F

Device        Start       End   Sectors  Size Type
/dev/sda1      2048    411647    409600  200M EFI System
/dev/sda2    411648   2508799   2097152    1G Microsoft basic data
/dev/sda3   2508800  41940991  39432192 18.8G Linux LVM
/dev/sda4  41940992 167772126 125831135   60G Linux filesystem


Disk /dev/mapper/vg01--vg-swap: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/vg01--vg-root: 76.8 GiB, 82460016640 bytes, 161054720 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/loop0: 227.4 MiB, 238411776 bytes, 465648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
root@debian:/#

Here is what I can now see in /mnt and /mnt/boot:

Code: Select all

root@debian:/mnt# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  snap  srv  sys  tmp  usr  var
root@debian:/mnt# cd boot
root@debian:/mnt/boot# ls
EFI
root@debian:/mnt/boot# cd EFI
root@debian:/mnt/boot/EFI# ls
BOOT  centos
root@debian:/mnt/boot/EFI# ls -l BOOT
total 1572
-rwxr-xr-x 1 root root 1244496 Aug  1  2020 BOOTX64.EFI
-rwxr-xr-x 1 root root  362264 Aug  1  2020 fbx64.efi
root@debian:/mnt/boot/EFI# ls -l centos
total 5824
-rwxr-xr-x 1 root root     134 Aug  1  2020 BOOTX64.CSV
drwxr-xr-x 2 root root    4096 Nov  8 06:39 fonts
-rwxr-xr-x 1 root root    5063 Jul  1  2020 grub.cfg
-rwxr-xr-x 1 root root    3389 Jul  1  2020 grub.cfg.rpmsave
-rwxr-xr-x 1 root root    1024 Apr 29 07:32 grubenv
-rwxr-xr-x 1 root root 2285848 Nov  8 06:39 grubx64.efi
-rwxr-xr-x 1 root root 1162400 Aug  1  2020 mmx64.efi
-rwxr-xr-x 1 root root 1238416 Aug  1  2020 shimx64-centos.efi
-rwxr-xr-x 1 root root 1244496 Aug  1  2020 shimx64.efi
root@debian:/mnt/boot/EFI#

What do you suggest now , please?

[jamesm75uk]

After comparing the contents of /mnt/boot with a known good Centos 7 server, I have realised that maybe /dev/sda2 is the correct disk partition to mount to /mnt/boot instead of /dev/sda1 and so I have unmounted /dev/sda1 and mounted /dev/sda2 to /mnt/boot. Here are the file contents of /mnt/boot now:

Code: Select all

root@debian:/mnt/boot# ls -l
total 360100
-rw-r--r--. 1 root root   192095 Jun  1  2021 config-4.18.0-305.3.1.el8.x86_64
-rw-r--r--. 1 root root   192887 Jun  1  2021 config-4.18.0-305.3.1.el8.x86_64+debug
-rw-r--r--. 1 root root   193933 Nov 16 14:51 config-4.18.0-348.2.1.el8_5.x86_64
-rw-r--r--. 1 root root   194888 Nov 16 14:42 config-4.18.0-348.2.1.el8_5.x86_64+debug
-rw-r--r--. 1 root root   193933 Dec 22 13:34 config-4.18.0-348.7.1.el8_5.x86_64
-rw-r--r--. 1 root root   194888 Dec 22 13:24 config-4.18.0-348.7.1.el8_5.x86_64+debug
drwxr-xr-x. 2 root root        6 Jun 23  2020 efi
drwxr-xr-x. 2 root root        6 Jul  1  2020 grub
drwx------. 2 root root       40 Jan  6 10:39 grub2
-rw-------. 1 root root 61909155 Jun 23  2020 initramfs-0-rescue-10053d8fcb0b42a699b821b64da733a2.img
-rw-------. 1 root root 29817686 Jun  4  2021 initramfs-4.18.0-305.3.1.el8.x86_64+debug.img
-rw-------. 1 root root 27714273 Dec 10 10:02 initramfs-4.18.0-305.3.1.el8.x86_64.img
-rw-------. 1 root root 29882272 Dec 10 10:01 initramfs-4.18.0-348.2.1.el8_5.x86_64+debug.img
-rw-------. 1 root root 27722608 Dec 10 10:01 initramfs-4.18.0-348.2.1.el8_5.x86_64.img
-rw-------. 1 root root 29883543 Apr 29 08:48 initramfs-4.18.0-348.7.1.el8_5.x86_64+debug.img
-rw-------. 1 root root 27829668 Jan  6 10:42 initramfs-4.18.0-348.7.1.el8_5.x86_64.img
drwxr-xr-x. 3 root root       21 Jul  1  2020 loader
lrwxrwxrwx. 1 root root       57 Dec 10 10:00 symvers-4.18.0-348.2.1.el8_5.x86_64+debug.gz -> /lib/modules/4.18.0-348.2.1.el8_5.x86_64+debug/symvers.gz
lrwxrwxrwx. 1 root root       51 Dec 10 10:00 symvers-4.18.0-348.2.1.el8_5.x86_64.gz -> /lib/modules/4.18.0-348.2.1.el8_5.x86_64/symvers.gz
lrwxrwxrwx. 1 root root       57 Apr 29 08:47 symvers-4.18.0-348.7.1.el8_5.x86_64+debug.gz -> /lib/modules/4.18.0-348.7.1.el8_5.x86_64+debug/symvers.gz
lrwxrwxrwx. 1 root root       51 Jan  6 10:39 symvers-4.18.0-348.7.1.el8_5.x86_64.gz -> /lib/modules/4.18.0-348.7.1.el8_5.x86_64/symvers.gz
-rw-------. 1 root root  4164308 Jun  1  2021 System.map-4.18.0-305.3.1.el8.x86_64
-rw-------. 1 root root  4983245 Jun  1  2021 System.map-4.18.0-305.3.1.el8.x86_64+debug
-rw-------. 1 root root  4259443 Nov 16 14:51 System.map-4.18.0-348.2.1.el8_5.x86_64
-rw-------. 1 root root  5086397 Nov 16 14:42 System.map-4.18.0-348.2.1.el8_5.x86_64+debug
-rw-------. 1 root root  4260003 Dec 22 13:34 System.map-4.18.0-348.7.1.el8_5.x86_64
-rw-------. 1 root root  5087761 Dec 22 13:24 System.map-4.18.0-348.7.1.el8_5.x86_64+debug
-rwxr-xr-x. 1 root root  6762688 Jun 23  2020 vmlinuz-0-rescue-10053d8fcb0b42a699b821b64da733a2
-rwxr-xr-x. 1 root root 10026120 Jun  1  2021 vmlinuz-4.18.0-305.3.1.el8.x86_64
-rwxr-xr-x. 1 root root 22289544 Jun  1  2021 vmlinuz-4.18.0-305.3.1.el8.x86_64+debug
-rwxr-xr-x. 1 root root 10210440 Nov 16 14:51 vmlinuz-4.18.0-348.2.1.el8_5.x86_64
-rwxr-xr-x. 1 root root 22707336 Nov 16 14:42 vmlinuz-4.18.0-348.2.1.el8_5.x86_64+debug
-rwxr-xr-x. 1 root root 10218632 Dec 22 13:34 vmlinuz-4.18.0-348.7.1.el8_5.x86_64
-rwxr-xr-x. 1 root root 22715528 Dec 22 13:24 vmlinuz-4.18.0-348.7.1.el8_5.x86_64+debug
root@debian:/mnt/boot#

Does this help at all, please?

[jamesm75uk]

I've found that /mnt/boot/grub2/grubenv is a symlink to ../efi/EFI/centos/grubenv:

Code: Select all

root@debian:/mnt/boot/grub2# ls -l
total 4
lrwxrwxrwx. 1 root root   25 Nov  8 06:39 grubenv -> ../efi/EFI/centos/grubenv
-rw-r--r--. 1 root root 1024 Jan  6 10:39 grubenv.new
root@debian:/mnt/boot/grub2# cd grubenv
-bash: cd: grubenv: No such file or directory
root@debian:/mnt/boot/grub2#

Does this mean I should mount /dev/sda1 to /mnt/boot/efi and then do some chroot stuff?

[TrevorH]

You'll need to mount both /boot and /boot/efi, in that order. The EFI boot partiton will be vfat or the BIOS would not be able to read it so it should be easy to tell which one it is.

Also, if you mount /, /boot and /boot/efi and also then bind mount (mount --bind ...) /proc /sys and /dev to their equivalents under /mnt then you should be able to chroot /mnt and effectively be inside your CentOS boot filesystem and anything you run should run its CentOS binary.

The problem is that if it is not the kernel that's not working then it's something else and that sounds terminal. But tehre are multiple reasons for entering emergency mode t boot time - the most usual one being that something listed in /etc/fstab did not mount properly. Maybe you can see the real cause of the boot problem by reading further back in the logs.

[jamesm75uk]

Thanks that did the trick. I used this web page as a reference to the mount and chroot commands. https://wiki.debian.org/RescueLive

Then this other webpage as a reference to the reinstallation of the latest kernel which is what fixed it in the end: https://ma.ttias.be/reinstall-the-linux ... s-or-rhel/

thanks for your help Trevor,

James