[Closed] RHEL 9 - single kernel option at boot

[lightman47]

OK, latest dilemma - I have a machine running Red Hat 9 that backs-up and updates weekly. It boots to Red Hat 9.0, despite it has kernels newer than that (9.3) installed. Also installed are the equivalent kernel-devel and kernel-header packages for each. The only the one shown in uname (below) shows up along with it's 'rescue' followed by the Windows bootloader option.

This morning, from this old kernel (only option) I ran "dnf -y reinstall kernel*.*" (which it did) in the hopes that a 'rebuild' would occur that would correct the situation, but alas, I am still where I am!

I also ran "grub2-set-default 0" and "grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg" - because I think that's where 9.0 stored it. Just now I tried:

Code: Select all

sudo grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline
Generating grub configuration file ...
Found Windows Boot Manager on /dev/nvme0n1p1@/efi/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for UEFI Firmware Settings ...
done
You have new mail in /var/spool/mail/ ...

Several details below - what might I have somehow botched? The machine otherwise runs well with this seeming to be the sole issue. As usual, this is no emergency and I am grateful for the help.

Thank you.

uname -a:

Code: Select all

Linux {machine name} 5.14.0-70.30.1.el9_0.x86_64 #1 SMP PREEMPT Fri Oct 14 10:30:46 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

sudo dnf list installed kernel.x86_64:

Code: Select all

Installed Packages
kernel.x86_64        5.14.0-70.30.1.el9_0         @rhel-9-for-x86_64-baseos-rpms
kernel.x86_64        5.14.0-362.13.1.el9_3        @rhel-9-for-x86_64-baseos-rpms
kernel.x86_64        5.14.0-362.18.1.el9_3        @rhel-9-for-x86_64-baseos-rpms

sudo cat /etc/dnf/dnf.conf:

Code: Select all

[main]
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False

sudo cat /etc/default/grub:

Code: Select all

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_wjoe-swap rd.lvm.lv=rhel_wjoe/root rd.lvm.lv=rhel_wjoe/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

entries from /boot/grub2/grubenv:

Code: Select all

# GRUB Environment Block
# WARNING: Do not edit this file by tools other than grub-editenv!!!
saved_entry=fa3af9fed5a64156ae748b5b6a01e465-5.14.0-362.18.1.el9_3.x86_64
boot_success=1
boot_indeterminate=9

Inxi report:

Code: Select all

System:
  Kernel: 5.14.0-70.30.1.el9_0.x86_64 arch: x86_64 bits: 64 Console: N/A Distro: Red Hat
    Enterprise Linux release 9.3 (Plow)
Machine:
  Type: Desktop Mobo: Micro-Star model: B450 GAMING PRO CARBON MAX WIFI (MS-7B85) v: 1.0
    serial: <filter> UEFI: American Megatrends LLC. v: 2.80 date: 01/13/2022
CPU:
  Info: 8-core model: AMD Ryzen 7 3700X bits: 64 type: MT MCP cache: L2: 4 MiB
  Speed (MHz): avg: 3600 min/max: 2200/4426 cores: 1: 3600 2: 3600 3: 3600 4: 3600 5: 3600
    6: 3600 7: 3600 8: 3600 9: 3600 10: 3600 11: 3600 12: 3600 13: 3600 14: 3600 15: 3600 16: 3600
Graphics:
  Device-1: NVIDIA GF100 [GeForce GTX 480] driver: nouveau v: kernel
  Device-2: SJ-180517-N 1080P Webcam driver: snd-usb-audio,uvcvideo type: USB
  Display: server: X.org v: 1.20.11 with: Xwayland v: 22.1.9 driver: gpu: nouveau tty: 80x40
    resolution: 1: 1920x1080 2: 1920x1080
  API: OpenGL Message: GL data unavailable in console for root.
  API: EGL Message:
Audio:
  Device-1: NVIDIA GF100 High Definition Audio driver: snd_hda_intel
  Device-2: AMD Starship/Matisse HD Audio driver: snd_hda_intel
  Device-3: SJ-180517-N 1080P Webcam driver: snd-usb-audio,uvcvideo type: USB
  API: ALSA v: k5.14.0-70.30.1.el9_0.x86_64 status: kernel-api
Network:
  Device-1: Intel Wireless-AC 9260 driver: iwlwifi
  IF: wlo1 state: down mac: <filter>
  Device-2: Intel I211 Gigabit Network driver: igb
  IF: enp34s0 state: up speed: 1000 Mbps duplex: full mac: <filter>
Bluetooth:
  Device-1: Intel Wireless-AC 9260 Bluetooth Adapter driver: btusb type: USB
  Report: btmgmt ID: hci0 state: up address: <filter> bt-v: 5.1
RAID:
  Device-1: md127 type: mdraid level: mirror status: active size: 7.28 TiB report: 2/2 UU
  Components: Online: 0: sdb 1: sda
Drives:
  Local Storage: total: raw: 16.37 TiB usable: 9.1 TiB used: 1.08 TiB (11.8%)
  ID-1: /dev/nvme0n1 vendor: Crucial model: CT2000P2SSD8 size: 1.82 TiB
  ID-2: /dev/sda vendor: Western Digital model: WD8004FRYZ-01VAEB0 size: 7.28 TiB
  ID-3: /dev/sdb vendor: Western Digital model: WD8004FRYZ-01VAEB0 size: 7.28 TiB
Partition:
  ID-1: / size: 69.97 GiB used: 17.86 GiB (25.5%) fs: xfs dev: /dev/dm-0
  ID-2: /home size: 846.93 GiB used: 13.51 GiB (1.6%) fs: xfs dev: /dev/dm-2
Swap:
  ID-1: swap-1 type: partition size: 31.46 GiB used: 0 KiB (0.0%) dev: /dev/dm-1
Sensors:
  System Temperatures: cpu: 65.8 C mobo: N/A gpu: nouveau temp: 38.0 C
  Fan Speeds (rpm): N/A gpu: nouveau fan: 1260
Info:
  Processes: 406 Uptime: 9m Memory: total: 64 GiB available: 62.42 GiB used: 5.35 GiB (8.6%)
  Init: systemd target: graphical (5) Client: shell wrapper inxi: 3.3.31

[TrevorH]

Does /sys/firmware/efi exist when you are booted?

[lightman47]

Code: Select all

 sudo ls -la /sys/firmware/
total 0
drwxr-xr-x.  7 root root 0 Feb 20 07:47 .
dr-xr-xr-x. 13 root root 0 Feb 20 12:47 ..
drwxr-xr-x.  6 root root 0 Feb 20 07:47 acpi
drwxr-xr-x.  4 root root 0 Feb 20 07:47 dmi
drwxr-xr-x.  6 root root 0 Feb 20 07:47 efi
drwxr-xr-x. 36 root root 0 Feb 20 13:37 memmap
drwxr-xr-x.  2 root root 0 Feb 20 12:47 qemu_fw_cfg

[TrevorH]

The output from `ls -la /etc/grub2.cfg /etc/grub2-efi.cfg` please

[lightman47]

Gladly -

Code: Select all

sudo ls -la /etc/grub2.cfg /etc/grub2-efi.cfg
lrwxrwxrwx. 1 root root 22 Jan  9 11:51 /etc/grub2.cfg -> ../boot/grub2/grub.cfg
lrwxrwxrwx. 1 root root 22 Jan  9 11:51 /etc/grub2-efi.cfg -> ../boot/grub2/grub.cfg

[lightman47]

Only after the latest attempt with the "--update-bls-cmdline" switch did that file exist -> /boot/grub2/grub.cfg

[lightman47]

back to this puzzle - I've been working on it off-and-on with no success

sudo grubby --default-kernel

Code: Select all

/boot/vmlinuz-5.14.0-362.24.1.el9_3.x86_64

But that is NOT what it boots!

uname -a

Code: Select all

Linux wjoe.xxxx 5.14.0-70.30.1.el9_0.x86_64 #1 SMP PREEMPT Fri Oct 14 10:30:46 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

This just boggles my mind. There are 3 kernels installed, the default is set to 0 in /etc/default/grub, it says it should be booting to the latest, but it only offers (and boots to) the OLDEST! Arrgh! Might it be the way RH numbered them - where the oldest actually has the higher version number?

Thank you.

[TrevorH]

Have you got mdadm RAID?

[lightman47]

I do - but for 2 add-in data drives.

[jlehtone]

I also ran "grub2-set-default 0" and "grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg" - because I think that's where 9.0 stored it. Just now I tried:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg --update-bls-cmdline

On EL9 the actual grub.cfg is in /boot/grub2/ and the /boot/efi/EFI/redhat/grub.cfg is a mere stub that includes the /boot/grub2/grub.cfg

Example of the stub:

Code: Select all

# cat /boot/efi/EFI/almalinux/grub.cfg 
search --no-floppy --fs-uuid --set=dev 14fb..5e95
set prefix=($dev)/grub2

export $prefix
configfile $prefix/grub.cfg

Where the 14fb..5e95 is the UUID of the filesystem of /boot

If system boots with UEFI, then UEFI loads /boot/efi/EFI/redhat/grubx64.efi that reads /boot/efi/EFI/redhat/grub.cfg,
which in turn would read /boot/grub2/grub.cfg

You wrote the actual grub.cfg into /boot/efi/EFI/redhat/grub.cfg. The /boot/grub2/grub.cfg in not used.
The /boot/efi/EFI/redhat/grub.cfg looks for 'grubenv', 'user.cfg', 'custom.cfg', etc from /boot/efi/EFI/redhat/
At least your latest grubenv is in /boot/grub2/ -- unused, unless you restore the /boot/efi/EFI/redhat/grub.cfg to the stub it was.
(Install of some grub package does create the stub file.)