kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

General support questions
larry.fahnoe
Posts: 5
Joined: 2020/07/30 13:50:51

kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by larry.fahnoe » 2020/07/30 14:15:26

Hello,

I applied last night's kernel / grub / shim security updates on a pair of Dell T110-II servers with UEFI boot. Now neither system will boot. After normal firmware and adapter initialization I get a blank screen and no opportunity to revert to the prior kernel (though I don't know if that would help). These systems have been running well for years & this is the first major CentOS update problem I've experienced, very frustrating! Any words of wisdom would be appreciated!!

Here is the summary of patches installed this morning:

Code: Select all

================================================================================
 Package                Arch      Version                      Repository  Size
================================================================================
Installing:
 kernel                 x86_64    3.10.0-1127.18.2.el7         updates     50 M
Updating:
 grub2-common           noarch    1:2.02-0.86.el7.centos       updates    729 k
 grub2-efi-x64          x86_64    1:2.02-0.86.el7.centos       updates    1.1 M
 grub2-tools            x86_64    1:2.02-0.86.el7.centos       updates    1.8 M
 grub2-tools-extra      x86_64    1:2.02-0.86.el7.centos       updates    1.0 M
 grub2-tools-minimal    x86_64    1:2.02-0.86.el7.centos       updates    174 k
 kernel-tools           x86_64    3.10.0-1127.18.2.el7         updates    8.1 M
 kernel-tools-libs      x86_64    3.10.0-1127.18.2.el7         updates    8.0 M
 mokutil                x86_64    15-7.el7_9                   updates     42 k
 python-perf            x86_64    3.10.0-1127.18.2.el7         updates    8.1 M
 shim-x64               x86_64    15-7.el7_9                   updates    680 k
Removing:
 kernel                 x86_64    3.10.0-1127.el7              @base       64 M

Transaction Summary
================================================================================
Install   1 Package
Upgrade  10 Packages
Remove    1 Package
The updates were successfully applied
--Larry

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by TrevorH » 2020/07/30 15:08:33

The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

larry.fahnoe
Posts: 5
Joined: 2020/07/30 13:50:51

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by larry.fahnoe » 2020/07/30 15:14:21

And for RHEL 7: https://bugzilla.redhat.com/show_bug.cgi?id=1862045

Another related with workarounds: https://bugs.centos.org/view.php?id=17631

Looks like a definite "brown paper bag" release...

--Larry

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by TrevorH » 2020/07/30 16:38:15

This is being tracked upstream on https://bugzilla.redhat.com/show_bug.cgi?id=1862045 (for CentOS 7).

I think it would be useful to gather some data about the exact types of machine that are being affected here. Makes? models? BIOS/UEFI versions? Secure Boot on or off? using UEFI or Legacy BIOS? How you applied the update? (I've seen one report that said the GUI software updater broke it and running yum from the command line after restoring back to the old versions worked).

There is a separate bz entry for the same problem on CentOS 8 and that's https://bugzilla.redhat.com/show_bug.cgi?id=1861977 which I'll also post on the CentOS 8 thread of a similar nature.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

larry.fahnoe
Posts: 5
Joined: 2020/07/30 13:50:51

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by larry.fahnoe » 2020/07/30 17:25:10

Makes? models? BIOS/UEFI versions? Secure Boot on or off? using UEFI or Legacy BIOS? How you applied the update?
Dell T110-II
BIOS 2.10.0
Secure Boot off (I think but did not find it explicitly listed in the BIOS settings)
UEFI boot
Updates applied via yum-cron

My steps to a temporary work-around:
1) boot rescue disk
2) chroot /mnt/sysimage
3) ip address add...
4) ip route add...
5) update /etc/resolv.conf if necessary
6) yum downgrade shim-x64 mokutil grub2\*
7) restore resolve.conf if updated
8) exit
9) reboot

System now boots and runs with the following:
grub2-common-2.02-0.81.el7.centos.noarch
grub2-efi-x64-2.02-0.81.el7.centos.x86_64
grub2-tools-2.02-0.81.el7.centos.x86_64
grub2-tools-extra-2.02-0.81.el7.centos.x86_64
grub2-tools-minimal-2.02-0.81.el7.centos.x86_64
mokutil-15-2.el7.centos.x86_64
shim-x64-15-2.el7.centos.x86_64

Should probably also blacklist the above packages until this is properly resolved.

--Larry

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by TrevorH » 2020/07/30 17:29:32

I think it would be better to add your details to the relevant bugzilla entry directly rather than me acting as a relay.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

larry.fahnoe
Posts: 5
Joined: 2020/07/30 13:50:51

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by larry.fahnoe » 2020/07/30 17:37:37

Indeed it would be & I have now done so, thanks Trevor.

--Larry

man_beach

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by man_beach » 2020/07/31 12:24:38

I have my original CentOS7 boot disk and can get it running from that, including internet (it's what I'm using now). Can anyone give me some tips on workarounds from a boot disk please? The simpler the better - I'm not that proficient.

Thanks

larry.fahnoe
Posts: 5
Joined: 2020/07/30 13:50:51

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by larry.fahnoe » 2020/07/31 12:48:00

Once you boot the CentOS CD/DVD, use Troubleshooting and then Rescue to get to the point where you will be able to access your filesystems. I don't do this often either and did not make notes of the prompts from the CentOS boot disk, but Troubleshooting and Rescue are the keywords to look for. The steps I posted earlier are then what you need to follow to downgrade the errant packages and be able to reboot. The only complication is that you'll likely need to set up networking manually, so before you go there, record your IP, net mask, and default gateway so that you can manually add those once you've booted off the CD. You may also need to set the nameserver IP in /etc/resolv.conf. Once you can ping something like www.google.com, you're good to go with the necessary yum downgrade command. If you did make changes to /etc/resolv.conf, once yum finishes downgrading the packages, revert resolv.conf to its original setting. Good luck & take heart that the process really isn't that difficult.

--Larry

man_beach

Re: kernel 3.10.0-1127.18.2 / grub 2-2.02-0.86 break UEFI boot

Post by man_beach » 2020/07/31 12:57:32

Thanks, Larry. I did look at 'Troubleshooting' but couldn't find any obvious 'Rescue'. I'll have another look.

Post Reply