Unable to boot CentOS 7.x since kernel 3.10.0-1160.36.2.el7.x86_64

[ib.hojme]

Hello,

I have searched but been unable to find a scenario quite like mine.
The latest kernel I can boot successfully from is 3.10.0-1160.31.1.el7.x86_64, all later kernels end up in a blank screen with a blinking cursor in the upper left hand corner. I have not been able to fugyre out where to find a log where this might be written to.
Can any one point me in the right direction in order to solve this issue ?

Thanks in advance.

Yours sincerely

Ib Højme

hw :
Lenovo P50, 32GB RAM, 2 disks (1x128GB, 1x500GB)

uname -a :
Linux l001730 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

messages :
[ 0.000000] Linux version 3.10.0-1160.31.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Thu Jun 10 13:32:12 UTC 2021
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-1160.31.1.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_GB.UTF-8

/boot :
-rw-r--r--. 1 root root 153596 28 apr 23:53 config-3.10.0-1160.25.1.el7.x86_64
-rw-r--r--. 1 root root 153596 10 jun 15:36 config-3.10.0-1160.31.1.el7.x86_64
-rw-r--r--. 1 root root 153596 21 jul 14:01 config-3.10.0-1160.36.2.el7.x86_64
-rw-r--r--. 1 root root 153596 31 aug 16:57 config-3.10.0-1160.41.1.el7.x86_64
-rw-r--r--. 1 root root 153596 7 sep 16:54 config-3.10.0-1160.42.2.el7.x86_64
drwx------. 3 root root 17 29 jul 2020 efi
drwxr-xr-x. 2 root root 4096 22 jul 2019 extlinux
drwxr-xr-x. 2 root root 27 22 jul 2019 grub
drwx------. 5 root root 132 13 sep 07:05 grub2
-rw-------. 1 root root 74028321 22 jul 2019 initramfs-0-rescue-ca2010da78034ec0b08d83b28bcf1387.img
-rw-------. 1 root root 33536770 17 jun 07:13 initramfs-3.10.0-1160.25.1.el7.x86_64.img
-rw-------. 1 root root 15920143 7 maj 10:38 initramfs-3.10.0-1160.25.1.el7.x86_64kdump.img
-rw-------. 1 root root 33531710 17 jun 07:13 initramfs-3.10.0-1160.31.1.el7.x86_64.img
-rw-------. 1 root root 15916232 17 jun 07:15 initramfs-3.10.0-1160.31.1.el7.x86_64kdump.img
-rw-------. 1 root root 33639044 6 sep 07:48 initramfs-3.10.0-1160.36.2.el7.x86_64.img
-rw-------. 1 root root 15913987 8 aug 19:30 initramfs-3.10.0-1160.36.2.el7.x86_64kdump.img
-rw-------. 1 root root 33641883 6 sep 07:51 initramfs-3.10.0-1160.41.1.el7.x86_64.img
-rw-------. 1 root root 33533463 13 sep 07:05 initramfs-3.10.0-1160.42.2.el7.x86_64.img
-rw-r--r--. 1 root root 320662 28 apr 23:53 symvers-3.10.0-1160.25.1.el7.x86_64.gz
-rw-r--r--. 1 root root 320759 10 jun 15:36 symvers-3.10.0-1160.31.1.el7.x86_64.gz
-rw-r--r--. 1 root root 320757 21 jul 14:01 symvers-3.10.0-1160.36.2.el7.x86_64.gz
-rw-r--r--. 1 root root 320757 31 aug 16:57 symvers-3.10.0-1160.41.1.el7.x86_64.gz
-rw-r--r--. 1 root root 320757 7 sep 16:54 symvers-3.10.0-1160.42.2.el7.x86_64.gz
-rw-------. 1 root root 3618381 28 apr 23:53 System.map-3.10.0-1160.25.1.el7.x86_64
-rw-------. 1 root root 3620234 10 jun 15:36 System.map-3.10.0-1160.31.1.el7.x86_64
-rw-------. 1 root root 3620596 21 jul 14:01 System.map-3.10.0-1160.36.2.el7.x86_64
-rw-------. 1 root root 3620596 31 aug 16:57 System.map-3.10.0-1160.41.1.el7.x86_64
-rw-------. 1 root root 3620596 7 sep 16:54 System.map-3.10.0-1160.42.2.el7.x86_64
-rwxr-xr-x. 1 root root 6639904 22 jul 2019 vmlinuz-0-rescue-ca2010da78034ec0b08d83b28bcf1387
-rwxr-xr-x. 1 root root 6773352 28 apr 23:53 vmlinuz-3.10.0-1160.25.1.el7.x86_64
-rwxr-xr-x. 1 root root 6773352 10 jun 15:36 vmlinuz-3.10.0-1160.31.1.el7.x86_64
-rwxr-xr-x. 1 root root 6777448 21 jul 14:01 vmlinuz-3.10.0-1160.36.2.el7.x86_64
-rwxr-xr-x. 1 root root 6773352 31 aug 16:57 vmlinuz-3.10.0-1160.41.1.el7.x86_64
-rwxr-xr-x. 1 root root 6773352 7 sep 16:54 vmlinuz-3.10.0-1160.42.2.el7.x86_64

[TrevorH]

Have you install proprietary video drivers from nvidia or ATI?

[ib.hojme]

Hello Trevor,

not knowingly so to speak
On the other hand, there is a nouveau driver which I beleive is the NVidia driver.

Module Size Used by

i915 1859232 30
nouveau 1899494 1
drm_panel_orientation_quirks 17180 1 drm
wmi 21636 3 mxm_wmi,nouveau,intel_wmi_thunderbolt
video 24538 2 i915,nouveau

The X11 driver installed is xorg-x11-drv-nouveau-1:1.0.15.1.el7 (64-bit)

Yours sincerely

Ib Højme

[TrevorH]

No, nouveau will not be the problem, or at least not the problem that I'm thinking of, as it is included as part of the distro kernel so does not need to be rebuilt for the new kernel (it already has been).

Your next step is to go read logs. Start with /var/log/messages and /var/log/Xorg..*.log and see if there are clues in those.

[tunk]

Probably won't work, can you get to console login with crtl+alt+F2?
If you can, then it could be a problem with the grapichs card and driver.

[ib.hojme]

Hello,

I have not had any success with changing to a console as described.
So I'll go with the logs for now.

Yours sincerely

Ib Højme

[jlehtone]

Probably won't work, can you get to console login with crtl+alt+F2?

Normally the GUI X11 session is on Ctrl+Alt+F1 and F2-F6(?) are virtual consoles.
If there are additional GUI sessions, then they use F2, F3, ... (Should not be case with you, but when X11 goes haywire ...)

The another way to get to virtual console is to boot to multi-user.target (formerly known as "runlevel 3").
In GRUB menu choose "edit entry" and add to kernel command-line parameters: systemd.unit=multi-user.target

Although, if you just want to read the logs from previous (failed) boot, then adding single is easier to type.
IIRC, the "runlevel 1" does not even write more to logs, so they are all from the previous boot.

[ib.hojme]

Thanks,

I'll try it next time it happens

Yours sincerely

Ib

[ib.hojme]

TrevorH » 2021/09/14 12:41:45

I have looked through the /var/crash/127.0.0*/vmcore*.txt and it looks as I also have an issue with the nouveau driver.
Could the 2 be related ? I have attached vmcore-dmesg.txt.gz

Yours sincerely

Ib Højme

[TrevorH]

That vmcore dmesg is from the kernel you say works - 3.10.0-1160.31.1 and though it contains crashes that seem to come from nouveau, they did not start until 17760 seconds after the boot, close to 5 hours. It also shows a lot of AER errors on an Intel 8086:a114. That device id seems to be

a114 100 Series/C230 Series Chipset Family PCI Express Root Port #5

A quick google on that error message seems to say that it's a power management problem and you can "fix" it by appending pcie_aspm=off to the kernel command line. Whether that error has anything to do with your current problem is another question.

Since that's the wrong dmesg output, you need to go back to the logs and read those. Try /var/log/messages and/or /var/log/secure.

Are your keyboard LEDs flashing when you get the problem? Do you have the parameters 'rhgb' and/or 'quiet' or 'nomodeset' specified on your current kernel command line? If so remove them so you can better see any error messages at boot time. They may be hiding whatever error you are getting.