Booting latest Kernel failed on multiple systems (kernel-3.10.0-1160.80.1.el7.x86_64)

[TrevorH]

That bugs entry says it only crashes when kvm is used so a database server would probably be safe so long as you're not also running VMs on it.

[silvio]

Is it good or bad, if AMD CPU from 2019 is so similar to old Intel CPU (from 2009?) that same bug affects both?

For the record, the .80 did install and boot cleanly with Intel Core i7-6700. Or so it seems. (Undefined behaviour can be sneaky.)

We had a 7700 system as a dev playground ....
The kernel is working on this cpu without a problem at the moment.

Linux version 3.10.0-1160.80.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Nov 8 15:48:59 UTC 2022

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
Stepping: 9
CPU MHz: 4164.038
CPU max MHz: 4200,0000
CPU min MHz: 800,0000
BogoMIPS: 7200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities

So thanks for the (old) Intel info.

Silvio

[silvio]

That bugs entry says it only crashes when kvm is used so a database server would probably be safe so long as you're not also running VMs on it.

I have no kvm running on any server we have ....
The systems a plain minimal installs with only the software dependencies we need for our software and virtualization is not on the list

[root@xxx ~]# rpm -qa|grep kvm
[root@xxx ~]#

[root@xxx ~]# rpm -qa|grep virt
virt-what-1.18-4.el7_9.1.x86_64

virt-what is a dependency of tuned and for this reason on every system we have.

Silvio

[jlehtone]

On my system the .80 kernel is noticeably larger file, but the initramfs is about same:

Code: Select all

$ ls -lh /boot
[jlehtone@el7 ~]# ls -lh /boot
total 263M
-rw-r--r--. 1 root root 151K Aug 10 19:25 config-3.10.0-1160.76.1.el7.x86_64
-rw-r--r--. 1 root root 151K Nov  8 17:53 config-3.10.0-1160.80.1.el7.x86_64
drwx------. 3 root root  16K Jan  1  1970 efi
drwxr-xr-x. 2 root root 4.0K Sep 24  2021 grub
drwx------. 2 root root 4.0K Jul 31 14:08 grub2
-rw-------. 1 root root  90M Nov 24 09:54 initramfs-3.10.0-1160.76.1.el7.x86_64.img
-rw-------. 1 root root  90M Nov 24 09:56 initramfs-3.10.0-1160.80.1.el7.x86_64.img
drwx------. 2 root root  16K Sep 24  2021 lost+found
-rw-r--r--. 1 root root 314K Aug 10 19:25 symvers-3.10.0-1160.76.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314K Nov  8 17:53 symvers-3.10.0-1160.80.1.el7.x86_64.gz
-rw-------. 1 root root 3.5M Aug 10 19:25 System.map-3.10.0-1160.76.1.el7.x86_64
-rw-------. 1 root root 3.5M Nov  8 17:53 System.map-3.10.0-1160.80.1.el7.x86_64
-rwxr-xr-x. 1 root root 6.5M Aug 10 19:25 vmlinuz-3.10.0-1160.76.1.el7.x86_64
-rwxr-xr-x. 1 root root 6.8M Nov  8 17:53 vmlinuz-3.10.0-1160.80.1.el7.x86_64

(Size of initramfs depends on what you do include and I have NVidia drivers ...)

[silvio]

On my system the .80 kernel is noticeably larger file, but the initramfs is about same:

Code: Select all

$ ls -lh /boot
[jlehtone@el7 ~]# ls -lh /boot
total 263M
-rw-r--r--. 1 root root 151K Aug 10 19:25 config-3.10.0-1160.76.1.el7.x86_64
-rw-r--r--. 1 root root 151K Nov  8 17:53 config-3.10.0-1160.80.1.el7.x86_64
drwx------. 3 root root  16K Jan  1  1970 efi
drwxr-xr-x. 2 root root 4.0K Sep 24  2021 grub
drwx------. 2 root root 4.0K Jul 31 14:08 grub2
-rw-------. 1 root root  90M Nov 24 09:54 initramfs-3.10.0-1160.76.1.el7.x86_64.img
-rw-------. 1 root root  90M Nov 24 09:56 initramfs-3.10.0-1160.80.1.el7.x86_64.img
drwx------. 2 root root  16K Sep 24  2021 lost+found
-rw-r--r--. 1 root root 314K Aug 10 19:25 symvers-3.10.0-1160.76.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314K Nov  8 17:53 symvers-3.10.0-1160.80.1.el7.x86_64.gz
-rw-------. 1 root root 3.5M Aug 10 19:25 System.map-3.10.0-1160.76.1.el7.x86_64
-rw-------. 1 root root 3.5M Nov  8 17:53 System.map-3.10.0-1160.80.1.el7.x86_64
-rwxr-xr-x. 1 root root 6.5M Aug 10 19:25 vmlinuz-3.10.0-1160.76.1.el7.x86_64
-rwxr-xr-x. 1 root root 6.8M Nov  8 17:53 vmlinuz-3.10.0-1160.80.1.el7.x86_64

(Size of initramfs depends on what you do include and I have NVidia drivers ...)

Same here on the working Intel system

-rw-r--r--. 1 root root 151K 5. Apr 2022 config-3.10.0-1160.62.1.el7.x86_64
-rw-r--r--. 1 root root 151K 18. Mai 2022 config-3.10.0-1160.66.1.el7.x86_64
-rw-r--r--. 1 root root 151K 28. Jun 15:41 config-3.10.0-1160.71.1.el7.x86_64
-rw-r--r--. 1 root root 151K 10. Aug 16:25 config-3.10.0-1160.76.1.el7.x86_64
-rw-r--r--. 1 root root 151K 8. Nov 15:53 config-3.10.0-1160.80.1.el7.x86_64
drwxr-xr-x. 3 root root 4,0K 3. Sep 2018 efi
drwxr-xr-x. 2 root root 4,0K 3. Sep 2018 grub
drwx------. 5 root root 4,0K 24. Nov 11:45 grub2
-rw-------. 1 root root 51M 3. Sep 2018 initramfs-0-rescue-86825fc8078e4f09a512172d8b97bbe0.img
-rw-------. 1 root root 20M 19. Aug 09:25 initramfs-3.10.0-1160.62.1.el7.x86_64.img
-rw-------. 1 root root 20M 20. Mai 2022 initramfs-3.10.0-1160.66.1.el7.x86_64.img
-rw-------. 1 root root 20M 19. Aug 09:25 initramfs-3.10.0-1160.71.1.el7.x86_64.img
-rw-------. 1 root root 20M 11. Nov 13:14 initramfs-3.10.0-1160.76.1.el7.x86_64.img
-rw-------. 1 root root 20M 24. Nov 11:45 initramfs-3.10.0-1160.80.1.el7.x86_64.img
drwx------. 2 root root 16K 3. Sep 2018 lost+found
-rw-r--r--. 1 root root 314K 5. Apr 2022 symvers-3.10.0-1160.62.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314K 18. Mai 2022 symvers-3.10.0-1160.66.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314K 28. Jun 15:42 symvers-3.10.0-1160.71.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314K 10. Aug 16:25 symvers-3.10.0-1160.76.1.el7.x86_64.gz
-rw-r--r--. 1 root root 314K 8. Nov 15:53 symvers-3.10.0-1160.80.1.el7.x86_64.gz
-rw-------. 1 root root 3,5M 5. Apr 2022 System.map-3.10.0-1160.62.1.el7.x86_64
-rw-------. 1 root root 3,5M 18. Mai 2022 System.map-3.10.0-1160.66.1.el7.x86_64
-rw-------. 1 root root 3,5M 28. Jun 15:41 System.map-3.10.0-1160.71.1.el7.x86_64
-rw-------. 1 root root 3,5M 10. Aug 16:25 System.map-3.10.0-1160.76.1.el7.x86_64
-rw-------. 1 root root 3,5M 8. Nov 15:53 System.map-3.10.0-1160.80.1.el7.x86_64
-rwxr-xr-x. 1 root root 6,0M 3. Sep 2018 vmlinuz-0-rescue-86825fc8078e4f09a512172d8b97bbe0
-rwxr-xr-x. 1 root root 6,5M 5. Apr 2022 vmlinuz-3.10.0-1160.62.1.el7.x86_64
-rwxr-xr-x. 1 root root 6,5M 18. Mai 2022 vmlinuz-3.10.0-1160.66.1.el7.x86_64
-rwxr-xr-x. 1 root root 6,5M 28. Jun 15:42 vmlinuz-3.10.0-1160.71.1.el7.x86_64
-rwxr-xr-x. 1 root root 6,5M 10. Aug 16:25 vmlinuz-3.10.0-1160.76.1.el7.x86_64
-rwxr-xr-x. 1 root root 6,8M 8. Nov 15:53 vmlinuz-3.10.0-1160.80.1.el7.x86_64

[silvio]

If you look into

rpm -q --changelog kernel-3.10.0-1160.80.1.el7.x86_64

it is quit a lot output.

Silvio

[jlehtone]

Yes, most of which are about security:

Code: Select all

$ rpm -q --changelog kernel-3.10.0-1160.80.1.el7.x86_64 | sed "/3.10.0-1160.76.1.el7/q" | grep -c "^-"
52
$ rpm -q --changelog kernel-3.10.0-1160.80.1.el7.x86_64 | sed "/3.10.0-1160.76.1.el7/q" | grep "^-" | grep -c CVE
40

Could something new in the security and the OpenScap together be the explanation for the early boot fail?

[silvio]

My Bugreport in the RedHat Bugtracker was marked as a duplicate to bug https://bugzilla.redhat.com/show_bug.cgi?id=2122158 .
This bug shows problems with SME on AMD systems with kdump enabled and RedHat marked SME as tech preview in 7.9.

The strange thing for me is, we have this enabled since installation and we have also kdump disabled :
The 3.10.0-1160.76.1.el7.x86_64 is running at the moment with mem_encryption=on without any problems:

Linux version 3.10.0-1160.76.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Aug 10 16:21:17 UTC 2022
[ 0.000000] AMD Secure Memory Encryption (SME) active

[silvio]

Today I installed the new 3.10.0-1160.81.1.el7.x86_64 kernel and checked it.
Same result, machine does not start ...

If i disable "mem_encrypt=on" , the system starts normal .

So it looks to me like it the "new" mem_encrypt is the show stopper.

Silvio

[TrevorH]

There was a new kernel released for RHEL 7 today that says it fixes an issue booting on Xeon e55xx chips. That kernel has yet to be built for CentOS 7 yet but look out for -1160.88.1.el7.