[WORKAROUND] Kernel Panic booting CentOS 7, Intel J2900 CPU

[thearcher]

Howdy,

I've been running Fedora for years, but I want to switch to CentOS. I have a new system with an ASRock Q2900-ITX motherboard (Celeron J2900 CPU). During boot, there's a kernel panic, and the system reboots.

I've tried booting with acpi=off, and with noapic. I've changed the SATA controller from AHCI to IDE.

A few lines before/during the kernel panic are below. I recorded the video, so typed this in by hand.

Code: Select all

[207.2] rtc_cmos 00:00: alarms up to one month. y3k. 242 bytes nvram. hpet irqs
[  207.4] Intel P-state driver initializing
[  207.6] Intel pstate controlling: cpu 0
[  207.8] Intel pstate controlling: cpu 1
[  208.0] Intel pstate controlling: cpu 2
[  208.2] BUG: unable to handle kernel [  208.2] Intel pstate controlling: cpu 3
[  208.4] cpuidle: using governor menu
[  208.4] BUG: unable to handle kernel [  208.6] hidraw: raw HID event driver (C) Jiri Kosina
[  208.6] NULL pointer dereference[  208.8] usbcore: registered new interface driver usbhid
 at          (null)
[  209.0] usbhid: USB HID core driver
[  209.0] IP:[  209.2] drop_monitor: Initializing network drop monitor service
 [<            (null)>]         (null)
[  209.4] TCP: cubic registered
[  209.4]PGD 0 [  209.6] Initializing XFRM netlink socket

[  209.8] NET: Registered protocol family 10
[  209.8] Oops: 0010 [#1] [  210.0] NET: Registered protocol family 17 SMP
[  210.2] Loading compiled-in X.509 certificates

[  210.4] Loaded X.509 cert 'CentOS Linux kpatch signing key: <key>'
[  210.4] Modules linked in:[  210.6] Loaded X.509 cert 'CentOS Linux Driver update signing key: <key>'

[  210.8] Loaded X.509 cert 'CentOS Linux kernel signing key: <key>'
[  210.8] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.10.0-123.el7.x86_64 #1
[  211.0] registered taskstats version 1
[  211.0] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Q2900-ITX. BIOS P1.10 10/31/2014
[  211.2] Key type trusted registered
[  211.2] task: ffff88023447e660 ti: ffff8802344b4000 task.ti: ffff8802344b4000
[  211.4] Key type encrypted registered
[  211.4] RIP: 0010:[<0000000000000000>] [  211.6] IMA: No TPM chip found. activating TPM-bypass!
[  211.8] RSP: 0000:ffff88023fd86d70  EFLAGS: 00010046
[  212.0] RAX: ffffffff818e4fe0 RBX: ffff88023fd86ef8 RCX: 000000000000c6d0
...
[  217.3] Call Trace:
[  217.5]  <NMI>
[  217.7]  [<ffffffff81029344>] ? intel_pmu_handle_irq+0x274/0x3f0
[  218.3]  [<ffffffff8101a603>] ? native_sched_clock+0x13/0x80
[  218.7]  [<ffffffff8101a679>] ? sched_clock+0x9/0x10
...

Fedoras 19 and 20 boot. I installed 20, and got this info:

Code: Select all

== BEGIN uname -rmi ==
3.11.10-301.fc20.x86_64 x86_64 x86_64
== END   uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
fedora-release-notes-20-0.5.noarch
fedora-release-20-1.noarch
== END   rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
Fedora release 20 (Heisenbug)
== END   cat /etc/redhat-release ==

== BEGIN getenforce ==
Enforcing
== END   getenforce ==

== BEGIN free -m ==
             total       used       free     shared    buffers     cached
Mem:          7692       2078       5614         56        127       1374
-/+ buffers/cache:        576       7116
Swap:         7823          0       7823
== END   free -m ==

== BEGIN rpm -qa kernel\* | sort ==
kernel-3.11.10-301.fc20.x86_64
kernel-devel-3.11.10-301.fc20.x86_64
kernel-headers-3.11.10-301.fc20.x86_64
== END   rpm -qa kernel\* | sort ==

== BEGIN lspci -nn ==
00:00.0 Host bridge [0600]: Intel Corporation ValleyView SSA-CUnit [8086:0f00] (rev 0e)
00:02.0 VGA compatible controller [0300]: Intel Corporation ValleyView Gen7 [8086:0f31] (rev 0e)
00:13.0 IDE interface [0101]: Intel Corporation ValleyView 4-Port SATA Storage Controller [8086:0f21] (rev 0e)
00:14.0 USB controller [0c03]: Intel Corporation ValleyView USB xHCI Host Controller [8086:0f35] (rev 0e)
00:1a.0 Encryption controller [1080]: Intel Corporation ValleyView SEC [8086:0f18] (rev 0e)
00:1b.0 Audio device [0403]: Intel Corporation ValleyView High Definition Audio Controller [8086:0f04] (rev 0e)
00:1c.0 PCI bridge [0604]: Intel Corporation ValleyView PCI Express Root Port [8086:0f48] (rev 0e)
00:1c.1 PCI bridge [0604]: Intel Corporation ValleyView PCI Express Root Port [8086:0f4a] (rev 0e)
00:1c.2 PCI bridge [0604]: Intel Corporation ValleyView PCI Express Root Port [8086:0f4c] (rev 0e)
00:1c.3 PCI bridge [0604]: Intel Corporation ValleyView PCI Express Root Port [8086:0f4e] (rev 0e)
00:1f.0 ISA bridge [0601]: Intel Corporation ValleyView Power Control Unit [8086:0f1c] (rev 0e)
00:1f.3 SMBus [0c05]: Intel Corporation ValleyView SMBus Controller [8086:0f12] (rev 0e)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 11)
04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
== END   lspci -nn ==

== BEGIN lsusb ==
Bus 002 Device 002: ID 174c:3074 ASMedia Technology Inc. 
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 007: ID 046e:6000 Behavior Tech. Computer Corp. 
Bus 001 Device 006: ID 0557:8021 ATEN International Co., Ltd CS1764A [CubiQ DVI KVMP Switch]
Bus 001 Device 004: ID 0409:005a NEC Corp. HighSpeed Hub
Bus 001 Device 003: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 001 Device 005: ID 125f:c08a A-DATA Technology Co., Ltd. C008 Flash Drive
Bus 001 Device 002: ID 174c:2074 ASMedia Technology Inc. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
== END   lsusb ==

== BEGIN rpm -qa kmod\* kmdl\* ==
kmod-15-1.fc20.x86_64
kmod-libs-15-1.fc20.x86_64
== END   rpm -qa kmod\* kmdl\* ==

Is there some other kernel flag I should try? Can I install a newer kernel on the boot disk, or otherwise boot from a newer kernel and then start the installer?

Thanks!

(Edit: realized I should put the CPU in the title.)

[TrevorH]

Might be worth reading https://community.oracle.com/thread/356 ... 0&tstart=0

[thearcher]

I tried Trevor's suggestion of adding the boot passwords. It didn't help though.

I placed the drive in a different Intel-based PC, and installed CentOS 7 there. Then I installed all updates, including kernel 3.10.0-123.9.3. Moving the drive back to the J2900 box, the reboot/kernel panic still happens once the Intel p-state driver starts initializing. If I have time, I'll try to learn enough git to find the differences between the Fedora 19/20 kernel's p-state driver and CentOS 7's kernel p-state driver. Otherwise, I'll just install updates monthly(?) until the necessary Fedora changes make it to CentOS.

One other thing I should research: whether the microcode for the J2900 CPU is installed.

[thearcher]

More evidence that it is the Intel p-state driver. Adding the following kernel parameter allows the system to boot:

Code: Select all

intel_pstate=disable

Using the "no_hwp" value isn't enough to prevent the crash.

I'm going to run with this, keeping an eye on the temperature, until the kernel doesn't have this issue.

UPDATE

While the system boots, it is not stable. Still researching.


UPDATE

This system has an integrated Intel graphics device. These devices require that the kernel use mode setting. I had added "nomodeset" to make it easier to read the boot messages. Removing the nomodeset parameter resolved the instability.