kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

General support questions
mayankdhw2020
Posts: 12
Joined: 2020/05/27 09:44:12

kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by mayankdhw2020 » 2020/05/27 10:20:27

Hi, I am facing issue when upgrading centos server via yum from kernel Linux 3.10.0-1062.4.1.el7.x86_64 to kernel Linux 3.10.0-1127.8.2.el7.x86_64. Server is hosted on Azure platform and it did not show anything under boot diagnostic and not able to connect to serial console. After stopping and then starting the server again from Azure, it shows that "Kernel Panic - not syncing: Fatal exception" . I am attaching the screenshots taken from Azure console for the server which shows that the server got struck at this point.
sftp-screenshot.JPG
sftp-screenshot.JPG (40.11 KiB) Viewed 4348 times
sftp-prod-vm01-screenshot.JPG
sftp-prod-vm01-screenshot.JPG (99.99 KiB) Viewed 4348 times

mayankdhw2020
Posts: 12
Joined: 2020/05/27 09:44:12

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by mayankdhw2020 » 2020/05/27 17:31:31

Please note that I am getting the same issue on all centos servers running with this kernel. Please help if anyone has fix for this issue ! Thanks in advance.

ModusHack
Posts: 3
Joined: 2020/05/27 20:32:11

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by ModusHack » 2020/05/27 20:38:48

As am I, on a number of VM installs.

tszpak
Posts: 1
Joined: 2016/10/24 16:12:57

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by tszpak » 2020/05/27 21:19:08

Hello.

I faced the same problem. I found out the machine started with no grub config file, which disappeared. The machine is booted from efi. The standard grub.cfg in /boot/grub2/grub.cfg seems to be ok, but in fact it’s not used in my configuration.

After grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg the machine started.


Tomasz

mayankdhw2020
Posts: 12
Joined: 2020/05/27 09:44:12

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by mayankdhw2020 » 2020/05/28 10:57:20

This doesn't work for me. I have tried creating grub2 but the same issue persists again.

ModusHack
Posts: 3
Joined: 2020/05/27 20:32:11

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by ModusHack » 2020/05/28 21:10:09

I had to catch the boot in time (the software we use for VM management is just fast enough). I selected another kernel (the previous one, 3.10.0-1127.el7.x86_64). Once I did that, the system booted normally. Then I removed the offending kernel from grub.cfg so that end user reboots won't fail.

mayankdhw2020
Posts: 12
Joined: 2020/05/27 09:44:12

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by mayankdhw2020 » 2020/05/29 14:39:50

But in my case after upgrading to this kernel 3.10.0-1127.el7.x86_64 server isn't coming up. As it is hosted on Azure, so I am not able to select previous kernel to boot from.

ModusHack
Posts: 3
Joined: 2020/05/27 20:32:11

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by ModusHack » 2020/05/29 21:31:49

That's the trouble. If you can't get to the grub menu, I don't know what you can do. Sorry, and good luck.

mayankdhw2020
Posts: 12
Joined: 2020/05/27 09:44:12

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by mayankdhw2020 » 2020/06/04 17:22:32

Can anyone tell me if there's a bug in kernel 3.10.0-1127 or this might be an issue with Azure hosted CentOS servers ! :roll:

sysengrnz
Posts: 2
Joined: 2020/06/10 04:12:11

Re: kernel panic on 3.10.0-1127.8.2.el7.x86_64 systems

Post by sysengrnz » 2020/06/10 04:38:53

In addition to my previous post, we have two centos 7 servers that have updated fine and booting ok with the latest kernel.

Now the only real difference I can see here is that on a system that has booted fine, /etc/centos-release only shows the latest update. Doesn't show multiple lines for previous version and updated version. The systems that have booted don't use Logical/physical volume groups. Just a single Premium SSD Disk with SSE/PMK encryption and an single data disk attached.

The system that wont boot has as single SSD Premium OS Disk, and then has 13 1TB data drives attached (Standard Data Disks). The 13 1TB drives are part of a logical volume group using LVM2 format.

The panic seemed to suggest possible kernel/disk/driver fault? Not sure if its relevant to the investigation although seems like the new kernel couldn't find the operating system disk? A few lines preceding the panic and then hopefully caught the key event that would suggest where it faulted?

[ 1.368247] rtc_cmos 00:00: setting system clock to 2020-06-10 02:36:45 UTC (1591756605)
[ 1.376545] Waiting 300sec before mounting root device...
[ 1.443479] random: fast init done
[ 1.448182] psmouse serio1: alps: Unknown ALPS touchpad: E7=12 00 64, EC=12 00 64
[ 1.657202] psmouse serio1: trackpoint: failed to get extended button data, assuming 3 buttons
[ 5.264153] psmouse serio1: trackpoint: IBM TrackPoint firmware: 0x01, buttons: 3/3
[ 5.271603] input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/input/input2
[ 98.091788] random: crng init done
[ 240.533731] INFO: task swapper/0:1 blocked for more than 120 seconds.
[ 240.540143] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 240.546140] swapper/0 D ffff93d1dd79acc0 0 1 0 0x00000000
[ 240.553806] Call Trace:
[ 240.556820] [<ffffffffa809e5b5>] ? console_unlock+0x265/0x4b0
[ 240.564212] [<ffffffffa8785da9>] schedule+0x29/0x70
[ 240.568113] [<ffffffffa87837f8>] schedule_timeout+0x168/0x2d0
[ 240.572769] [<ffffffffa80ae150>] ? __internal_add_timer+0x130/0x130
[ 240.578485] [<ffffffffa87796d8>] ? printk+0x60/0x77
[ 240.582158] [<ffffffffa80af0bf>] msleep+0x2f/0x40
[ 240.586350] [<ffffffffa8d8b825>] prepare_namespace+0x2c/0x174
[ 240.591079] [<ffffffffa8d8b412>] kernel_init_freeable+0x222/0x249
[ 240.596220] [<ffffffffa8d8ab28>] ? initcall_blacklist+0xb0/0xb0
[ 240.601531] [<ffffffffa876e6b0>] ? rest_init+0x80/0x80
[ 240.606627] [<ffffffffa876e6be>] kernel_init+0xe/0x100
[ 240.611307] [<ffffffffa8792d37>] ret_from_fork_nospec_begin+0x21/0x21
[ 240.616398] [<ffffffffa876e6b0>] ? rest_init+0x80/0x80
[ 301.382111] md: Waiting for all devices to be available before autodetect
[ 301.387590] md: If you don't use raid, use raid=noautodetect
[ 301.393255] md: Autodetecting RAID arrays.
[ 301.397441] md: autorun ...
[ 301.402032] md: ... autorun DONE.
[ 301.405140] List of all partitions:
[ 301.408683] No filesystem could mount root, tried:
[ 301.413477] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 301.414470] CPU: 6 PID: 1 Comm: swapper/0 Tainted: G ------------ T 3.10.0-1127.10.1.el7.x86_64 #1
[ 301.414470] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007 06/02/2017
[ 301.414470] Call Trace:
[ 301.414470] [<ffffffffa877ffa5>] dump_stack+0x19/0x1b
[ 301.414470] [<ffffffffa8779541>] panic+0xe8/0x21f
[ 301.414470] [<ffffffffa8d8b794>] mount_block_root+0x291/0x2a0
[ 301.414470] [<ffffffffa8d8b7f6>] mount_root+0x53/0x56
[ 301.414470] [<ffffffffa8d8b935>] prepare_namespace+0x13c/0x174
[ 301.414470] [<ffffffffa8d8b412>] kernel_init_freeable+0x222/0x249
[ 301.414470] [<ffffffffa8d8ab28>] ? initcall_blacklist+0xb0/0xb0
[ 301.414470] [<ffffffffa876e6b0>] ? rest_init+0x80/0x80
[ 301.414470] [<ffffffffa876e6be>] kernel_init+0xe/0x100
[ 301.414470] [<ffffffffa8792d37>] ret_from_fork_nospec_begin+0x21/0x21
[ 301.414470] [<ffffffffa876e6b0>] ? rest_init+0x80/0x80
[ 301.414470] Kernel Offset: 0x27000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff

Andrew

Post Reply