Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x86_64

Issues related to applications and software problems
Post Reply
wumingzi
Posts: 3
Joined: 2014/03/27 20:39:13

Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x86_64

Post by wumingzi » 2014/03/28 18:21:35

Run environment.

Host server:

HP Compaq 8200 Elite Convertible Minitower PC
Intel Core i7-2600 CPU @ 3.40 GHz.
16.0 GB RAM
Windows 7 Enterprise N Operating system

Oracle VirtualBox 4.3.10 r93012
VM Instance has 4096MB RAM, 80 GB on a single partition.

CentOS 6.5 has been installed and running flawlessly for 4 months.
Last working kernel was kernel-2.6.32-431.5.1.el6.x86_64
Last yum update installed kernel-2.6.32-431.11.2.el6.x86_64

Following yum update, the following message appeared after boot:

Code: Select all

Kernel panic — not syncing: VFS: Unable to mount root fs on unknown—block(0,0)
Pid: 1, comm: swapper Not tainted 2.6.32-431.11.2.el6.x86_64 #1
Call Trace:
[ <ffffffff81527823> ] ? panic+0xa7/0x16f
[ <ffffffff81c27432> ] ? mount_block_root+0x2160x2cb
[ <ffffffff81002930> ] ? bstat+0x2b0/0x980
[ <ffffffff81c2753d> ] ? mount_root+0x56/0x5a
[ <ffffffff81c276b1> ] ? prepare_namespace+0x170/0x1a9
[ <ffffffff81c2692a> ] ? kernel_init+0x2e1/0x2f7
[ <ffffffff8100c20a> ] ? child_rip+0xa/0x20
[ <ffffffff81c26649> ] ? kernel_init+0x0/0x2f7
[ <ffffffff8100c200> ] ? child_rip+0x0/0x2O
(A screen shot is attached in case my fingers and/or OCR are in error)

Other posts on the subject of similar kernel panics have usually attributed this to a glitch in the install process. People have rolled back to the previous kernel, reinstalled the new kernel, and life is good.

I have successfully rolled back the previous kernel, so I am able to boot the CentOS instance. Unfortunately, I have run 'yum update' to reinstall the kernel about 3 times now, and post-reboot we always wind up with a similar crash screen.

Does anyone have any ideas as to how to work around this without avoiding 'yum update' for the next six months?
Attachments
Screen shot at crash
Screen shot at crash
CentOS-6.png (8.16 KiB) Viewed 29319 times
Last edited by wumingzi on 2014/03/28 22:29:06, edited 1 time in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x8

Post by TrevorH » 2014/03/28 19:39:14

Do you have space in the filesystem that contains /boot?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x8

Post by toracat » 2014/03/28 20:38:17

If the space turns out to be a non-issue, please post the content of /boot/grub/grub.conf .
CentOS Forum FAQ

wumingzi
Posts: 3
Joined: 2014/03/27 20:39:13

Re: Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x8

Post by wumingzi » 2014/03/28 22:21:28

Space doesn't seem to be an issue:

[loren@ipsum.dolor ~]$ df -h

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_loren-ipsumcentos-lv_root
                       50G  3.6G   44G   8% /
tmpfs                 1.9G  228K  1.9G   1% /dev/shm
/dev/mapper/vg_loren-ipsumcentos-lv_home
                       26G  3.6G   21G  16% /home
/dev/sda1             485M   75M  385M  17% /boot
[loren@ipsum.dolor ~]$ sudo cat /boot/grub/grub.conf

Code: Select all

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/loren-ipsumcentos-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-431.11.2.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-431.11.2.el6.x86_64 ro root=/dev/mapper/vg_loren-ipsumcentos-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=vg_loren-ipsumcentos/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_loren-ipsumcentos/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
title CentOS (2.6.32-431.5.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-431.5.1.el6.x86_64 ro root=/dev/mapper/loren-ipsumcentos-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=vg_loren-ipsumcentos/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_loren-ipsumcentos/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
	initrd /initramfs-2.6.32-431.5.1.el6.x86_64.img
title CentOS (2.6.32-431.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-431.el6.x86_64 ro root=/dev/mapper/vg_loren-ipsumcentos-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=vg_loren-ipsumcentos/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_loren-ipsumcentos/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
	initrd /initramfs-2.6.32-431.el6.x86_64.img

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x8

Post by toracat » 2014/03/28 23:30:26

The problem is that there is no initrd line for the latest kernel. This is an issue known to happen in certain but rather rare cases. The cause is not known. Usually reinstalling the kernel fixes it, but in your case it apparently did not work. In that case, you need to build the initramfs file manually. If you need assistance for this, do let us know.
CentOS Forum FAQ

Whoever
Posts: 1357
Joined: 2013/09/06 03:12:10

Re: Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x8

Post by Whoever » 2014/03/29 02:39:33

toracat wrote:The problem is that there is no initrd line for the latest kernel. This is an issue known to happen in certain but rather rare cases. The cause is not known. Usually reinstalling the kernel fixes it, but in your case it apparently did not work. In that case, you need to build the initramfs file manually. If you need assistance for this, do let us know.
I don't think that it is rare. I have seen it on several systems on our small network. Booting into an older kernel and re-installing the latest version fixed it in every case that I have seen.

User avatar
toracat
Site Admin
Posts: 7518
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: Kernel panic on boot CentOS 6.2 w/2.6.32-431.11.2.el6.x8

Post by toracat » 2014/03/29 18:21:40

Whoever wrote: I don't think that it is rare. I have seen it on several systems on our small network. Booting into an older kernel and re-installing the latest version fixed it in every case that I have seen.
Well, the term "rare" is a relative thing. :) It seems to happen to certain systems more often than others. At any rate I've added this thread to the bug report as another case.
CentOS Forum FAQ

wumingzi
Posts: 3
Joined: 2014/03/27 20:39:13

RESOLVED: Kernel panic on boot CentOS 6.2

Post by wumingzi » 2014/03/31 18:27:38

Found the problem. It's subtle, and applies only to this server.

Following the instructions to build a new initramfs, I ran:

dracut -f /boot/initramfs-2.6.32-431.11.2.el6.x86_64.img 2.6.32-431.11.2.el6.x86_64

This produced the following error:

E: Failed to install /lib/libnss_files*

/lib/libnss_files* does not exist. /lib64/libnss_files* does. That's what you'd expect on a 64-bit install.

So I thought "Hmmm. That's strange. It's a 64 bit environment. Why would it be looking for 32-bit libraries?"

I heard Obi-Wan Kenobi's voice calling out. "Use the source, Luke!"

This produced the following answers:

In /usr/share/dracut/modules.d/95udev-rules/install, the following code snippet determines which path to take to find libnss_files

Code: Select all

if ldd $(find_binary udevd) 2>/dev/null |grep -q /lib64/; then
    dracut_install /lib64/libnss_files*
else
    dracut_install /lib/libnss_files*
fi
The find_binary function walks the ${PATH} environment variable to find the udevd binary. It's the same code that powers the 'which' command. That's not interesting.

ldd on my system maps to /usr/bin/ldd, which the files command identifies as:

/usr/bin/ldd: a /usr/bin/bash script text executable

CentOS doesn't have a /usr/bin/bash executable. bash lives in /bin, as it should be.

Any calls to ldd will end in failure, which will not contain the /lib64 directory.

Ergo, the logic in dracut tells it to search the /lib directory for nss_files.

The ldd command calls /usr/bin/bash because of a previous install of an FC21 glibc-common library.

The fix was:

ln -s /bin/bash /usr/bin

dracut works. Initrd exists, grub is fixed, and the kernel boots.

Not sure how much protective code is needed in dracut to save people from hacking up their core OS.

Post Reply