CentOS 8, Failed to switch root, '/sysroot' Empty

Issues related to applications and software problems and general support
joseluisbz
Posts: 14
Joined: 2020/06/25 18:24:42

CentOS 8, Failed to switch root, '/sysroot' Empty

Post by joseluisbz » 2020/09/01 07:01:09

Today, I was starting my CentOS and I got Image

I was reading https://bugzilla.redhat.com/show_bug.cgi?id=1492208
https://askbot.fedoraproject.org/en/que ... n-os-tree/
https://www.reddit.com/r/linuxquestions ... t_on_boot/

But, My

Code: Select all

/sysroot
directory is empty, I don't know where get some copy.

Image
In the image, the only entry working is Rescue The CentOS entry that is above the Windows entry

Code: Select all

CentOS Linux (0-rescue-***)
Thanks for help!

mfabbri
Posts: 4
Joined: 2018/01/08 16:02:30

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by mfabbri » 2020/09/08 11:50:47

I have the same issue too, but I don't now how to fix it, I hope that someone could help.

Sondergaard
Posts: 4
Joined: 2020/11/03 01:39:41

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by Sondergaard » 2020/11/03 02:39:17

This failure happened to me upon updating from CentOS 8 kernel 4.18.0-193.6.3.el8_2.x86_64 to 4.18.0-193.28.1.el8_2.x86_64 running on a Mac Mini (late 2014). During the same dnf update process, a grub2 update was involved as well.

Well, this update bricked my bootloader (blank screen on reboot, not even grub). It was necessary to start my system from a rescue disk. I flailed around with grub2-mkconfig, grub2-install, dracut, etc. for hours (Linux system management is not my day job). Finally I got the system booting at least partway, but now I had the exact problem described by the Original Poster.

So, here is what I was able to determine, and a workaround that solved the problem for me.

1. /sysroot appears empty because it is not being mounted. The boot process is supposed to mount the main system root tree at the /sysroot directory located in the boot-time RAM disk. Once this is done, the boot process can get at the os-release file that the Switch Root process wants. However, if the boot process does not mount the main system root on /sysroot, /sysroot will appear as an empty directory.

2. Confirming this, I was able to start my system manually from the "Failed to start Switch Root" emergency mode by forcibly mounting my system root tree on /sysroot. I did this with

Code: Select all

# mount -t ext4 --source /dev/sda3 --target /sysroot
and then logging out of emergency mode with control-D. Of course you might have to use something else instead of /dev/sda3 depending on what partition has your system root tree, and if you have SELinux enabled and it has to reindex, you may have to go through this twice. But it worked! Now the problem was that I'd have to do this every time I rebooted the system.

3. So the question is, why is the boot process not mounting the main system root on /sysroot? There are no indications that the mount is being attempted and failing; it's as if the boot process isn't even trying.

4. When grub2-mkconfig creates the grub.cfg file (located in /boot/grub2/grub.cfg for non-UEFI systems) it knows very well where the kernel should look for the system root, so it places that information in the grub file to be passed to the kernel in the latter's command line. For example, in my grub.cfg file is a line that says...

Code: Select all

search --no-floppy --fs-uuid --set=root 67618b2c-9abe-49f8-bfb0-70398243e7c1
...which I believe is supposed to tell grub to pass this UUID along to the kernel command line so the kernel knows where to look for the main system root.

5. However, looking at /proc/cmdline in the emergency mode shell indicated that grub was NOT passing the system root information to the kernel on startup. That could explain the problem. But if so, why wasn't it?

6. Recent changes to grub moved the kernel specification out of grub.cfg and into a collection of text files under /boot/loader/entries. For example, my /boot/loader/entries directory looks like this:

Code: Select all

-rw-r--r--. 1 root root 382 Oct 31 14:05 87cabdae4e4c494983f30bd67668d0c1-0-rescue.conf
-rw-r--r--. 1 root root 373 Nov  1 19:49 87cabdae4e4c494983f30bd67668d0c1-4.18.0-193.28.1.el8_2.x86_64.conf
-rw-r--r--. 1 root root 353 Oct 31 14:07 87cabdae4e4c494983f30bd67668d0c1-4.18.0-193.6.3.el8_2.x86_64.conf
-rw-r--r--. 1 root root 323 Oct 31 14:07 87cabdae4e4c494983f30bd67668d0c1-4.18.0-193.el8.x86_64.conf
The default contents of the second file (for the most recent kernel) started out looking like this:

Code: Select all

title CentOS Linux (4.18.0-193.28.1.el8_2.x86_64) 8 (Core)
version 4.18.0-193.28.1.el8_2.x86_64
linux /vmlinuz-4.18.0-193.28.1.el8_2.x86_64
initrd /initramfs-4.18.0-193.28.1.el8_2.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id centos-20201022002823-4.18.0-193.28.1.el8_2.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
7. So I wondered ... what if something is going wrong with getting the system root tree ID passed to the kernel under this new /boot/loader/entries bootloading approach? So...

8. ...I manually added my root-tree specification to the "linux /vmlinz..." line in my /boot/loader/entries/87cabdae4e4c494983f30bd67668d0c1-4.18.0-193.28.1.el8_2.x86_64.conf as follows. Note that it's just "root=/dev/sda3" appended to the third line in the file; everything else is the same:

Code: Select all

title CentOS Linux (4.18.0-193.28.1.el8_2.x86_64) 8 (Core)
version 4.18.0-193.28.1.el8_2.x86_64
linux /vmlinuz-4.18.0-193.28.1.el8_2.x86_64 root=/dev/sda3
initrd /initramfs-4.18.0-193.28.1.el8_2.x86_64.img $tuned_initrd
options $kernelopts $tuned_params
id centos-20201022002823-4.18.0-193.28.1.el8_2.x86_64
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel
9. Ta-da, this worked! My system now boots properly without manual intervention.

I have no idea if this is due to a bug in the new grub system or a misconfiguration on my part. I'm guessing this will fail next time I do a kernel update. But for now it appears to be a successful workaround.

User avatar
TrevorH
Forum Moderator
Posts: 30358
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by TrevorH » 2020/11/03 03:38:52

That info should be coming from the $kernelopts variable that's on the options line. And that info is kept in the grubenv file which is either /boot/grub2/grubenv or some path under /boot/efi (don't use UEFI so don't know). Running grub2-editenv list should show it to you and it should look something like

Code: Select all

[root@centos8 ~]# grub2-editenv list 
saved_entry=2644ace6d2294695a6b4bd6aa4be174f-4.18.0-193.28.1.el8_2.x86_64
kernelopts=root=/dev/mapper/cl_centos8-root ro resume=/dev/mapper/cl_centos8-swap rd.lvm.lv=cl_centos8/root rd.lvm.lv=cl_centos8/swap crashkernel=0@0 
boot_success=1
boot_indeterminate=0
Did you run grub2-mkconfig -o /boot/grub2/grub.cfg (or its UEFI equivalent)? I suspect that would probably set that variable if it doesn't exist. Also that grubenv file seems excessively fragile and has to be exactly 1024 bytes in size and is padded with # symbols to fill up the rest of the space. It might be worth moving it elsewhere and seeing if grub2-mkconfig will recreate it if running it the first time did not fix it.
CentOS 6 died in November 2020 - migrate to a new version!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

Sondergaard
Posts: 4
Joined: 2020/11/03 01:39:41

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by Sondergaard » 2020/11/03 18:45:45

Thanks TrevorH. I did indeed run grub2-mkconfig -o /boot/grub2/grub.cfg (several times in fact, as I flailed around trying to unbrick the bootloader after the update). I didn't put any energy into the grubenv file, sounds like that was a good thing if it is so fragile. But if I run grub2-editenv list now, I get the following:

Code: Select all

saved_entry=87cabdae4e4c494983f30bd67668d0c1-4.18.0-193.28.1.el8_2.x86_64
boot_success=1
boot_indeterminate=5
kernelopts=root=UUID=67618b2c-9abe-49f8-bfb0-70398243e7c1 ro
The UUID for root matches that of my /dev/sda3. So it would appear the bootloader's failure to know where my root system tree is located is not due to a misconfigured grubenv file.

Seems pretty significant that the problem is solved by adding the simple "root=/dev/sda3" in the relevant kernel boot config under /boot/loader/entries. Otherwise the root tree identification is somehow not passed to the kernel, though it's supposed to be.

Also seems significant that this happened after the new, major update to grub2 in which it relies on these new /boot/loader/entries files for kernel specifications and not on the grub2.cfg file...?

ejjl
Posts: 3
Joined: 2020/11/09 07:49:37

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by ejjl » 2020/11/09 08:02:58

Thank you very much, Sondergaard, for your instructions. They helped me boot my system again.

I followed TervorH's suggestion and checked the grubenv, and root UUID was correct. Upon reboot I ran into the same issue.

I applied Sondergaard's workaround by editing the loader entry for the current kernel, but I assume that this is not a lasting solution.

Has a bug been filed for this?

ejjl
Posts: 3
Joined: 2020/11/09 07:49:37

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by ejjl » 2020/11/10 07:52:17

Thank you, Sondergaard, for the steps to work around this. They were the only thing that worked (my /sysroot was also empty, despite grub2 env having the correct UUID for the root partition). I also fear that a kernel update will break this again, and wonder if a bug report needs to be filed?

*****

I have had the same issue when I restarted after the latest update, which updated the following packages:

Code: Select all

dnf history info 78
Transaction ID : 78
Begin time     : Thu 05 Nov 2020 02:18:18 PM CET
Begin rpmdb    : 692:6b07281fbfbf67fc73501c8b332485f8a9887e48
End time       : Thu 05 Nov 2020 02:20:54 PM CET (156 seconds)
End rpmdb      : 692:8333655127ec3c8c5c959b5c8788def3c2d1c85c
User           :  <-redacted->
Return-Code    : Success
Releasever     : 8
Command Line   : update
Packages Altered:
    Install  kernel-4.18.0-193.28.1.el8_2.x86_64                                @BaseOS
    Install  kernel-core-4.18.0-193.28.1.el8_2.x86_64                           @BaseOS
    Install  kernel-modules-4.18.0-193.28.1.el8_2.x86_64                        @BaseOS
    Upgrade  qemu-guest-agent-15:2.12.0-99.module_el8.2.0+524+f765f7e0.4.x86_64 @AppStream
    Upgraded qemu-guest-agent-15:2.12.0-99.module_el8.2.0+385+c644c6e8.2.x86_64 @@System
    Upgrade  kernel-tools-4.18.0-193.28.1.el8_2.x86_64                          @BaseOS
    Upgraded kernel-tools-4.18.0-193.19.1.el8_2.x86_64                          @@System
    Upgrade  kernel-tools-libs-4.18.0-193.28.1.el8_2.x86_64                     @BaseOS
    Upgraded kernel-tools-libs-4.18.0-193.19.1.el8_2.x86_64                     @@System
    Upgrade  nftables-1:0.9.3-12.el8_2.1.x86_64                                 @BaseOS
    Upgraded nftables-1:0.9.3-12.el8.x86_64                                     @@System
    Upgrade  python3-nftables-1:0.9.3-12.el8_2.1.x86_64                         @BaseOS
    Upgraded python3-nftables-1:0.9.3-12.el8.x86_64                             @@System
    Upgrade  python3-perf-4.18.0-193.28.1.el8_2.x86_64                          @BaseOS
    Upgraded python3-perf-4.18.0-193.19.1.el8_2.x86_64                          @@System
    Upgrade  selinux-policy-3.14.3-41.el8_2.8.noarch                            @BaseOS
    Upgraded selinux-policy-3.14.3-41.el8_2.6.noarch                            @@System
    Upgrade  selinux-policy-targeted-3.14.3-41.el8_2.8.noarch                   @BaseOS
    Upgraded selinux-policy-targeted-3.14.3-41.el8_2.6.noarch                   @@System
    Upgrade  tzdata-2020d-1.el8.noarch                                          @BaseOS
    Upgraded tzdata-2020a-1.el8.noarch                                          @@System
    Upgrade  php-7.4.12-1.el8.remi.x86_64                                       @remi-modular
    Upgraded php-7.4.11-1.el8.remi.x86_64                                       @@System
    Upgrade  php-cli-7.4.12-1.el8.remi.x86_64                                   @remi-modular
    Upgraded php-cli-7.4.11-1.el8.remi.x86_64                                   @@System
    Upgrade  php-common-7.4.12-1.el8.remi.x86_64                                @remi-modular
    Upgraded php-common-7.4.11-1.el8.remi.x86_64                                @@System
    Upgrade  php-fpm-7.4.12-1.el8.remi.x86_64                                   @remi-modular
    Upgraded php-fpm-7.4.11-1.el8.remi.x86_64                                   @@System
    Upgrade  php-gd-7.4.12-1.el8.remi.x86_64                                    @remi-modular
    Upgraded php-gd-7.4.11-1.el8.remi.x86_64                                    @@System
    Upgrade  php-intl-7.4.12-1.el8.remi.x86_64                                  @remi-modular
    Upgraded php-intl-7.4.11-1.el8.remi.x86_64                                  @@System
    Upgrade  php-json-7.4.12-1.el8.remi.x86_64                                  @remi-modular
    Upgraded php-json-7.4.11-1.el8.remi.x86_64                                  @@System
    Upgrade  php-mbstring-7.4.12-1.el8.remi.x86_64                              @remi-modular
    Upgraded php-mbstring-7.4.11-1.el8.remi.x86_64                              @@System
    Upgrade  php-mysqlnd-7.4.12-1.el8.remi.x86_64                               @remi-modular
    Upgraded php-mysqlnd-7.4.11-1.el8.remi.x86_64                               @@System
    Upgrade  php-opcache-7.4.12-1.el8.remi.x86_64                               @remi-modular
    Upgraded php-opcache-7.4.11-1.el8.remi.x86_64                               @@System
    Upgrade  php-pdo-7.4.12-1.el8.remi.x86_64                                   @remi-modular
    Upgraded php-pdo-7.4.11-1.el8.remi.x86_64                                   @@System
    Upgrade  php-sodium-7.4.12-1.el8.remi.x86_64                                @remi-modular
    Upgraded php-sodium-7.4.11-1.el8.remi.x86_64                                @@System
    Upgrade  php-xml-7.4.12-1.el8.remi.x86_64                                   @remi-modular
    Upgraded php-xml-7.4.11-1.el8.remi.x86_64                                   @@System
    Upgrade  oniguruma5php-6.9.6-1.el8.remi.x86_64                              @remi-safe
    Upgraded oniguruma5php-6.9.5+rev1-4.el8.remi.x86_64                         @@System
    Removed  kernel-4.18.0-193.14.2.el8_2.x86_64                                @@System
    Removed  kernel-core-4.18.0-193.14.2.el8_2.x86_64                           @@System
    Removed  kernel-modules-4.18.0-193.14.2.el8_2.x86_64                        @@System
This server is in running the cloud, and appears not to be running UEFI (no /sys/firmware/efi present).

Not sure if any more info is needed; if so, please ask. First time posting here.

Sondergaard
Posts: 4
Joined: 2020/11/03 01:39:41

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by Sondergaard » 2020/11/10 16:39:07

Apparently since the new Boot Loader Specification (BLS) came out there have been instabilities on some setups involving the grubenv file that result in the $kernelopts variable not being set or read as desired.

See, for example, the similar RHEL 8 case https://bugzilla.redhat.com/show_bug.cgi?id=1651686. One of the contributors experienced reboot failure at switchroot due to a problem with his grubenv file. That bug involved add-on kernel options, however, not the basic option of identifying the root partition.

This may well be a bug; the similarity to other reported bugs with BLS and the fact the behavior manifests upon updating to the new BLS are suspicious. But I don't have the skill with the Linux bootloader to prepare and pursue a bug report responsibly. Perhaps someone with that skill will see this and have an idea what's going on.

Next time I update my kernel, I will try TrevorH's suggestion of deleting (that is, renaming) the grubenv file so it can be recreated. Possibly grubenv is invisibly corrupted in a way that the tools won't show but that may be corrected by regenerating the file from scratch. Wow, I haven't dealt with invisible file corruption for decades!

User avatar
TrevorH
Forum Moderator
Posts: 30358
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by TrevorH » 2020/11/10 22:40:52

Don't forget to check the size of the grubenv file, it has to be exactly 1024 bytes or it is invalid.
CentOS 6 died in November 2020 - migrate to a new version!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

ejjl
Posts: 3
Joined: 2020/11/09 07:49:37

Re: CentOS 8, Failed to switch root, '/sysroot' Empty

Post by ejjl » 2020/12/10 14:44:33

I have two installations of CentOS 8. Both of them have this problem.

I checked the grubenv file size and it is exactly 1024. Contents are correct, too.

Post Reply

Return to “CentOS 8 - General Support”