KVM guest fails to switch root

Issues related to applications and software problems and general support
Post Reply
kauer
Posts: 16
Joined: 2020/05/03 07:47:30

KVM guest fails to switch root

Post by kauer » 2020/06/21 01:55:23

I have a KVM virtual that will not boot properly. It stops on "Failed to start Switch Root". A screenshot is attached showing this.

I don't know how to access the system, as I have no serial console. Is there any other way in? I will try virt-manager, but my link is very slow and this has not been workable in the past.

Ironically I was backing up the system before adding a serial console. My backup process was:

- stop the VM
- dump the machine definition definition
- snapshot the VM to an external snapshot
- start the VM
- copy/compress the virtual disk to a backup location
- merge the snapshot back

All of that appears to have worked fine, has also worked fine in the past, but this time - no.

My plan is to try using guestmount to look at rdsosreport.txt, then restore from the backup just taken, but I suspect the backup just taken will have the same problem. Also, even if it provides a clue, I will still be unable to access the VM to rectify anything.

I realise that my urgency is no-one else's, but this is pretty urgent for me.

Regards, K.
Attachments
tims.jpg
tims.jpg (139.16 KiB) Viewed 320 times

kauer
Posts: 16
Joined: 2020/05/03 07:47:30

Re: KVM guest fails to switch root

Post by kauer » 2020/06/21 03:02:37

Minor update: After installing guestmount I was able to mount the (shutdown) virtual disk. Of course there was nothing in /run at all (RAM disk?)

kauer
Posts: 16
Joined: 2020/05/03 07:47:30

Re: KVM guest fails to switch root

Post by kauer » 2020/06/21 09:01:51

Further update: The backup failed with exactly the same error. IMHO this means that the act of taking the backup in some way corrupted the virtual disk, which then a) became the backup and b) was used by the restarted virtual.

I was able to restore from an older backup which had been taken using a simpler procedure:
- stop the VM
- dump the machine definition
- copy/compress the virtual disk
- start the VM

The reason for the procedure described in the OP is to have the VM running again sooner, while the copy/compress happens (which takes half an hour or so). Clearly there is a problem with it. Worse, it's an intermittent problem, because I've used that method quite a few times, but never with this disastrous result.

User avatar
TrevorH
Forum Moderator
Posts: 29149
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: KVM guest fails to switch root

Post by TrevorH » 2020/06/21 17:01:26

Unfortunately there is no useful information in your original screenshot so it is impossible to tell what the problem was now. If you have another example, before you switch away from that screen with the emergency mode on it, use Shift-PgUp to look back up the console output and see if you can see whatever the original error was.

The most common reason for entering emergency mode is that someone added an entry to /etc/fstab and did not verify that it was correct (using mount -a) before rebooting. An unmountable filesystem is sufficient reason for entering emergency mode unless it also includes the option 'nofail'.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

kauer
Posts: 16
Joined: 2020/05/03 07:47:30

Re: KVM guest fails to switch root

Post by kauer » 2020/06/22 09:11:59

Thanks, Trevor.

I'd have loved to see more information. However, that screen was a screenshot taken by virsh screenshot, not a real screenshot of a live terminal. If I'd had a live terminal I could have done much more investigation. I did try to take several really quick sequential screenshots, but it didn't work. There was no overlap and very large gaps.

I had made no changes to fstab. The VM had been rebooted recently without incident.

Things I checked (by looking at the non-booting virtual disk with guestmount were:
- was there an os-release file (yes)
- did it point into /usr/lib (yes)
- was the /usr/lib target an actual file? (yes)
- did the latest systemctl journal contain anything useful (no)

If you have any other ideas, I still have the dud virtual disk available for inspection. I could even try booting it I suppose, but without a serial console I have no access to the VM in emergency mode.

The good news is that after restoring from a week-old backup I gave the restored VM a serial console, so if it happens again I will have more tools at my disposal.

User avatar
TrevorH
Forum Moderator
Posts: 29149
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: KVM guest fails to switch root

Post by TrevorH » 2020/06/22 09:55:28

Use a vnc console (or a plain text one) then so you can see the errors.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

kauer
Posts: 16
Joined: 2020/05/03 07:47:30

Re: KVM guest fails to switch root

Post by kauer » 2020/06/22 12:33:19

Oh! I didn't know that VNC was a possibility :-(

From my OP:
I don't know how to access the system, as I have no serial console. Is there any other way in? I will try virt-manager, but my link [edit: actually the link the server is on] is very slow and this has not been workable in the past.
I suspect VNC would have had the same problem as virt-manager i.e., would have been too slow over the available link. But that is something I can now test with working VMs so I will know for next time. I'm going to have to read up on how virtuals are "spoken to". because I also don't really know how virt-manager connects to virtuals. Maybe virt-manager uses the VNC protocol? Anyway - more research needed.

Not sure what you mean by "a plain text one". A plain text one is what I was trying to get with the serial console. No network connectivity was possible.

User avatar
TrevorH
Forum Moderator
Posts: 29149
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: KVM guest fails to switch root

Post by TrevorH » 2020/06/22 12:45:54

Your VM has a vnc port open on localhost on the host machine. Use virsh vncdisplay $vm on the host to find out which one it is (it defaults to whatever is available but can be a fixed port if you make it so).

To get a plain text console you should just be able to run virsh console $vm
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

kauer
Posts: 16
Joined: 2020/05/03 07:47:30

Re: KVM guest fails to switch root

Post by kauer » 2020/06/22 13:39:05

To get a plain text console with virsh console name_of_vm, you do need a serial console on the VM. I didn't have one (and in fact getting one was the point of this exercise). It might be possible to connect to a pty, I haven't tried that.

For VNC the guest must be set up with VNC access enabled. When I run virsh vncdisplay name_of_vm for my guest, it says error: Failed to get VNC port. Is this domain using VNC?

Checking the dumped XML machine definition, it seems the virtual was created with SPICE graphics rather than VNC graphics, so no VNC without reconfiguring the VM. I have a vague memory of choosing SPICE graphics so that the system could use the matching guest enhancements that allow smoother capture/release of the mouse, auto-resizing in windows and such-like. In hindsight that was a bit silly as this guest is a system with no graphical interface. VNC would have been a better choice, as more flexible (i.e., not requiring virt-manager). But TBH the bandwidth really does not permit a graphical interface, even when it is just displaying a text console.

Ultimately, the right interface for this guest really is just a serial console (or pty ifthat is possible).

So I think I was snookered by circumstance from the start :-) But I do appreciate the ideas, and have certainly learned a lot from the experience.

User avatar
TrevorH
Forum Moderator
Posts: 29149
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: KVM guest fails to switch root

Post by TrevorH » 2020/06/22 16:22:48

Remmina (in EPEL) supports spice, I suspect other things will too. You could route that via an ssh tunnel and connect that way.
CentOS 6 will die in November 2020 - migrate sooner rather than later!
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 is dead, do not use it.
Full time Geek, part time moderator. Use the FAQ Luke

Post Reply

Return to “CentOS 8 - General Support”