[SOLVED] Booting into Emergency Mode

[jmacdougca]

Hey guys, my machine crashed yesterday and now is only booting into Emergency mode.

I read the man pages on journalctl, learned a few things for sure.

Before the crash I was running a team viewer instance while running facebook live. I think that combo gave the crash. My bad, I learn the hard way =(

I have taken out the RAM and put it back in, no luck.

I see BeeRich was in a similar situation as me, viewtopic.php?f=47&t=74236&p=312630&hil ... de#p312630.


Here are the errors from [journalctl -xb]:
orcacomputers.orcainbox systemd[1]: failed you apply kernel variables.

Systemd-udevd [995] failed to execute ‘/usr/lib/snapd/-device-helper’ ‘/usr/lib/snapd/snap-device-helper add snap_signal-desktop /devices/pci
Same with 996-1000
1000 ends in pc not pci like 995-999
Snaps are containerised software packages that are simple to create and install. So software is having problems loading.

bcma: bus0: no SPROM available.

Failed to start apply kernel variables.

kvm: disabled by bios | then it comes out again.

EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
EDAC sbridge: Couldn't find mci handler
EDAC sbridge: Failed to register device with error -19
kvm: disabled by bios
Similar issue maybe? viewtopic.php?t=71560

systemd-udev[1379-83]:failed to execute 'usr/lib/snapd/snap-device-helper' '/usr/lib/snapd/snap-device-helper add snap_pulseaudio_pulseaudio /devices/pci0000:00

systemd[1]: Timed out waiting for device dev-mapper-orca\x2dhome.device.

systemd[1]: Timed out waiting for device dev-mapper-orca\x2droot.

[desertcat]

Hey guys, my machine crashed yesterday and now is only booting into Emergency mode.

I read the man pages on journalctl, learned a few things for sure.

Before the crash I was running a team viewer instance while running facebook live. I think that combo gave the crash. My bad, I learn the hard way =(

I have taken out the RAM and put it back in, no luck.

I see BeeRich was in a similar situation as me, viewtopic.php?f=47&t=74236&p=312630&hil ... de#p312630.


Here are the errors from [journalctl -xb]:
orcacomputers.orcainbox systemd[1]: failed you apply kernel variables.

Systemd-udevd [995] failed to execute ‘/usr/lib/snapd/-device-helper’ ‘/usr/lib/snapd/snap-device-helper add snap_signal-desktop /devices/pci
Same with 996-1000
1000 ends in pc not pci like 995-999
Snaps are containerised software packages that are simple to create and install. So software is having problems loading.

bcma: bus0: no SPROM available.

Failed to start apply kernel variables.

kvm: disabled by bios | then it comes out again.

EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
EDAC sbridge: Couldn't find mci handler
EDAC sbridge: Failed to register device with error -19
kvm: disabled by bios
Similar issue maybe? viewtopic.php?t=71560

systemd-udev[1379-83]:failed to execute 'usr/lib/snapd/snap-device-helper' '/usr/lib/snapd/snap-device-helper add snap_pulseaudio_pulseaudio /devices/pci0000:00

systemd[1]: Timed out waiting for device dev-mapper-orca\x2dhome.device.

systemd[1]: Timed out waiting for device dev-mapper-orca\x2droot.

Did you read that last entire thread? The last post in the last thread reads:
by alpha754293 » 2019/09/05 17:48:44
Turns out that it was the hard drive failure that was sending the system into emergency mode, and not because of the EDAC.

Do you have any way to check the hard drive to see if it is working?!? If it is an OLD drive you might want to replace it, and while you are at it replace it with an SSD.

A quick thing you can do to tell if it is a HDD problem or not is to drop into BIOS and see if the BIOS recognizes if you have a HDD or not. If it does not see a drive chances are you have a dead drive. You can prove it by replacing it with another drive you knows works. If the BIOS sees that drive, then swap in the original drive and see if it recognizes it. If it does not the drive is REALLY and sincerely DEAD!!

[jmacdougca]

desertcat thanks for the reply. good catch! I saw the hard disk failure in the thread actually. I said to myself "naw the problem cant be hard drive related." lol

It seems all my drives are being recognized by the bios as per the pictures. I had a drive fail on me a while back, I remember the dead drive was not recognized by the bios.

Thoughts?

[jmacdougca]

oh I forgot to mention, I double checked the cables on the motherboard and hard drives before making this post.

[jmacdougca]

updated lsblk picture as requested

[jmacdougca]

Emergency mode picture as requested

[desertcat]

oh I forgot to mention, I double checked the cables on the motherboard and hard drives before making this post.

Well now we are getting somewhere. The next thing to check is if you are booting in UEFI or Legacy BIOS. If you pick the WRONG boot type it will drop you into Emergency Mode (ask me how I know).

Most Linux distros prefer Legacy BIOS, but many can work with UEFI which is a relic of M$ Windows specifically Windows 8.1 and 10. The DEFAULT in all BIOS's is WIDOWS UEFI, but all or many can be set to use Legacy BIOS.

My current BIOS is for the ASUS M5A97 R2.0, and when I built this machine almost exactly 6 years ago I set it so that it could do either Legacy BIOS or Windows (UEFI). The result is when I boot something I will get a picture of the same device, one device say UEFI and the other has nothing (it is the Legacy BIOS device). I slide the Legacy Device to have Priority over any UEFI Device. That said there are some occasions where one or more UEFI device or devices will hijack my preferences. When that happens I suddenly find my machine dropping me into Emergency Mode. Solution: go into BIOS and reset the order and then SAVE. As an added safe guard I go to BOOT and then I select the SSD that is Legacy BIOS and click on it to start the POST and Boot process.

If you don't know if you are set up in UEFI or Legacy BIOS drop into BIOS GoTo BOOT devices and see if you have two devices with the same ID. One will say UEFI the other will be blank (Legacy). Click on one. If once the machine comes up it drop you back into Emergency Mode then repeat the process and select the other.

One hint you can look at is when you set up your partitions if you chose /boot/grub it is expecting to use a Legacy BIOS; if on the other hand you set it up to use /boot/efi/grub then it is expect to use UEFI BIOS. If you have the OS set up to use /boot/grub, but Legacy BIOS has not been enabled it will continually drop you into Emergency Mode. The Solution is go into BIOS and configure it to enable Legacy BIOS and then save and reboot the machine once you have Identified the Legacy devices and it should work. Here the Users Manual is your friend. Usually the adjustments will be found under the Boot tab, and then you need to wade through the various options to enable some and disable others. You want to either totally disable Windows ONLY options, or enable Windows and Legacy devices, or Legacy devices ONLY. Windows + Other (Legacy) is what I chose.

Best Fishes.

D'Cat

[jmacdougca]

Here is the sytem log of my machine:
[journalctl -xb]| https://www.termbin.com/rw6q
[journalctl -b] |https://www.termbin.com/noqq

Output of: [tail -n+0 /etc/sysctl.conf /etc/sysctl.d/*.conf /run/sysctl.d/*.conf /usr/lib/sysctl.d/*.conf | nc termbin.com 9999] | https://termbin.com/y0dg

[desertcat]

updated lsblk picture as requested

There are two potential things I see that *may* giving you problems. You have both a /boot *and* a /boot/efi it is usually one or the other, but not both.

The other thing is your file system type. When I installed my system years ago. I purposely chose CUSTOM install so I had maximum control over everything and set everything to use the EXT4 file system. It is possible that your either your /boot or /boot/efi or both have become corrupted. if either has become corrupted and IF either or both is some form of the EXT (ext, etx2, ext3,ext4) fs then running fsck can usually fix the problem; OTOH if it some type of fs such as LVM, XFS, ZFS,btrfs, or any other of other fs types how to fix them will require something different. Lucky for you the most common fs used in CentOS are LVM, ext*, and xfs. If it is etx* fsck is relly fairly easy to use; if it /them is/are LVM or xfs I can't help you.

But I am getting ahead of myself. First go into BIOS ==> BOOT ==> find the HDD or SSD that has your CentOS release on it ==> select it and see if you can boot directly from it -- this will over ride the boot order. While you are there see if you have two of the EXACT same devices. One will be Legacy, and one will be UEFI. Try booting from both. If one or both fail you could have a corrupted boot partition. One last thing to try if you have not tried it already is to re-seat all the memory.

A final thing you could try -- and it has happened to me -- is if it is a SATA drive, find a NEW SATA data cable and connect it to the drive. Sometimes or reasons unknown the cable simply goes bad, replacing the cable will usually fix the problem. I also keep spare SATA data cables around just for this reason.

Wish I knew what is causing your problem. Going into BIOS and over riding the boot order and seeing what is in there is the first step to try. Step 2 would be to swap out the SATA data cable Step 3 is to assume a corrupted boot file and try to fix it... if, that is, you know what the file system is. I suspect you problem maybe that you have two different boot partitions: a /boot and a /boot/efi partitions and it may be causing an internal conflict, but I'm not sure of this. Step 4 if all else fails do a complete reinstall just don't reformat your /home partition. Please keep me posted. I'll be "curious" to see if you solve the problem if so how.

Stay safe; Be Well

Best Fishes

D'Cat

[desertcat]

Emergency mode picture as requested

I have a final question: Is this hooked to a KVM switch?!? If so try taking it off the KVM switch and hook a Mouse, Keyboard, and Monitor directly into the monitor and try booting. It's a long shot but that eliminated another potential cause of failure.

D'Cat