How to recover broken yum and rpm from the nss update bug

General support questions
Post Reply
User avatar
fakemoth
Posts: 40
Joined: 2006/01/26 18:40:00
Location: Transylvania
Contact:

How to recover broken yum and rpm from the nss update bug

Post by fakemoth » 2015/01/18 07:02:51

Hello, first of all this thread is necessary because I can't find any answer here for now. I am one of the poor souls with a server destroyed by the bug and there are only some threads by various users, but no one answers how to solve it in fact at all, or only by ambigous and useless "you shouldn't..." "you should"...
Let me say it: you, the CentOS and RHEL teams SHOULDN'T let something like this happen. I think you realise how many machines got screwed and don't have snapshots or something like this, not with CentOS 6.x. More: why did this happened in 17th january, when you know about this since 14th, was that hard just to pull down that stupid update?

Yeap. I did it :)

I didn't do any particular thing, just a yum update; there were 3 updates, 2 nss related and the wbm-server by the Webmin team. It updated just one of the nss stuff and the server went berserk. Founding merely no info about this at that time, I started to search the web about how to solve it. And I might got into a worse situation. I take pride in my backups, I have all the critical data in all kind of safe places - but I usually don't backup the whole filesystem as the backups/restores never get it quite right so it is better to go with a fresh OS; and I didn't knew that something like this might happen of course, else I would have added the /var/lib directory... For me it was about the confs, the databases, the user data and so on. So I don't have a backup of that directory.

All the posts were going with rpm -vv --rebuilddb. Which apparently did nothing but really destroying the rpm database:

Code: Select all

[root@ns1 ~]# yum update
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
[root@ns1 ~]# less /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
/var/cache/yum/x86_64//base/mirrorlist.txt: No such file or directory
I than tried this:

Code: Select all

[root@ns1 rpm]# cd /var/lib/rpm/
[root@ns1 rpm]# rm -f /var/lib/rpm/__db*
[root@ns1 rpm]# db_dump Packages-BAKUP | db_load Packages
[root@ns1 rpm]# rpm --rebuilddb
[root@ns1 rpm]# yum update
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
[root@ns1 rpm]# 
This is as serious as it can get, take a look:

Code: Select all

[root@ns1 rpm]# rpm -q httpd
package httpd is not installed
[root@ns1 rpm]# yum install httpd
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
Just some info:

Code: Select all

[root@ns1 rpm]# uname -a
Linux ns1.mumu.ro 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@ns1 rpm]# cat /etc/redhat-release
CentOS release 6.6 (Final) 
What else can I do?
And are there any other dangers besides the rpm and yum not working? Think not but I really don't know, just asking? I am hoping to get over this week and install a CentOS 7 the next week-end...
Don't take the name of root in vain!

glennzo
Posts: 54
Joined: 2005/03/06 19:44:55

Re: How to recover broken yum and rpm from the nss update bu

Post by glennzo » 2015/01/18 09:28:50

Maybe this link will help you. Sounds exactly like the issue you're experiencing.

http://kiteplans.info/2015/01/15/solved ... ey-id-bad/
Powered by Fedora and CentOS

User avatar
fakemoth
Posts: 40
Joined: 2006/01/26 18:40:00
Location: Transylvania
Contact:

Re: How to recover broken yum and rpm from the nss update bu

Post by fakemoth » 2015/01/18 14:13:25

This is not helping, as I mentioned, the rpm database was completely ruined by rpm --rebuilddb.

Here it is the output, just in case, of that tutorial:

Code: Select all

[root@ns1 ~]# yumdownloader nss-softokn-freebl
Loaded plugins: fastestmirror, refresh-packagekit
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Cannot find a valid baseurl for repo: base
[root@ns1 ~]# wget ftp://195.220.108.108/linux/centos/6.6/updates/x86_64/Packages/nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm
--2015-01-18 16:11:10--  ftp://195.220.108.108/linux/centos/6.6/updates/x86_64/Packages/nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm
           => “nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm”
Connecting to 195.220.108.108:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /linux/centos/6.6/updates/x86_64/Packages ... done.
==> SIZE nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm ... 169940
==> PASV ... done.    ==> RETR nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm ... done.
Length: 169940 (166K) (unauthoritative)

100%[==============================================================================================>] 169,940      621K/s   in 0.3s    

2015-01-18 16:11:11 (621 KB/s) - “nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm” saved [169940]

[root@ns1 ~]# rpm2cpio nss-softokn-freebl-3.14.3-19.el6_6.x86_64.rpm | cpio -idmv
./etc/prelink.conf.d
./etc/prelink.conf.d/nss-softokn-prelink.conf
./lib64/libfreebl3.so
./lib64/libfreeblpriv3.chk
./lib64/libfreeblpriv3.so
./usr/lib64/libfreebl3.so
./usr/lib64/libfreeblpriv3.chk
./usr/lib64/libfreeblpriv3.so
./usr/share/dracut/modules.d/05nss-softokn
./usr/share/dracut/modules.d/05nss-softokn/install
960 blocks
[root@ns1 ~]# cp ./lib64/libfreeblpriv3.* /lib64
[root@ns1 ~]# yum update
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Update Process
Loading mirror speeds from cached hostfile
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. $releasever is not a valid release or hasnt been released yet/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/$releasever/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base
[root@ns1 ~]# 
Don't take the name of root in vain!

User avatar
TrevorH
Site Admin
Posts: 33220
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: How to recover broken yum and rpm from the nss update bu

Post by TrevorH » 2015/01/18 16:09:58

Your trouble now is that you have wiped out your rpm database so that it no longer knows about any of the packages that were previously installed. You need to restore /var/lib/rpm from a backup or you need to recreate it (which someone posted recently was possible but I have no idea how).

To anyone else suffering this problem: do not under any circumstances run rpm --rebuilddb as this will only make your problems far worse.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

DonX
Posts: 33
Joined: 2014/11/05 07:24:54
Location: Michigan
Contact:

Re: How to recover broken yum and rpm from the nss update bu

Post by DonX » 2015/01/19 00:23:57

TrevorH is absolutely right about not to run rpm --rebuilddb. I experienced the exact problem as fakemoth and I ran the rebuild command and it made things worse. It took my server host basically 2 days to correct this problem.

fakemoth, if you are with a host company, ask them to correct the problem for you. They should be able to.

User avatar
fakemoth
Posts: 40
Joined: 2006/01/26 18:40:00
Location: Transylvania
Contact:

Re: How to recover broken yum and rpm from the nss update bu

Post by fakemoth » 2015/01/19 06:39:29

Already did all that :)

I am managing the server, so I just need the info about how to rebuild the database. No more "whys" and "hows" about what happened as it is pretty clear at this stage.

Does anyone know how to rebuild the rpm database? Pls read what I already tried...
Don't take the name of root in vain!

DonX
Posts: 33
Joined: 2014/11/05 07:24:54
Location: Michigan
Contact:

Re: How to recover broken yum and rpm from the nss update bu

Post by DonX » 2015/01/20 08:17:02

fakemoth,

See this thread: viewtopic.php?f=13&t=50630 particularly page 2 for information on how to rebuild the rpm database AFTER fixing nss-softokn; see post by avij. Hopefully it'll be of some help for you. Good luck! :)

Post Reply