Centos Recover broken rpmdb or missing packages
Did you ever encountered broken rpmdb ? Missing packages, filesystem corruption at /var/lib/rpm and ran out of backups or a restore isn’t possible ? You can fix it in Centos Recover broken rpmdb or missing packages and on other RHEL based systems too.
Let’s assume that worst case scenario. Let’s assume that:
- rpmdb (/var/lib/rpm/rpm__xxx) is broken or missing
- /var/lib/rpm/Packages is broken, missing or worst, empty
- no backups exists for the above files
- no /var/log/rpmdb.log(.*) files available
Recover package lists
All you got to go on now are the yum logs, found in several /var/log/yum.log(.*) files. These logs will list ALL installed or updated packages (including previously removed versions) and some verbose logging text. Save all these files to a safe location as soon as possible. Later combine them into one and clean up all extraneous information leaving them a pure package list, similar to:
This might contain multiple versions of the same packages, don’t worry about that.
Preparing the workplace
You (hopefully) have a functional yum installed on the system, but you will need a yum plugin that is likely not installed. Unfortunately, you cannot use yum to install it since in lack of a rpm database you’ll be prompted to install about half of the packages on the server… again.
The manual approach of downloading the rpm file for yum-plugin-downloadonly (from the CentOS repositories) and manually installing it with rpm (skipping dependencies) is needed:
rpm -i yum-plugin-downloadonly-VERSION.rpm --nodeps
Prepare a work folder on the system:
Place the previously compiled packages list file in this folder in rpm-repair/list.txt
Grab packages and re-enter them into the db
Create a rpm-repair/grab.sh file with the following contents:
for word in $(cat list.txt); do yum install $word -y --downloadonly --downloaddir=/rpm-repair/packages/ ; done;
and make it executable (chmod +x grab.sh).
Create a rpm-repair/repair.sh file with the following contents:
for f in `ls packages`; do rpm -ivh --justdb --force --nodeps packages/$f; done
Also make this file executable (chmod +x repair.sh).
One you’ve got all these set up, run ./grab.sh and sit back and wait (you may want to run this is in a screen if you’re working remotely to avoid having to start again if connection drops).
Once all rpm files are downloaded, run ./repair.sh and wait some more.
When both operations are complete, your rpm db should be at least partially functional. However there will most likely be multiple versions of the same package registered in the db. This can be cleaned up with:
for f in `package-cleanup --dupes`; do echo $f; rpm -e --justdb --nodeps $f; done;
package-cleanup is part of the yum-utils package (which at this stage can be installed directly with yum, yay!)
There may be a couple of packages not registered (maybe they weren’t present in the log, maybe their rpm was no longer available at the particular logged version), but these should not be critical and might get updated/installed the next time you run yum update. The end result may not be perfect but it’s a lot better than being left with a non-update-able system (or having to reinstall it from scratch at this very moment).