I’ve been pushing out CentOS 5.4 on a number of test systems this week, and I came upon a very interesting, very insidious, and very annoying problem.
When running the upgrade, yum upgrade seems to run without a hitch, and returns completely successfully with no errors or warnings. However, what actually happens in the background is that the cleanup process breaks silently, and the package manager gets completely filled up with entries for duplicate packages that shouldn’t be allowed to coexist. I was alerted to the problem by rkhunter, which notified me during its post-reboot run that several files were mismatched versus what the package manager thought they should contain. Now, if you rpm -qa a package with matching versions installed, the order they come back is arbitrary and depends on how they end up in your RPM BDB database. When rkhunter called rpm –verify, it was running against the older version, which was failing the checksum comparison.
The number of package errors that rkhunter actually caught paled in comparison to the huge number of screwed up package entries on the system.
This usually doesn’t cause a problem. In most cases, if the cleanup portion fails, you can just run yum-complete-transaction and it will pick up where it left off. For whatever reason, this doesn’t work here.
After hitting this problem, if you try to run another update, you get output like this:
ruby-1.8.5-5.el5_2.6.i386 from installed has depsolving problems
--> Missing Dependency: ruby-libs = 1.8.5-5.el5_2.6 is needed by package ruby-1.8.5-5.el5_2.6.i386 (installed)
openssl-devel-0.9.8e-7.el5.i386 from installed has depsolving problems
--> Missing Dependency: openssl = 0.9.8e-7.el5 is needed by package openssl-devel-0.9.8e-7.el5.i386 (installed)
2:vim-enhanced-7.0.109-4.el5_2.4z.i386 from installed has depsolving problems
--> Missing Dependency: vim-common = 2:7.0.109-4.el5_2.4z is needed by package 2:vim-enhanced-7.0.109-4.el5_2.4z.i386 (installed)
gcc-c++-4.1.2-44.el5.i386 from installed has depsolving problems
--> Missing Dependency: gcc = 4.1.2-44.el5 is needed by package gcc-c++-4.1.2-44.el5.i386 (installed)
samba-client-3.0.33-3.7.el5.i386 from installed has depsolving problems
--> Missing Dependency: samba-common = 3.0.33-3.7.el5 is needed by package samba-client-3.0.33-3.7.el5.i386 (installed)
Error: Missing Dependency: vim-common = 2:7.0.109-4.el5_2.4z is needed by package 2:vim-enhanced-7.0.109-4.el5_2.4z.i386 (installed)
Error: Missing Dependency: samba-common = 3.0.33-3.7.el5 is needed by package samba-client-3.0.33-3.7.el5.i386 (installed)
Error: Missing Dependency: openssl = 0.9.8e-7.el5 is needed by package openssl-devel-0.9.8e-7.el5.i386 (installed)
Error: Missing Dependency: ruby-libs = 1.8.5-5.el5_2.6 is needed by package ruby-1.8.5-5.el5_2.6.i386 (installed)
Error: Missing Dependency: gcc = 4.1.2-44.el5 is needed by package gcc-c++-4.1.2-44.el5.i386 (installed)
You could try using --skip-broken to work around the problem
You could try running: package-cleanup --problems
package-cleanup --dupes
rpm -Va --nofiles --nodigest
I cooked up a hairy one-liner to find the duplicates:
rpm -qa --queryformat="%{name}.%{arch}\n" | sort | uniq -d | perl -ne 's/(.*)\.(.*)/\1/g; print' | xargs rpm -qa --queryformat="%{name}-%{version}-%{release}.%{arch}\n" | sort
(It’s only so long because you need to match the arch on x86_64, and rpm -qa doesn’t play nicely with packagename.arch-format names. Interestingly, though, I’ve only experienced the problem on the i386 servers that I’ve upgraded.)
Here’s the output on one host following a supposedly successful upgrade:
apr-1.2.7-11.el5_3.1.i386
apr-1.2.7-11.i386
apr-util-1.2.7-7.el5_3.2.i386
apr-util-1.2.7-7.el5.i386
audit-libs-1.7.13-2.el5.i386
audit-libs-1.7.7-6.el5_3.3.i386
autofs-5.0.1-0.rc2.102.i386
autofs-5.0.1-0.rc2.131.el5_4.1.i386
centos-release-5-3.el5.centos.1.i386
centos-release-5-4.el5.centos.1.i386
cpio-2.6-20.i386
cpio-2.6-23.el5.i386
cpuspeed-1.2.1-5.el5.i386
cpuspeed-1.2.1-8.el5.i386
crash-4.0-7.2.3.el5.centos.i386
crash-4.0-8.9.1.el5.centos.i386
cups-libs-1.3.7-11.el5_4.3.i386
cups-libs-1.3.7-8.el5_3.4.i386
device-mapper-1.02.28-2.el5.i386
device-mapper-1.02.32-1.el5.i386
device-mapper-event-1.02.28-2.el5.i386
device-mapper-event-1.02.32-1.el5.i386
device-mapper-multipath-0.4.7-23.el5_3.4.i386
device-mapper-multipath-0.4.7-30.el5_4.2.i386
dmidecode-2.10-2.el5_4.i386
dmidecode-2.7-1.28.2.el5.i386
dos2unix-3.1-27.1.i386
dos2unix-3.1-27.2.el5.i386
e2fsprogs-1.39-20.el5.i386
e2fsprogs-1.39-23.el5.i386
e2fsprogs-libs-1.39-20.el5.i386
e2fsprogs-libs-1.39-23.el5.i386
ethtool-6-2.el5.i386
ethtool-6-3.el5.i386
gcc-c++-4.1.2-44.el5.i386
gcc-c++-4.1.2-46.el5_4.1.i386
glibc-2.5-34.i686
glibc-2.5-42.i686
iptables-1.3.5-4.el5.i386
iptables-1.3.5-5.3.el5_4.1.i386
iptables-ipv6-1.3.5-4.el5.i386
iptables-ipv6-1.3.5-5.3.el5_4.1.i386
kernel-2.6.18-128.1.10.el5.i686
kernel-2.6.18-92.1.18.el5.i686
kernel-headers-2.6.18-128.1.10.el5.i386
kernel-headers-2.6.18-164.6.1.el5.i386
kpartx-0.4.7-23.el5_3.4.i386
kpartx-0.4.7-30.el5_4.2.i386
krb5-devel-1.6.1-31.el5_3.3.i386
krb5-devel-1.6.1-36.el5.i386
krb5-libs-1.6.1-31.el5_3.3.i386
krb5-libs-1.6.1-36.el5.i386
krb5-workstation-1.6.1-31.el5_3.3.i386
krb5-workstation-1.6.1-36.el5.i386
less-394-5.el5.i386
less-394-6.el5.i386
lftp-3.5.1-2.fc6.i386
lftp-3.7.11-4.el5.i386
libgcc-4.1.2-44.el5.i386
libgcc-4.1.2-46.el5_4.1.i386
libgomp-4.3.2-7.el5.i386
libgomp-4.4.0-6.el5.i386
libselinux-1.33.4-5.1.el5.i386
libselinux-1.33.4-5.5.el5.i386
libselinux-devel-1.33.4-5.1.el5.i386
libselinux-devel-1.33.4-5.5.el5.i386
libsemanage-1.9.1-3.el5.i386
libsemanage-1.9.1-4.4.el5.i386
libsepol-1.15.2-1.el5.i386
libsepol-1.15.2-2.el5.i386
libstdc++-4.1.2-44.el5.i386
libstdc++-4.1.2-46.el5_4.1.i386
libuser-0.54.7-2.1.el5_4.1.i386
libuser-0.54.7-2.el5.5.i386
libX11-1.0.3-11.el5.i386
libX11-1.0.3-9.el5.i386
libxml2-2.6.26-2.1.2.7.i386
libxml2-2.6.26-2.1.2.8.i386
libxml2-python-2.6.26-2.1.2.7.i386
libxml2-python-2.6.26-2.1.2.8.i386
m2crypto-0.16-6.el5.3.i386
m2crypto-0.16-6.el5.6.i386
mysql-5.0.45-7.el5.i386
mysql-5.0.77-3.el5.i386
mysql-devel-5.0.45-7.el5.i386
mysql-devel-5.0.77-3.el5.i386
mysql-server-5.0.45-7.el5.i386
mysql-server-5.0.77-3.el5.i386
neon-0.25.5-10.el5_4.1.i386
neon-0.25.5-10.el5.i386
newt-0.52.2-12.el5_4.1.i386
newt-0.52.2-12.el5.i386
nfs-utils-lib-1.0.8-7.2.z2.i386
nfs-utils-lib-1.0.8-7.6.el5.i386
nscd-2.5-34.i386
nscd-2.5-42.i386
nss-3.12.2.0-4.el5.centos.i386
nss-3.12.3.99.3-1.el5.centos.2.i386
numactl-0.9.8-7.el5.i386
numactl-0.9.8-8.el5.i386
openssl-devel-0.9.8e-12.el5.i386
openssl-devel-0.9.8e-7.el5.i386
pam-0.99.6.2-4.el5.i386
pam-0.99.6.2-6.el5.i386
perl-5.8.8-18.el5_3.1.i386
perl-5.8.8-27.el5.i386
redhat-rpm-config-8.0.45-29.el5.noarch
redhat-rpm-config-8.0.45-32.el5.centos.noarch
rsh-0.17-38.el5.i386
rsh-0.17-40.el5.i386
ruby-1.8.5-5.el5_2.6.i386
ruby-1.8.5-5.el5_3.7.i386
ruby-irb-1.8.5-5.el5_2.6.i386
ruby-irb-1.8.5-5.el5_3.7.i386
samba-client-3.0.33-3.15.el5_4.i386
samba-client-3.0.33-3.7.el5.i386
sqlite-3.3.6-2.i386
sqlite-3.3.6-5.i386
strace-4.5.18-2.el5_3.3.i386
strace-4.5.18-5.el5.i386
tzdata-2009f-1.el5.noarch
tzdata-2009o-2.el5.noarch
udev-095-14.20.el5_3.i386
udev-095-14.21.el5.i386
vim-enhanced-7.0.109-4.el5_2.4z.i386
vim-enhanced-7.0.109-6.el5.i386
vim-minimal-7.0.109-4.el5_2.4z.i386
vim-minimal-7.0.109-6.el5.i386
ypbind-1.19-11.el5.i386
ypbind-1.19-12.el5.i386
yum-metadata-parser-1.1.2-2.el5.i386
yum-metadata-parser-1.1.2-3.el5.centos.i386
You need to go through these and remove the outdated package versions, one by one. (If you’re confused about which is newer, you can run rpm -qi <packagename> and see, among other details, the date the package was built.) This should be a safe operation; the package manager reference-counts files, and won’t remove a file belonging to multiple packages until all of those packages have been removed, even though you should never have multiple packages owning the same file in the first place. I’m fairly sure that removing these packages manually shouldn’t trigger the %postun scripts, and that the package manager will figure out that removing one version while you have a newer one installed means it’s an upgrade instead of an uninstall. If you’re worried, though, you can do an rpm -e –justdb to remove only the package database entries for the files while not running the scripts or actually removing any files.
Following the removal of the stale packages, a yum -y upgrade should fix the remaining issues.
It’s important to note that the packages do all upgrade — running an rpm –verify on the package after removing the old version does not result in any checksum mismatches or any other visible strangeness. The old versions simply don’t get removed from the package manager, which wreaks havoc on your dependency graphs.
I don’t know what’s causing the problem, but I think it might have something to do with where the upgrades to rpm/yum are placed in the middle of the transaction. Will report back after the next batch of updates, in which I will update rpm and yum first before proceeding with the remainder of the upgrade.