To enable disk statistics collection/trending, edit /etc/cron.d/sysstat and change the following:

*/10 * * * * root /usr/lib64/sa/sa1 1 1

to this:

*/10 * * * * root /usr/lib64/sa/sa1 -d 1 1

(Obviously, replace “lib64″ with “lib” as appropriate for i386 systems.)

Either wait for the next sa log rotation (at midnight) for sa1 to begin collecting disk statistics, or delete your current day’s statistics. sa1, for whatever historical reason, does not add new counters to an existing sa log file.

I’ve always done my due diligence in making sure upgrades go smoothly. As a result, I have a habit of tirelessly poring over release notes and the “known issues” section therein. However, I got burned this week when I failed to read all of the release notes.

CentOS has a documentation page for the 5.0 series. And as of this writing, the documentation page links to a document called Release Notes. It does not, however, link to a completely different document that also is called Release Notes. I had read the release notes on the documentation page, but not the CentOS-specific release notes document which was only linked from the front page. I suppose it’s my fault for not noticing that 5.0 through 5.3 all have CentOS release notes links pointing to the wiki, and thinking that the wiki might be a good place to look.

Upon asking about my upgrade issues, the always-helpful folks in #centos berated me for not Googling correctly for the release notes, accused me of trolling when I pointed out that I did find (and read) the release notes but that there was a documentation problem, and asked me why I would dare to criticize the free efforts done by a volunteer in maintaining the documentation. Obviously, after finding the only document called “Release Notes” listed under CentOS’s documentation for 5.4, on the page where this documentation would normally be, the perfectly reasonable, thinking man’s approach to the problem would be to Google for CentOS release notes.

After much soul-searching and reflection, and a few minutes spent filing a bug report about the documentation page, I did find the answers I was looking for in the CentOS-specific release notes tucked away on their wiki:

• CentOS 5.4 includes glibc and kernel updates. For yum updates the recommended procedure is:

yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum update
shutdown -r now

So, here’s the morals of the story:

• If you try to run the whole upgrade at once using yum upgrade, there is a good chance that you will break your system going from 5.3 to 5.4. Follow the documentation, and update your packages in the order given above, and you should be just fine.
• If you think you’re missing an important piece of documentation, you probably are.

Did you ever have one of those weeks where everything you learned seemed to be choreographed into place? I think that I’m learning much broader lessons this week about the nature and the danger of assumptions, as the Lone Sysadmin would tell you about me. (Bob Plankers, it turns out, is very much not a “goon,” and one can make a very big ass of themselves by assuming other people are familiar with the other meanings of such a word.)

When running the upgrade, yum upgrade seems to run without a hitch, and returns completely successfully with no errors or warnings. However, what actually happens in the background is that the cleanup process breaks silently, and the package manager gets completely filled up with entries for duplicate packages that shouldn’t be allowed to coexist. I was alerted to the problem by rkhunter, which notified me during its post-reboot run that several files were mismatched versus what the package manager thought they should contain. Now, if you rpm -qa a package with matching versions installed, the order they come back is arbitrary and depends on how they end up in your RPM BDB database. When rkhunter called rpm –verify, it was running against the older version, which was failing the checksum comparison.

The number of package errors that rkhunter actually caught paled in comparison to the huge number of screwed up package entries on the system.

This usually doesn’t cause a problem. In most cases, if the cleanup portion fails, you can just run yum-complete-transaction and it will pick up where it left off. For whatever reason, this doesn’t work here.

After hitting this problem, if you try to run another update, you get output like this:

ruby-1.8.5-5.el5_2.6.i386 from installed has depsolving problems
  --> Missing Dependency: ruby-libs = 1.8.5-5.el5_2.6 is needed by package ruby-1.8.5-5.el5_2.6.i386 (installed)
openssl-devel-0.9.8e-7.el5.i386 from installed has depsolving problems
  --> Missing Dependency: openssl = 0.9.8e-7.el5 is needed by package openssl-devel-0.9.8e-7.el5.i386 (installed)
2:vim-enhanced-7.0.109-4.el5_2.4z.i386 from installed has depsolving problems
  --> Missing Dependency: vim-common = 2:7.0.109-4.el5_2.4z is needed by package 2:vim-enhanced-7.0.109-4.el5_2.4z.i386 (installed)
gcc-c++-4.1.2-44.el5.i386 from installed has depsolving problems
  --> Missing Dependency: gcc = 4.1.2-44.el5 is needed by package gcc-c++-4.1.2-44.el5.i386 (installed)
samba-client-3.0.33-3.7.el5.i386 from installed has depsolving problems
  --> Missing Dependency: samba-common = 3.0.33-3.7.el5 is needed by package samba-client-3.0.33-3.7.el5.i386 (installed)
Error: Missing Dependency: vim-common = 2:7.0.109-4.el5_2.4z is needed by package 2:vim-enhanced-7.0.109-4.el5_2.4z.i386 (installed)
Error: Missing Dependency: samba-common = 3.0.33-3.7.el5 is needed by package samba-client-3.0.33-3.7.el5.i386 (installed)
Error: Missing Dependency: openssl = 0.9.8e-7.el5 is needed by package openssl-devel-0.9.8e-7.el5.i386 (installed)
Error: Missing Dependency: ruby-libs = 1.8.5-5.el5_2.6 is needed by package ruby-1.8.5-5.el5_2.6.i386 (installed)
Error: Missing Dependency: gcc = 4.1.2-44.el5 is needed by package gcc-c++-4.1.2-44.el5.i386 (installed)
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest

I cooked up a hairy one-liner to find the duplicates:

rpm -qa --queryformat="%{name}.%{arch}\n" | sort | uniq -d | perl -ne 's/(.*)\.(.*)/\1/g; print' | xargs rpm -qa --queryformat="%{name}-%{version}-%{release}.%{arch}\n" | sort


(It’s only so long because you need to match the arch on x86_64, and rpm -qa doesn’t play nicely with packagename.arch-format names. Interestingly, though, I’ve only experienced the problem on the i386 servers that I’ve upgraded.)

Here’s the output on one host following a supposedly successful upgrade:


apr-1.2.7-11.el5_3.1.i386
apr-1.2.7-11.i386
apr-util-1.2.7-7.el5_3.2.i386
apr-util-1.2.7-7.el5.i386
audit-libs-1.7.13-2.el5.i386
audit-libs-1.7.7-6.el5_3.3.i386
autofs-5.0.1-0.rc2.102.i386
autofs-5.0.1-0.rc2.131.el5_4.1.i386
centos-release-5-3.el5.centos.1.i386
centos-release-5-4.el5.centos.1.i386
cpio-2.6-20.i386
cpio-2.6-23.el5.i386
cpuspeed-1.2.1-5.el5.i386
cpuspeed-1.2.1-8.el5.i386
crash-4.0-7.2.3.el5.centos.i386
crash-4.0-8.9.1.el5.centos.i386
cups-libs-1.3.7-11.el5_4.3.i386
cups-libs-1.3.7-8.el5_3.4.i386
device-mapper-1.02.28-2.el5.i386
device-mapper-1.02.32-1.el5.i386
device-mapper-event-1.02.28-2.el5.i386
device-mapper-event-1.02.32-1.el5.i386
device-mapper-multipath-0.4.7-23.el5_3.4.i386
device-mapper-multipath-0.4.7-30.el5_4.2.i386
dmidecode-2.10-2.el5_4.i386
dmidecode-2.7-1.28.2.el5.i386
dos2unix-3.1-27.1.i386
dos2unix-3.1-27.2.el5.i386
e2fsprogs-1.39-20.el5.i386
e2fsprogs-1.39-23.el5.i386
e2fsprogs-libs-1.39-20.el5.i386
e2fsprogs-libs-1.39-23.el5.i386
ethtool-6-2.el5.i386
ethtool-6-3.el5.i386
gcc-c++-4.1.2-44.el5.i386
gcc-c++-4.1.2-46.el5_4.1.i386
glibc-2.5-34.i686
glibc-2.5-42.i686
iptables-1.3.5-4.el5.i386
iptables-1.3.5-5.3.el5_4.1.i386
iptables-ipv6-1.3.5-4.el5.i386
iptables-ipv6-1.3.5-5.3.el5_4.1.i386
kernel-2.6.18-128.1.10.el5.i686
kernel-2.6.18-92.1.18.el5.i686
kernel-headers-2.6.18-128.1.10.el5.i386
kernel-headers-2.6.18-164.6.1.el5.i386
kpartx-0.4.7-23.el5_3.4.i386
kpartx-0.4.7-30.el5_4.2.i386
krb5-devel-1.6.1-31.el5_3.3.i386
krb5-devel-1.6.1-36.el5.i386
krb5-libs-1.6.1-31.el5_3.3.i386
krb5-libs-1.6.1-36.el5.i386
krb5-workstation-1.6.1-31.el5_3.3.i386
krb5-workstation-1.6.1-36.el5.i386
less-394-5.el5.i386
less-394-6.el5.i386
lftp-3.5.1-2.fc6.i386
lftp-3.7.11-4.el5.i386
libgcc-4.1.2-44.el5.i386
libgcc-4.1.2-46.el5_4.1.i386
libgomp-4.3.2-7.el5.i386
libgomp-4.4.0-6.el5.i386
libselinux-1.33.4-5.1.el5.i386
libselinux-1.33.4-5.5.el5.i386
libselinux-devel-1.33.4-5.1.el5.i386
libselinux-devel-1.33.4-5.5.el5.i386
libsemanage-1.9.1-3.el5.i386
libsemanage-1.9.1-4.4.el5.i386
libsepol-1.15.2-1.el5.i386
libsepol-1.15.2-2.el5.i386
libstdc++-4.1.2-44.el5.i386
libstdc++-4.1.2-46.el5_4.1.i386
libuser-0.54.7-2.1.el5_4.1.i386
libuser-0.54.7-2.el5.5.i386
libX11-1.0.3-11.el5.i386
libX11-1.0.3-9.el5.i386
libxml2-2.6.26-2.1.2.7.i386
libxml2-2.6.26-2.1.2.8.i386
libxml2-python-2.6.26-2.1.2.7.i386
libxml2-python-2.6.26-2.1.2.8.i386
m2crypto-0.16-6.el5.3.i386
m2crypto-0.16-6.el5.6.i386
mysql-5.0.45-7.el5.i386
mysql-5.0.77-3.el5.i386
mysql-devel-5.0.45-7.el5.i386
mysql-devel-5.0.77-3.el5.i386
mysql-server-5.0.45-7.el5.i386
mysql-server-5.0.77-3.el5.i386
neon-0.25.5-10.el5_4.1.i386
neon-0.25.5-10.el5.i386
newt-0.52.2-12.el5_4.1.i386
newt-0.52.2-12.el5.i386
nfs-utils-lib-1.0.8-7.2.z2.i386
nfs-utils-lib-1.0.8-7.6.el5.i386
nscd-2.5-34.i386
nscd-2.5-42.i386
nss-3.12.2.0-4.el5.centos.i386
nss-3.12.3.99.3-1.el5.centos.2.i386
numactl-0.9.8-7.el5.i386
numactl-0.9.8-8.el5.i386
openssl-devel-0.9.8e-12.el5.i386
openssl-devel-0.9.8e-7.el5.i386
pam-0.99.6.2-4.el5.i386
pam-0.99.6.2-6.el5.i386
perl-5.8.8-18.el5_3.1.i386
perl-5.8.8-27.el5.i386
redhat-rpm-config-8.0.45-29.el5.noarch
redhat-rpm-config-8.0.45-32.el5.centos.noarch
rsh-0.17-38.el5.i386
rsh-0.17-40.el5.i386
ruby-1.8.5-5.el5_2.6.i386
ruby-1.8.5-5.el5_3.7.i386
ruby-irb-1.8.5-5.el5_2.6.i386
ruby-irb-1.8.5-5.el5_3.7.i386
samba-client-3.0.33-3.15.el5_4.i386
samba-client-3.0.33-3.7.el5.i386
sqlite-3.3.6-2.i386
sqlite-3.3.6-5.i386
strace-4.5.18-2.el5_3.3.i386
strace-4.5.18-5.el5.i386
tzdata-2009f-1.el5.noarch
tzdata-2009o-2.el5.noarch
udev-095-14.20.el5_3.i386
udev-095-14.21.el5.i386
vim-enhanced-7.0.109-4.el5_2.4z.i386
vim-enhanced-7.0.109-6.el5.i386
vim-minimal-7.0.109-4.el5_2.4z.i386
vim-minimal-7.0.109-6.el5.i386
ypbind-1.19-11.el5.i386
ypbind-1.19-12.el5.i386
yum-metadata-parser-1.1.2-2.el5.i386
yum-metadata-parser-1.1.2-3.el5.centos.i386


You need to go through these and remove the outdated package versions, one by one. (If you’re confused about which is newer, you can run rpm -qi <packagename> and see, among other details, the date the package was built.) This should be a safe operation; the package manager reference-counts files, and won’t remove a file belonging to multiple packages until all of those packages have been removed, even though you should never have multiple packages owning the same file in the first place. I’m fairly sure that removing these packages manually shouldn’t trigger the %postun scripts, and that the package manager will figure out that removing one version while you have a newer one installed means it’s an upgrade instead of an uninstall. If you’re worried, though, you can do an rpm -e –justdb to remove only the package database entries for the files while not running the scripts or actually removing any files.

Following the removal of the stale packages, a yum -y upgrade should fix the remaining issues.

It’s important to note that the packages do all upgrade — running an rpm –verify on the package after removing the old version does not result in any checksum mismatches or any other visible strangeness. The old versions simply don’t get removed from the package manager, which wreaks havoc on your dependency graphs.

I don’t know what’s causing the problem, but I think it might have something to do with where the upgrades to rpm/yum are placed in the middle of the transaction. Will report back after the next batch of updates, in which I will update rpm and yum first before proceeding with the remainder of the upgrade.