More on CentOS 5.3 to 5.4

So, here’s a humbling, humiliating and slightly funny follow-up to my last blog post:

I’ve always done my due diligence in making sure upgrades go smoothly. As a result, I have a habit of tirelessly poring over release notes and the “known issues” section therein. However, I got burned this week when I failed to read all of the release notes.

CentOS has a documentation page for the 5.0 series. And as of this writing, the documentation page links to a document called Release Notes. It does not, however, link to a completely different document that also is called Release Notes. I had read the release notes on the documentation page, but not the CentOS-specific release notes document which was only linked from the front page. I suppose it’s my fault for not noticing that 5.0 through 5.3 all have CentOS release notes links pointing to the wiki, and thinking that the wiki might be a good place to look.

Upon asking about my upgrade issues, the always-helpful folks in #centos berated me for not Googling correctly for the release notes, accused me of trolling when I pointed out that I did find (and read) the release notes but that there was a documentation problem, and asked me why I would dare to criticize the free efforts done by a volunteer in maintaining the documentation. Obviously, after finding the only document called “Release Notes” listed under CentOS’s documentation for 5.4, on the page where this documentation would normally be, the perfectly reasonable, thinking man’s approach to the problem would be to Google for CentOS release notes.

After much soul-searching and reflection, and a few minutes spent filing a bug report about the documentation page, I did find the answers I was looking for in the CentOS-specific release notes tucked away on their wiki:

  • CentOS 5.4 includes glibc and kernel updates. For yum updates the recommended procedure is:

yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum update
shutdown -r now

So, here’s the morals of the story:

  • If you try to run the whole upgrade at once using yum upgrade, there is a good chance that you will break your system going from 5.3 to 5.4. Follow the documentation, and update your packages in the order given above, and you should be just fine.
  • If you think you’re missing an important piece of documentation, you probably are.

Did you ever have one of those weeks where everything you learned seemed to be choreographed into place? I think that I’m learning much broader lessons this week about the nature and the danger of assumptions, as the Lone Sysadmin would tell you about me. (Bob Plankers, it turns out, is very much not a “goon,” and one can make a very big ass of themselves by assuming other people are familiar with the other meanings of such a word.)

2 Comments

  1. Yeah, this is pretty much why I don’t do upgrades en masse, its a LOT easier to just start fresh and move your settings from a backup. I can’t really think of any time when I did a major upgrade and it ended well.

  2. I’ve found this to be the case with certain upgrades, but RHEL/CentOS’s minor releases have been pretty good to me in this regard — this is actually the first time in all the time I’ve been administering CentOS systems that I’ve been burned by something like this. Major releases are another story; I’ve found those to be especially problematic even going from one Fedora release to the next (Fedora 8->9 on my build system was a particular nightmare that broke pretty much everything about the system). But again, it’s mostly my own fault for missing the CentOS-specific release notes — I didn’t experience anything that couldn’t have been avoided by finding the correct documentation.

    I still plan to update in this way for minor releases and update packs, as my only damage has been minor, hasn’t affected any user-facing aspects of a system and is much quicker than doing clean reinstalls for the majority of systems. However, I’m absolutely not denigrating your approach: I still absolutely am keeping all of my configuration and data backups on hand when facing OS upgrades like this. I keep most systems managed by Puppet, all system configurations in revision control, and everything’s ready to go when a colorful object of choice hits the fan.

    As we move more towards a virtualized infrastructure, where we can easily stand up clones of virtual machines to test upgrades, I suspect a lot of these problems will go away. As it stands today, though, I’m glad we have at least a basic release management framework in place and were able to catch these problems before they made their way onto any important systems.

    In any case, this is certainly a big motivation for me to move forward with finishing my Spacewalk deployment. Yum has definitely presented us with certain major issues in the release management process.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

© 2014 @jgoldschrafe

Theme by Anders NorenUp ↑