Linux vs. Solaris packaging: it’s a philosophical thing

I thought this was a post worth making because this was the hangup that kept me, as an eight-year Linux user, from really getting Solaris.

One of the biggest questions I see repeated all across the Internet is, “why can’t Solaris’s package management be more like Linux?” Criticisms abound both of Solaris’s SysV packaging format and the way that Solaris packages have to be installed. Solaris’s opponents claim that the Linux packaging system is far superior, Solaris’s is stuck in the 20th century, and Solaris has to adapt or survive. OpenSolaris introduced the Image Packaging System (IPS), designed by Ian Murdock, the founder of the Debian project, largely to bring many of Solaris’s detractors back into the fold by providing another way of doing things. But how much difference does it make in the long run for Solaris as a platform?

Many of the questions and doubts about the Solaris packaging model stem from a very Linux-centric way of functioning. What I would like to explain is why the impedance mismatch between Linux and Solaris packaging is not so much a technological divide as it is a philosophical one.

I’m going to start by explaining how FreeBSD does things, because I think it fits neatly right in the middle of the Linux and the Solaris way of managing installed software.

A FreeBSD installation consists of two discrete platforms. The first is the base system, which is a set of system binaries and core services like FTP, NTP, DNS, DHCP and SMTP software. These are considered to be part of the operating system; they are managed by the installer and updated when you update the OS to a new release. The base system is installed under /usr, and other programs not part of the operating system should not be installed there.

The second is the third-party application layer, which consists of binary packages and “ports,” which are instructions for how to build an application from source. You might compare it to Gentoo’s portage system, or maybe to building all of your Red Hat packages from source RPM. The ports system goes beyond a simple “./configure && make && make install” in that it provides automatic dependency resolution, nice GUI interfaces to common compile options, installation registries and pre/post install/uninstall scripts the same way that a binary package manager would. Packages from the third-party ports/packages system are installed under /usr/local, separate from the base system.

The goal of this system is to keep the two layers as orthogonal as possible, meaning that it limits the surface area where they touch. The base system, for example, contains a copy of OpenSSL. But if you build an application in the ports tree, it will pull in its own copy of OpenSSL that will be used by the programs in /usr/local. The idea is that if you keep the two layers as separate as possible, you can upgrade the underlying system trivially without worrying about all of your third-party dependencies breaking on you. You can also keep your third-party programs from breaking your OS upgrade. And unlike in Linux, if you rely mostly on vendor-supplied libraries, it’s still very easy to install very modern software on a not-very-modern version of your OS.

In Linux, the solution to a major operating system upgrade is to back up your important data files, reformat your partitions and create your system from scratch on the new operating system. This is fine for systems of trivial complexity, but becomes very burdensome when you have an enterprise product like an ERP system or a digital collections manager and you would really, really like to just be able to upgrade the OS without everything breaking on you. One of the obnoxious idiosyncrasies of Linux is that when you go to upgrade, your vendor’s new packages may conflict with something in a third-party RPM you’ve installed. Third-party software can actually break your ability to upgrade the base system because everything shares the same hierarchies and you may encounter a lot of unintended conflicts.

Solaris’s packaging system has historically been the SysV package, which provides dependency resolution and many of the other amenities of modern packaging systems, but there was never a delivery mechanism for simple Internet- or network-based delivery. Many organizations NFS mount a directory full of packages. In many ways, it’s closer to Slackware’s idea of packages than most modern formats like .rpm or .deb. Blastwave was the first community organization to bring Internet-based package management, complete with automatic dependency resolution, to Solaris, but it did so with its own packages, not by touching the base system.

Solaris takes the FreeBSD approach to a more extreme level, partly out of fragmentation and partly out of necessity. Third-party packaging groups like Blastwave and SunFreeware operate independently of one another. Because of this, rather than a /usr vs. /usr/local separation, each Sun packaging group basically builds its own platform, isolated in its own directory hierarchy. Blastwave uses /opt/csw, SunFreeware uses /usr/sfw, and the old Cool Stack suite of web stack packages (which is now part of the Glassfish Web Stack) resides in /opt/csk.

The consequence of this approach is that if you, as an internal packager producing packages for your organization, want to take a piece of software and make a SysV package out of it, you need to build the platform underneath it first. It’s not as simple as writing an Apache package, because you need to rely on your own complex hierarchy of libraries too. When you’re now maintaining 40 packages instead of the 1 you really wanted to build, it becomes simpler to just rely on rsync from a reference system instead. And if you’re running OpenSolaris in production (and there are lots of perfectly valid reasons to do so), you probably don’t want to rely too heavily on vendor-supplied packages because the distribution is a moving target that changes dramatically every six months.

In many environments, the orthogonal-platform approach isn’t a bad thing. You’re probably dealing heavily with change control in the enterprise anyway, and it’s nice to not have to worry quite as much about a Solaris patch bringing down critical system services. Visible Ops teaches us that the most highly-available IT organizations patch far less frequently and rely more on good release management processes and testing updates in a group. Essentially, in a highly change-controlled environment, you’re essentially going to be building your own distribution, whether that involves rsyncing out Solaris binaries or manually creating well-tested update channels in a Red Hat Network Satellite server. And as with FreeBSD, when you need to perform a major OS upgrade on a highly complex system, it dramatically reduces the chances that something is going to break as a result of the vendor’s updates.

In many other scenarios, it is a bad thing. Many server configurations are very simple — LAMP stacks or Mailman servers, for instance — and you don’t need to put the same effort into maintaining them that you would an ERP or CRM system, a single sign-on portal or other important enterprise services. If the system breaks horribly, it can be rebuilt very easily. For the majority of organizations, most systems are like this, and the ability to very quickly bootstrap a system with needed services is still a big draw to the enterprise consumer. And from a security perspective, keeping four different copies of a library on your system, that are all used by different programs, means that there are four times as many security updates to make, and four times as many chances to let something slip through the cracks. Often it means several different configurations to maintain. For this reason, many organizations ignore Blastwave entirely. (Lots of others spurn third-party packages entirely out of security concerns, quite understandably.)

Linux attempts to create an all-inclusive platform where all software is on the same playing field, so to speak. Third-party packages rely on system libraries in the same way that the vendor’s packages do, for better or for worse, and everything benefits from (or breaks from) updates to system packages. For minor updates, this is a great thing. For major updates, this prevents the majority of systems with sufficiently complex configurations from ever being able to perform an in-place upgrade. The downside is mitigated a little bit by the fact that the package management system makes it quite a bit easier to get the new system up and running again.

But what makes Linux special among these three approaches is that there’s absolutely nothing keeping you from designing your own isolated platform using your own dependencies, just like you would on BSD or Solaris. BSD and Solaris try to enforce this separation, while Linux gives you enough rope to hang yourself with if you’re so inclined.

There’s perfectly valid reasoning for all of these approaches, and I don’t think it’s a bad thing that administrators are able to pick which platform to use based on the situation. It’s important to remember that Solaris isn’t lagging in the 20th century — it’s just a grizzled war veteran who understands the realities of enterprise IT administration.

5 Comments

  1. I disagree with this view; I think that Solaris packaging is bad for reasons unrelated to how widely it’s used (or how many packages Solaris includes by default). Instead, I dislike it because it works badly and doesn’t give me the information I need. I wrote a long entry to explain why in more depth.

  2. Oh, wow, it feels like ages since I’ve updated. Thanks for the comments.

    I didn’t mean for this to turn into too much of a discussion about Sun’s packaging system in general; it’s been covered before in much greater depth by people with much greater knowledge than I on all things Solaris. It was mostly intended to explain to Linux-centric people why Solaris’s third-party package ecosystem is so bare-bones compared to Linux’s, and why a lot of effort goes into building a full stack from source. Since nobody else is commenting, though, there’s no reason to be afraid of a little derail. :)

    SysV packaging is definitely showing its age, and you’ll find few outside of the hardcore Sun faithful who actually try to defend it. IPS in OpenSolaris is still dog slow, much slower than any modern package manager ought to be, but I think it does address most of your concerns. Sun has a decent introduction to IPS packaging in their OpenSolaris docs. Also see the command reference for pkg(1), which has some other useful information in it.

    Here’s the nickel version:

    IPS handles upgrading pretty seamlessly. Not only does it handle upgrades in the sane way expected of a modern package manager, but it also can take a full ZFS snapshot of your boot environment so you can roll back to a known-working package set without a lot of fuss. (This isn’t hard using RHN Satellite or Spacewalk either, from the Red Hat side, but it takes a lot more plumbing and infrastructure and relies on your ability to downgrade packages cleanly.)

    Checksums are handled well by “pkg verify”, as are other things that any other modern package manager could sanity-check, like permissions. OpenSolaris loses points by not having the output format easily parseable/scriptable like the output of “rpm –verify” but the functionality is in place and usable, at least.

    Querying and listing packages is done through “pkg contents” and “pkg search”. These functions work identically on local and remote packages, which gives IPS a tiny leg up on its Linux counterparts.

    It also manages dependencies pretty well — that was more or less the entire point of bringing Ian Murdock on board with Sun’s platform team. The lack of a regular Internet-oriented package manager like most Linux distros have had for a very long time is definitely something that’s scared a lot of Linux users away from Solaris.

    Of course, this means pretty much nothing to you if you’re not running OpenSolaris, which takes some balls to do at this point. And, like any major change in an OS like Solaris, it introduces its own set of problems. Here’s a quote from the pkg(5) manpage about package version numbers:

    The first part is the component version. For components tightly bound to OpenSolaris, this will usually be the value of ‘uname -r’ for that version of OpenSolaris. For a component with its own development lifecycle, this sequence will be the dotted release number, such as ‘2.4.10’.

    How very intuitive, Sun.

  3. Joshua Rodman

    2 June, 2010 at 6:35 PM

    Hilariously wrong.

    I’ve had Linux systems seemlessly upgraded across versions of Debian for over 7 years without service interruption. This includes production systems providing live services. People do not reformat instead of upgrade.

    • You seem to be under the impression that I’m arguing with your viewpoint here. Quite the contrary — I think that in an ideal world, these kinds of upgrades should be really trivial. And if you don’t run custom-compiled local versions of software, and if you don’t run proprietary software (especially the “enterprise” variant) that’s frightfully specific about its linked sonames, they stay really trivial. Debian has shown remarkable ability, probably more than any other Linux distribution, in being able to manage these kinds of upgrades.

      If everything you run comes from a package that’s tested by the vendor, and you’re upgrading from one compatible package ecosystem to another, and your config file formats don’t change, your upgrade should work just fine (though results may vary in the wonderfully fragile world of Red Hat). That, however, has absolutely nothing to do with the argument, because that’s simply not the way that the world of proprietary software on commercial UNIX distributions works. The vendors don’t run in lock-step with your operating system, and they don’t provide special repositories so that the Etch version of their software is replaced with the Lenny version when you upgrade. (In fact, you’re remarkably lucky if they even support anything besides Red Hat or SUSE at all.)

      As a Debian guy, you’re probably more used to the FOSS side of the spectrum, so let me just use a generic example from that arena. Say that for whatever reason, you’ve compiled a local version of PHP against OpenLDAP 2.3, and your distribution decides it wants to start using OpenLDAP 2.4’s ABI/soname-incompatible client libraries instead. You upgrade, and suddenly you’ve introduced a problem for yourself if you haven’t made a habit of specifically linking against a ton of your own manually-tracked dependencies, because your libldap2-3.so.0 happens to not exist anymore. You now need to hunt down and re-link all of your dependencies, some of which may not be apparent until certain edge cases during run-time are encountered and a handful of DSO’s are loaded on demand and your programs begin mysteriously crashing because they’re trying to call externs that don’t actually exist.

      Thankfully, we haven’t had ABI changes the size of libc5->libc6 in many years, but it’s absolutely conceivable that an upgrade can break a great many things if you rely on the fact that system libraries are going to stay one particular ABI version between releases.

  4. Joshua Rodman

    8 July, 2010 at 6:37 PM

    You’ve just shown why the Linux approach is the right one.

    On Linux, you can choose just how much you want to maintain yourself vs let the system maintain, which means you can minimize this type of pain.

    Note that I have supported various commercial packages on Debian systems without a problem either. Debian doesn’t remove old sos on upgrade unless requested, so things don’t break, even across major upgrades, even when they have very odd binary dependencies. I have libc5 gcc 2.8-specific C++ x86 binaries that still work, no effort required.

    Contraily, with a system like FreeBSD or Solaris, the pain is typically entirely upon the local administrator to deal with the most recent apache problem.

    I can’t tell you how many times as a commercial software vendor I’ve had to drop everything to deal with some major incompatability introduced by a minor update from a proprietary unix vendor, even though they insist that they’re always ABI compatible. Typically these are situations where you can claim the external symbol signatures haven’t changed, even when the behavior, location, environment variables, or other aspects have.

    As a proprietary software vendor it is easier to support Linux systems than any other unix. That includes all the flavors. Really.

Leave a Reply

Your email address will not be published.

© 2018 @jgoldschrafe

Theme by Anders NorenUp ↑