Puppet is a fairly complicated little product once you start to look under the covers, and by now it’s pretty widely know that for larger environments, moving from 2.6 to 2.7 isn’t a particularly straightforward upgrade. Most of people’s various pain points relate to the deprecation of dynamic scoping in favor of lexical scoping and parameterized classes, but there’s some other gotchas that haven’t been as widely publicized. Here’s a few.

Undefined template variables have changed

Previously, if you attempted to look up a variable from a template, and that variable did not exist, it would return a Ruby nil, which is a fairly intuitive and straightforward behavior that a lot of people came to rely on in their conditionals. In Puppet 2.7, however, this value is now the symbol :undefined. Ensure that all of your templates are not running under the assumption that undefined variables return the value nil.

Globbing imports are now considered undefined behavior

If you have this guy at the top of any of your manifests for some reason (like Puppet’s autoloader being horrendous until the 2.6 series):

import '*.pp'

Chances are that it will not work, and instead it will return an error that your class is not defined. Ensure that your classes and defines are all named name.pp and let the autoloader do its thing instead. It should work fine, even for nested classes inside subdirectories.

–show_diff is no longer enabled by default in –noop mode

Some people have operations toolchains that rely on Puppet’s –noop mode showing a diff for each file that it’s going to modify on the next real run. Do note that these scripts will need to be updated to explicitly specify the –show_diff option — the new default behavior is now to log these diffs to syslog instead.

Beyond these three, I had a fairly straightforward upgrade of our Puppet environment. Happy hunting!

The three of you who have been following this blog for awhile have probably noticed that around February of this year, the number of topics I’ve blogged about has dropped pretty significantly. That’s because I left my jack-of-all-trades systems engineer job to take a position as a systems integration lead with Time Inc., a position dealing primarily with the difficult tasks of systems automation and configuration management.

While I love the job, and have a great deal of fondness for the people I work with, I do have to say that the amount of new technology I’ve gotten exposure to has been fairly limited. Though I’ve learned a lot about how a really well-oiled machine runs things, most of my technical posts have been about the somewhat generic subjects of Puppet and Linux, and they haven’t been as varied in scope or dimension as I’d really like.

However, I’ve just accepted a position as Systems and Storage Manager for Cold Spring Harbor Laboratory, where I expect to be spending a lot more time working with the open-source community and working with a team to develop clever solutions to the problems faced by many cash-constrained IT organizations. Being that we have a mission to better the world through scientific research, what better place to contribute to open-source?

(CSHL has a long and storied history of contributions to open-source software, likely dating back even further than Lincoln Stein’s still-used CGI.pm CPAN module).

Expect a lot more good stuff on this blog towards the end of the year as I get to finish up several things I never thought I’d have a chance to (IBM SAN data recovery, I’m looking at you).

I’ve just pushed a pile of important updates into the holyhandgrenade repo. Here’s a quick rundown of the most important changes:

-thirdparty repo

I’ve added another repository, holyhandgrenade-thirdparty, in which I redistribute rebuilds from other people’s SRPMs in an attempt to cut down on the amount of unnecessary dependencies. In particular, I’m trying to kill all the dependencies on the RBEL repo, which I’ve become increasingly unhappy with on account of them doing everything totally differently from Fedora upstream (shit, there’s unmodified rebuilds of openSUSE packages in there!)

I could probably push most of this stuff into the main repo, but I don’t want to seem like I’m taking authorship credit away from some people who really deserve it, like T.C. Hollingsworth who has put a ton of work into his packages in the Node.JS ecosystem. Since he hasn’t provided builds for RHEL 6, I have.

Moving right along.

Node.JS packages

I’ve started keeping a large supply of Node.JS packages supporting Etsy’s statsd and some other endeavors. As it stands, it’s more than enough to support Node.JS standalone, but not enough to daemonize it the de facto standard way (the Forever library). Stay tuned, as this is where I’ll be focusing most of my packaging attention in the next few weeks.

Many of these are in -thirdparty, but a large number that I’m writing will start to make their way into the main -stable and -testing repos.

More statsd and Graphite goodness

I’ve added a bunch of other statsd/Graphite-related packages, specifically:

• collectd-carbon (collectd Python plugin to export statistics to Graphite)
• collectd-graphite (collectd Perl plugin to export statistics for Python)
• python-statsd (synchronous statsd client for Python)
• python-gstatsd (asynchronous/Twisted statsd client/server for Python)

Additionally, the Graphite packages (carbon, whisper, and graphite-web) received significant updates.

A pure-C implementation of statsd will be pushed as soon as I get around to checking it out.

As part of the recent set of updates I’m pushing to the holyhandgrenade-testing repo, I pushed some updated Graphite packages which contain three init scripts for Carbon:

• carbon-aggregator
• carbon-cache
• carbon-relay

As before, I’m making a special post to draw search engine attention to these in case they end up being useful for anyone not using my packages. As usual, you can find these scripts on GitHub:

• carbon-aggregator init script
• carbon-cache init script
• carbon-relay init script 

Note: These are specific to my Graphite packages, which means they specify carbon-{aggregator,cache,relay}.py files in /usr/bin instead of /opt/graphite. If you are using the default /opt/graphite hierarchy, you must change the $exec variables in the scripts.

Happy graphing! 

I created this as part of the uWSGI package that I’m publishing later this week, but I thought this might also be useful to people not using the package, so here’s a separate post! Hopefully it saves somebody some work.

This script, inspired by many scripts before it for Mongrel and other app servers, looks through /etc/uwsgi and launches an instance for each .ini/.json/.xml/.yaml/.yml file it finds. It expects the directories /var/log/uwsgi and /var/run/uwsgi to exist.

You can find the script on my GitHub page for the RPM:

https://github.com/jgoldschrafe/rpm-uwsgi/blob/master/SOURCES/uwsgi.init

Well, it took me a number of hours of beating on it, but I wrestled Graphite into being FHS-compliant and packaged it up on the holyhandgrenade-testing repo. They’re largely untested and a bit rougher around the edges than I’d like, but they seem to work.

The current version in the repo is 0.9.7c, as it was much easier to rip apart the version I was already using. I’m hoping to have the latest 0.9.9 version up soon.

Update: The packages in the holyhandgrenade-testing repo are now up to date with version 0.9.9.

As with my other packages, you can track changes to the specs through the GitHub repos:

• rpm-graphite-web
• rpm-python-carbon
• rpm-python-whisper

Note the following changes from the standard distribution:

You’ve probably figured out by now that I’m completely insane. I typically don’t let this leak out and affect other people, but it seems that a chunk of my home lab has found its way onto the Internet. As a result, now I have a yum repo.

It’s just for RHEL6 and derivatives right now, and only on x86_64 (is anyone still using i386?), but I’ll probably start cross-compiling for CentOS 5 if anyone has a need.

Right now, the holyhandgrenade repo contains Ruby Enterprise Edition (existing packages on the Internet don’t build for RHEL6) and all Rubygem prerequisites for Chef built as RPM against Ruby Enterprise Edition. RBEL is still needed for the things this repo doesn’t contain (CouchDB, RabbitMQ, etc.).

As a bonus, if you install Chef from this repo, it will actually work. As of this writing, that’s not the case with RBEL. Hooray!

You can install the repo with:

rpm -Uvh http://repo.holyhandgrenade.org/rhel/stable/6/x86_64/holyhandgrenade-release-1.0-2.el6.hhg.noarch.rpm

I was in a rush to get this live, so there’s no GPG signing of packages yet. That will happen soon, I promise.

I’ve also created separate GitHub projects for each package. You can view my GitHub page here.

Now I can start work on that Chef tutorial.

In Part 1, I discussed how storage performance is typically measured in random IOPS, and talked about how to calculate them for a single spinning disk and a RAID array. Today, I’m going to get into the nitty-gritty of striping in RAID-5 and RAID-6, and discuss how to determine the optimal stripe width for your server configuration.

For a lot of workloads, this will be premature optimization. I’d advise you not to think too hard about your storage subsystem unless you’re actually worried that you will be I/O-constrained. Most of these considerations, implemented appropriately, will cut down on your total number of disk operations, but won’t make things faster on an undersubscribed system, where rotational latency and seek times are probably your only pertinent bottlenecks. It’s a better idea to invest your time elsewhere, like finding ways to make your systems easier to manage.

Also note that this article won’t tell you how to get all the numbers you need to properly size your array — not yet, and I plan on getting to that in the near future — but I hope to give you an understanding of what to watch out for as well as a starting point for figuring out how to profile your own applications.

Revisiting nomenclature

From Part 1:

• Segment size: The amount of data written to a single disk within a RAID stripe.
• Stripe width: The amount of data contained in a single RAID stripe (segment size × number of data-bearing disks).

Understand your application

If there’s one thing I need to hammer on over and over and over, it’s that you need to understand your application in order to make storage decisions. In particular, there are a few details you should pay attention to, and in the next few days I’ll cover how to find them.

First, if you’re running a commercially-supported application, your vendor probably has some advice on how your RAID array should be configured. That should be your starting point. If you don’t find any specific recommendations, you may be able to find some information about the software.

For now, though, I’ll cover a few basic things you should be asking.

• Type of I/O: Is your workload predominantly sequential or random? What’s your percentage of reads to writes?
• Size of I/O: What is the typical read or write size in your application? How much data does the application read or write, buffered, at once?
• Coalescing: Does your application batch writes in order to cut down on the number of discrete I/O operations before sending them to disk? Does your OS? Does your filesystem?
• Alignment: It doesn’t matter if your application requests data in nice, even, stripe-sized chunks if those don’t line up perfectly with your data on disk. Much of the time, despite the best efforts of application developers, the underlying filesystem, volume manager, or partition table can introduce unwanted alignment problems that split your I/O over disks or between RAID stripes. I’ll be covering this more in Part 3.
  

Understand your vendor

For the remainder of this post, I’m basically going to ignore caching. It’s incredibly important — maybe more important to your performance than all the disk-level recommendations in here combined — but each vendor does it so differently that it’s impossible to make useful generalizations. The important thing is that your controller has a battery-backed write-back cache, that the battery is installed, and that your cache is working.

Please, don’t take anything I say here as gospel. There’s huge variances in the way things are implemented between RAID controllers. Certain optimizations may work on one type of card that don’t work on another. Certain controllers, interfaces, or storage networks flat-out might not perform well on certain configurations.

Bottom line: read your documentation, and consider your vendor’s recommendations.

Mixing workload types: don’t

In my first draft of this post, I forgot this. It’s important.

There are two main types of workloads: sequential and random. Do not mix these on the same array because your random I/O will screw up your sequential I/O by making your drives seek all over the place.

If possible, keep your reads and writes separate as well — this generally reduces contention. For example, if you’re running a database with a separate transaction log, like Microsoft SQL Server or Oracle, keep it on a separate volume. If you’re running an XFS filesystem that’s doing a lot of random I/O, you can keep the journal device on another array for better performance. (Note that this may add another point of failure for your volume, and that may not be acceptable.)

If you’re using a SAN that allows you to create multiple LUNs backed by the same physical array,keep in mind that your LUNs are backed by the same set of disks, and from a disk performance perspective it makes almost no difference whether one LUN or a hundred are being written to. Contention on the array will be contention on the array regardless of whether it’s the same filesystem or not.

Segment sizing

Segment sizes have different impacts, and are arrived at in different ways, depending on what type of array you’re using.

Striping without parity (RAID-0, RAID-0+1)

Because you’re not calculating parity, stripe width is literally irrelevant. That makes calculating your ideal segment size a whole lot easier. I’m going to go over the four main kinds of I/O, and my recommendations for how to deal with them.

Sequential reads: If your workload requests very large I/O sizes for long periods of time, like processing very large files by using very large reads, you’ll benefit from keeping this smaller so you can stream off of multiple disks at once — you want to aim for as large a block size as possible that will still allow you to saturate all of your disks. If it doesn’t, and it synchronously asks for small pieces of data at a time, you’ll get better concurrency if you use larger block sizes and leave your other disks free to service requests from other processes/threads.

To arrive at a number, start with a large block size and incrementally decrease it until the next step down doesn’t get any faster. If you’re not sure, 128k-512k is usually a good range.

Sequential writes: Like the above, if your application streams sequential data very quickly to disk, set your segment size a bit smaller so your controller will be able to saturate multiple disks at once. If it doesn’t issue very large writes to the controller, you’ll benefit from a very large block size; this will help to keep your other disks free while one disk is being written at a time.

To arrive at a number, start with a large block size and incrementally decrease it until the next step down doesn’t get any faster. If you’re not sure, 128k-512k is usually a good range.

Random reads: The important detail with random reads is that each read operation should come from as few disks as possible. You want to set your segments to be at least as large as your average read size to minimize the number of disks needed for any particular read. For a huge majority of applications, setting it too large won’t have nearly as much of an impact as setting it to small.

Profiling your application will get your ideal numbers, but anything smaller than 32k generally isn’t recommended — in addition to the disk I/O penalties, segment sizes this small tend to overburden the controller and cause latency problems. If you’re not sure, 64k-128k will get you good all-around performance with most applications that are heavy on small random reads. If your random reads are larger and pseudo-sequential, like in Microsoft Exchange 2010, you may want to go as high as 256k.

Random writes: As with reads, each write should go to as few disks as possible; sizing your segments too small causes unnecessary seeks and latency. Your software documentation should help you determine the best size for random writes. If you’re not sure, a 64k-128k stripe width usually works very well, with some vendors recommending 256k or even higher. Again, run your own benchmarks and draw your own conclusions.

Striping with parity (RAID-5, RAID-6)

With RAID-5 and RAID-6 and mixed read/write workloads, you should typically determine your optimal stripe width, and then use that number to calculate the appropriate segment size. This can be complicated to do correctly, so it will take me the next few sections to completely explain.

How does RAID-5 really work?

Warning: there be math and binary numbers ahead.

RAID-5 uses parity, a sort of binary checksum, to facilitate drive rebuilds.

Consider the following programming problem that I’ve had asked a few times at job interviews:

You’re given a list of 99 integers from 1 to 100 inclusive. Each integer in the list can occur only one time. Find which integer is missing from the list.

If you’re a math nerd, this should be very straightforward: you take the sum of numbers from 1 to 100, and subtract the sum of all numbers in the list, and you’ll end up with the one that’s missing. After all, we know from middle school algebra that if 5500 – x = 5461, there can only be one value for x.

RAID-5 works on the same principle, but instead of plain addition and subtraction, it uses a special binary operation called XOR (exclusive or). XOR has the following truth table:

One way to think of it is that for X XOR Y, if Y if 1 then you flip the value of X.

Wikipedia notes the following important property of exclusive or operations.

If using binary values for true (1) and false (0), then exclusive or works exactly like addition modulo 2.

This is exactly what we’re doing: we’re taking the sum of each bit across the stripe, and throwing out everything except the least-significant digit.

So let’s start with an over-simplified case. We have a bunch of bytes in a RAID block. In real life, a block would be several kilobytes large, but I don’t have room for that in a table. We’ll pretend that each block is one byte instead. The algorithm stays the same.

To calculate the parity for a block, you simply XOR each byte together. In the following chart, Old Parity is the running total up to this point (i.e. each block’s Old Parity is the previous block’s New Parity), and New Parity is the value after XORing each data block into the parity block.

Then a disk fails, and we don’t know the contents of one block in the stripe:

We just reverse the process and XOR all the remaining numbers together to get our disk’s contents back:

The way that parity is calculated can put all sorts of extra strain on your disks.
Stripe widths and the performance impact of parity

A common misconception among many system administrators is that because most hardware RAID cards perform these XOR operations in hardware using specialized accelerator chips, RAID-5 writes should be fast. This isn’t true; there’s actually substantial disk-level slowdown involved with parity calculations, and those performance hits will never go away.

The layout of data on a RAID array

Recapping the above, a five-disk RAID-5 array might look like this:

Boundary crossings

Any of the lines in the above table is a boundary – if you’re reading from or writing to more than one block, you’re making a boundary crossing. Each of these boundary crossings (inter-disk or inter-stripe) incurs a different performance hit, which I’ll describe momentarily.

The parity block of a given stripe has to always be consistent with the data in it. This means that it’s recalculated, updated, and stored again on every single write to the stripe.

Now, recall from the above section that the parity of a block is calculated by XORing together each bit and getting a unique, reversible result. To do this, we remove it from the parity block by XORing the parity block with the written block’s old value. Then, we XOR it with the new value. To do this, though, we need to know the value of the block we’re replacing. In other words: in order to calculate the parity, we need to read each block in the RAID stripe before it’s written. Ouch.

Let’s say you have a 5-disk RAID-5 array (4 data blocks and 1 parity block), with a 32 KiB segment size on each disk, giving you a 128 KiB stripe width. You then write 64 KiB of data to the beginning of the stripe, which is enough to completely overwrite the first two blocks, but not enough to overwrite the entire RAID stripe.

Because you need to read every block you’re writing, your three apparent disk operations (2 data writes and 1 parity write) becomes six operations instead (2 data reads, 1 parity read, 2 data writes, and 1 parity write). You’re literally doubling the amount of I/O to facilitate a single operation. You’re cutting your write performance in half because of the disk I/O overhead in updating the parity block.

(Sidebar: A good RAID implementation will never need to read more than half the disks in a stripe to calculate parity. You can either read the disks you’re writing, and adjust the parity block accordingly, or you can read the disks you aren’t writing and just calculate a new parity block from scratch.)

It’s much easier to throw out the parity information altogether and calculate it from scratch, which we can do when we make full-stripe writes. When writing an entire stripe to disk, the controller already has all of that stripe’s data in memory, and can just calculate the parity without performing any extra reads. Being able to perform nothing but full-stripe writes is the holy grail of RAID-5 write performance, but it can hurt your read performance.

Array sizing for full-stripe write performance

Most performance-sensitive database applications will write blocks or pages that are 2^n bytes large, e.g. 4k, 8k, 32k, and so forth. In an ideal scenario, you want your stripe width to match your write size in order to eliminate stripe boundary crossings and take advantage of full-stripe writes. If you can’t do that because the writes are too small, you want your stripe width to match your per-disk segment size in order to limit the number of disks that have to be read when re-calculating parity.

In order to maximize those full-stripe writes, you have to carefully consider the number of drives in your array, and not just the segment size. If your main application writes randomly in 32k chunks, a 6-disk RAID-5 (with 5 data blocks per stripe) will never be able to have a 32k stripe width. A 5-disk RAID-5 (with 4 data blocks per stripe) can achieve this easily, though, with an 8k segment size.

In order to properly size and stripe your array, you need to do the following things:

1. Profile your server’s workload to determine your typical write size
2. Calculate your target stripe width, which should generally be 2^n, based on your typical write size
3. Figure out what segment size and disk count will get you to that number

The big tradeoff

You’ve probably figured out by now that RAID-5 tends to have better write performance when stripe widths are small (but not so small that they cause the controller latency issues), and better read performance when stripe widths are large. You will never, ever get great performance at both. Don’t even try. But hopefully, the several thousand useless words I’ve just spit out on RAID-5 will get you good enough app performance where you won’t want to hang yourself.

I’m hoping that Part 3 will cover disk alignment, and Part 4 will cover how to profile your applications on Linux, Solaris and Windows.

When we as computer users think of disk performance, we usually think about streaming, sequential performance, otherwise known as throughput. Desktop operating systems have trained us to think in this way, because the most prominent display of disk speed that your average person sees is an Explorer or Finder window showing file copy progress — we know that our music collection is being copied at 25 MB per second, for example. This measurement is a good fit for the task, because it gives us the best approximation of how long it will take until the file copy is finished.

In the server world, though, this generally isn’t how disk performance is measured. Servers are shared resources that do much more complicated things with data than typical desktop systems. Most database access is highly random — you pull a record here, a record there, and piece them together in the application. Rows in a MySQL table are usually no more than a couple of kilobytes each, and the rows you need to join together to service one complex query typically live all over the disk. For most other server-side applications, small files are accessed a lot, and large files are accessed infrequently. So instead of throughput, which is measured in bytes/sec, we typically work with a different measurement called IOPS (pronounced i-ops).

IOPS stands for I/O Operations Per Second, and it refers to the average number of random small reads and writes that a disk drive can perform in one second. Let’s start looking at some numbers and calculating something useful.

IOPS for a single [spinning] disk

Even though SSDs are gaining a lot of traction on servers because they supply a lot of random IOPS, I’m going to ignore them here. Why? Because their electronics are too sophisticated to model like spinning disks. To get the IOPS numbers for an SSD, your best bet is to look up some independent benchmarks. If those are unavailable, run your own. If that’s not doable, well, you’ll just have to trust the numbers that your vendor gives you.

To calculate for a single spinning disk, though, you just need need two numbers: the disk’s rotational speed, and its seek time. Below is a table I’ve shamelessly lifted from Ronny Egner:

As you can see, an average 7200 RPM disk will net you about 70 IOPS, where a 15,000 RPM disk will give you closer to 170. These numbers can be affected by NCQ, caching, and other drive features. (In a RAID array, many of these drive features are disabled, because the controller does better when performance is more deterministic.)

IOPS for a RAID array

I’m going to ignore caching for simplicity.

Things get complicated here because of the huge variance in the ways that RAID arrays work — I’m sure some people will disagree with the observations I’ve made here. I’m going to assume you have a passing familiarity with the most common RAID levels — if not, Wikipedia has a decent reference. However, I’m going to clarify a handful of definitions I’m going to be using, because the storage industry can’t agree on nomenclature:

• Segment size: The amount of data written to a single disk within a RAID stripe.
• Stripe width: The amount of data contained in a single RAID stripe (segment size × number of data-bearing disks).

When calculating stripe width for RAID-4/5/6, do not include disks used for storing parity.

RAID-0 (striping without mirroring)

RAID-0 doesn’t need to do any special calculations, and it doesn’t need to write anything twice. As a result, all of the disks can be used at the same time for random reads and writes with no penalties. All stripe width calculations should be made with regards to sequential I/O, rather than random I/O.

IOPS: (IOPS per disk × number of disks)

RAID-0+1 (striping with mirroring)

I’m folding RAID-1 in here.

Most controllers will interleave reads between the mirrored disks in the array, which will double your random read performance versus a single disk. Note that this only speeds up random access, because rotational latency rather than seek time is your bottleneck for sequential I/O. Write performance is slightly worse than a single disk because the same data needs to be written to two disks — for a given write, whichever drive has the longer seek time will be slowing you down. Caching usually makes this irrelevant.

Read IOPS: (IOPS per disk × number of disks)
Write IOPS: (IOPS per disk × (number of disks / 2))

RAID-5 (striping with distributed parity)

RAID-5 is a really complicated case that’s heavily reliant on the relationship between your stripe width and your average I/O size, and that complication is why most major database vendors like Oracle will recommend never running on RAID-5.

Read IOPS: (IOPS per disk × (number of disks – 1))
Write IOPS: (IOPS per disk × (number of disks – 1) × RAID-5 write penalty)

The write penalty is a variable scaling factor that varies depending on how well your workload matches your array configuration. At best, there’s virtually no penalty at all. At worst, you might be getting 20% of your expected disk performance. I’m going to go over why that is, and how to optimize your RAID-5 arrays, in Part 2.

(Duncan Epping over at Yellow Bricks has a post where he uses some constant ratios for his RAID write penalties. I have a philosophical disagreement with this idea, but I’ll link to it because my answer of “it depends” isn’t really constructive either.)

RAID-6 (striping with double parity)

Performance characteristics are almost identical to RAID-5, with two significant differences. First, the parity calculations take an order of magnitude more processing power, because the algorithms are much more sophisticated. Second, two disks in each stripe are reserved for parity data — these disks will not contribute to your IOPS.

Read IOPS: (IOPS per disk × (number of disks – 2))
Write IOPS: (IOPS per disk × (number of disks – 2) × RAID-6 write penalty)

Aggregating I/O profiles

One final thing to note: if you have enough concurrent sequential I/O tasks running at the same time, your I/O profile turns from sequential to random. The array is trying to keep these requests from being starved, and slow is usually better than no data at all, so it starts seeking all over the place instead of streaming a nice, even line of consecutive blocks off the disk. Keep this very much in mind when determining which measurement, IOPS vs. sequential throughput, is a more useful measurement for the workload you’re trying to size.

In Part 2, I’ll go over the impact of stripe sizing and how almost everybody does it completely wrong.

Well, maybe you are and maybe you aren’t. I have no idea. But if your shop is anything like the majority of IT shops I’ve seen, then this assessment is probably on the money.

The runbook is one of the most pervasively mediocre, poorly thought-out and badly-implemented concepts in the entire IT industry. For those of you who are unfamiliar with the term, the runbook is basically a “how can grandma run this application?” document.

Their use should be very strongly scrutinized.

When all you have is a hammer, the whole world looks like a nail; or, don’t use a runbook when you need a script

This is so obvious that it should never need documenting for anybody, for any reason, yet I’m constantly seeing people write runbooks that are just lists of actions for a system operator to take, one after the other, when something goes wrong. There are literally no decision points where a human needs to form an intelligent thought to execute this runbook. The runbook reads like a script.

A script. For a person.

Something has gone wrong. The train has flown off the rails.

Isn’t the entire point of technology to make people more productive? So why are we taking something that’s essentially a mechanical, computerized task, easily performed by a script or program, and turning it into a format that needs to be blasted from an output device into someone’s eyes, processed by a human brain, jammed onto a keyboard and mouse, and then back into the infrastructure? Shouldn’t we be skipping the middleman?

Working on fun, thought-provoking problems goes much further for staff happiness and retention than having a cabal of people whose job title is Guy Who Pushes the Button. I completely understand the need for staff engagement. But the right time for that is during build-out and engineering, not in the middle of a crisis where the business is losing money or pissing off customers. I assure you that they’re a lot madder about the outage than your ops team is about it being too easy to fix.

In a crisis, failures should be obvious and recovery should be automated as much as possible to minimize the impact of human error. This brings me to my next point.

A good monitoring system, not a dumb manual process, should tell you what’s wrong

There’s always going to be exceptions to this, of course. Computers are bad at deriving context about why an application’s performance profile has changed. If your page views are a hundred times higher today than they were yesterday because your site ended up on the front page of Digg or Reddit, your site will not be performing the same as it did yesterday. There will always be times where you need a human keeping an eye on the performance charts (humans are much better at reading graphs than computers are) and trying to figure out why things aren’t working the way they’re supposed to be.

(Those of you following the DevOps movement: look up that video about Etsy’s dashboards. The best and brightest ops people these days are keeping an eye on business metrics, like sales figures or numbers of code deployments, rather than low-level system metrics.)

But for a lot of other cases, the runbook is representative of somebody being lazy and not correctly integrating the process with the monitoring system. Any line saying “watch out for _____” should be immediately suspect. Human brains are really powerful, and really good at figuring out real problems. Your ops engineers should be focusing their time on unknown unknowns. If you know what the performance criteria are that signal a problem, you should be monitoring for those conditions automatically. There are a lot of statistical models that can help you, if you’re willing to put in the effort to use them.

Systems should be self-healing

The best of IT shops often simply don’t do this unless they’re integrating the component into a much bigger high-availability project. I’ve found two main reasons.

The first is that admins and engineers seem to believe if they spend enough time building infrastructures correctly in the first place, there won’t be repeatable failures. If you’re going to put the effort into writing a bunch of code to make a system more reliable, shouldn’t you put that effort into just making sure it never happens?

Well, yes and no. Some failures are incredibly difficult to prevent but really easy to detect and really easy to recover from, especially if not all the factors are under your control. But other failures also have highly complex causes, and it may take several break-fix iterations before the problem actually disappears. If you’re building out a reliable service, isn’t it better to cut downtime by 95% for 90% of cases where the problem occurs, rather than eliminating 100% of downtime for 50% of cases?

I’m not saying that technical debt is somehow a good thing, but motivated operations people have accomplished really great things for their end-users with duct tape and staples. There’s nothing wrong with working around a problem as long as the fix isn’t fragile and it doesn’t impede your ability to maintain the application down the road. It doesn’t necessarily mean you’re avoiding the problem; rather, you’re finding better places to invest your time.

This brings us to the second reason: people don’t trust the idea that a system can automatically recover itself from failure. And, really, it ties into the first a little bit: we think our infrastructures are too good to suffer these minor outages, especially from obvious causes. But they aren’t, and a little creative engineering can keep a minor situation from turning into a minor outage, or a minor outage from turning into a major one. And we all have SLAs, even if there’s nothing formal and your boss’s idea of a service level is “keep the systems running well enough where I don’t feel compelled to fire you.”

Take special note of this if you don’t control your applications. As an aside, I used to work at a small web hosting business a number of years ago. We had a number of customers running ASP applications on top of IIS, which is Microsoft’s web server platform. Every once in awhile, a customer’s website would suffer a crash of their application pool, because classic ASP wasn’t good at releasing resources if you weren’t a diligent coder. We couldn’t control the code our customers ran on their sites, but we could monitor their sites and restart the application pool if it started to toss errors in a very specific way.

Many operating systems, like Solaris and Windows, take a very pragmatic approach to the problem. If the service crashes, restart it. If it crashes more than X times, leave it down and let the admin deal with it. These are obvious. Some non-obvious things you might want to consider regardless of how you’re handling high-availability:

1. When the filesystem containing /var/log is almost to capacity, compress or delete old logs before the volume fills up.
2. When daily cronjob X fails because a network service is down, retry it a few times with an exponential backoff instead of waiting until cron runs it again tomorrow.
3. If an application crashes and writes a specific error message into the logs indicating what made it fail, identify the problem, fix it, and start the service back up without human intervention.

Everyone has common, repeatable failures in their infrastructure, though the precise definition of “common” may vary from shop to shop. Not all of these issues will cause outages, especially if the infrastructure is designed for high availability, but let’s not pretend that all our applications are perfect. At the same time, let’s not delude ourselves into thinking that eh, app crashed, restart the daemon is always an adequate solution to a problem. It takes some thinking about the application and understanding it.

Runbooks aren’t always bad

I can think of the following cases where runbooks are useful to an IT organization:

1. Ensuring there’s a contingency plan if the script goes wrong and nobody knows how to fix it
2. Orienting and coordinating staff in an emergency, so everyone knows the appropriate responsibilities, escalations and handoffs
3. Solidifying a process that has so many moving parts that, even though it may take days to document, it might take weeks, months or years to get automated properly