holyhandgrenade.org

The three of you who have been following this blog for awhile have probably noticed that around February of this year, the number of topics I’ve blogged about has dropped pretty significantly. That’s because I left my jack-of-all-trades systems engineer job to take a position as a systems integration lead with Time Inc., a position dealing primarily with the difficult tasks of systems automation and configuration management.

While I love the job, and have a great deal of fondness for the people I work with, I do have to say that the amount of new technology I’ve gotten exposure to has been fairly limited. Though I’ve learned a lot about how a really well-oiled machine runs things, most of my technical posts have been about the somewhat generic subjects of Puppet and Linux, and they haven’t been as varied in scope or dimension as I’d really like.

However, I’ve just accepted a position as Systems and Storage Manager for Cold Spring Harbor Laboratory, where I expect to be spending a lot more time working with the open-source community and working with a team to develop clever solutions to the problems faced by many cash-constrained IT organizations. Being that we have a mission to better the world through scientific research, what better place to contribute to open-source?

(CSHL has a long and storied history of contributions to open-source software, likely dating back even further than Lincoln Stein’s still-used CGI.pm CPAN module).

Expect a lot more good stuff on this blog towards the end of the year as I get to finish up several things I never thought I’d have a chance to (IBM SAN data recovery, I’m looking at you).

I created this as part of the uWSGI package that I’m publishing later this week, but I thought this might also be useful to people not using the package, so here’s a separate post! Hopefully it saves somebody some work.

This script, inspired by many scripts before it for Mongrel and other app servers, looks through /etc/uwsgi and launches an instance for each .ini/.json/.xml/.yaml/.yml file it finds. It expects the directories /var/log/uwsgi and /var/run/uwsgi to exist.

You can find the script on my GitHub page for the RPM:

https://github.com/jgoldschrafe/rpm-uwsgi/blob/master/SOURCES/uwsgi.init

Well, it took me a number of hours of beating on it, but I wrestled Graphite into being FHS-compliant and packaged it up on the holyhandgrenade-testing repo. They’re largely untested and a bit rougher around the edges than I’d like, but they seem to work.

The current version in the repo is 0.9.7c, as it was much easier to rip apart the version I was already using. I’m hoping to have the latest 0.9.9 version up soon.

Update: The packages in the holyhandgrenade-testing repo are now up to date with version 0.9.9.

As with my other packages, you can track changes to the specs through the GitHub repos:

• rpm-graphite-web
• rpm-python-carbon
• rpm-python-whisper

Note the following changes from the standard distribution:

You’ve probably figured out by now that I’m completely insane. I typically don’t let this leak out and affect other people, but it seems that a chunk of my home lab has found its way onto the Internet. As a result, now I have a yum repo.

It’s just for RHEL6 and derivatives right now, and only on x86_64 (is anyone still using i386?), but I’ll probably start cross-compiling for CentOS 5 if anyone has a need.

Right now, the holyhandgrenade repo contains Ruby Enterprise Edition (existing packages on the Internet don’t build for RHEL6) and all Rubygem prerequisites for Chef built as RPM against Ruby Enterprise Edition. RBEL is still needed for the things this repo doesn’t contain (CouchDB, RabbitMQ, etc.).

As a bonus, if you install Chef from this repo, it will actually work. As of this writing, that’s not the case with RBEL. Hooray!

You can install the repo with:

rpm -Uvh http://repo.holyhandgrenade.org/rhel/stable/6/x86_64/holyhandgrenade-release-1.0-2.el6.hhg.noarch.rpm

I was in a rush to get this live, so there’s no GPG signing of packages yet. That will happen soon, I promise.

I’ve also created separate GitHub projects for each package. You can view my GitHub page here.

Now I can start work on that Chef tutorial.

In Part 1, I discussed how storage performance is typically measured in random IOPS, and talked about how to calculate them for a single spinning disk and a RAID array. Today, I’m going to get into the nitty-gritty of striping in RAID-5 and RAID-6, and discuss how to determine the optimal stripe width for your server configuration.

For a lot of workloads, this will be premature optimization. I’d advise you not to think too hard about your storage subsystem unless you’re actually worried that you will be I/O-constrained. Most of these considerations, implemented appropriately, will cut down on your total number of disk operations, but won’t make things faster on an undersubscribed system, where rotational latency and seek times are probably your only pertinent bottlenecks. It’s a better idea to invest your time elsewhere, like finding ways to make your systems easier to manage.

Also note that this article won’t tell you how to get all the numbers you need to properly size your array — not yet, and I plan on getting to that in the near future — but I hope to give you an understanding of what to watch out for as well as a starting point for figuring out how to profile your own applications.

Continue reading »

When we as computer users think of disk performance, we usually think about streaming, sequential performance, otherwise known as throughput. Desktop operating systems have trained us to think in this way, because the most prominent display of disk speed that your average person sees is an Explorer or Finder window showing file copy progress — we know that our music collection is being copied at 25 MB per second, for example. This measurement is a good fit for the task, because it gives us the best approximation of how long it will take until the file copy is finished.

In the server world, though, this generally isn’t how disk performance is measured. Servers are shared resources that do much more complicated things with data than typical desktop systems. Most database access is highly random — you pull a record here, a record there, and piece them together in the application. Rows in a MySQL table are usually no more than a couple of kilobytes each, and the rows you need to join together to service one complex query typically live all over the disk. For most other server-side applications, small files are accessed a lot, and large files are accessed infrequently. So instead of throughput, which is measured in bytes/sec, we typically work with a different measurement called IOPS (pronounced i-ops).

IOPS stands for I/O Operations Per Second, and it refers to the average number of random small reads and writes that a disk drive can perform in one second. Let’s start looking at some numbers and calculating something useful.

Continue reading »

Well, maybe you are and maybe you aren’t. I have no idea. But if your shop is anything like the majority of IT shops I’ve seen, then this assessment is probably on the money.

The runbook is one of the most pervasively mediocre, poorly thought-out and badly-implemented concepts in the entire IT industry. For those of you who are unfamiliar with the term, the runbook is basically a “how can grandma run this application?” document.

Their use should be very strongly scrutinized.

Continue reading »