Home › Monthly Archives › May 2010

Monitoring Windows MPIO through Nagios

Jeff May 30, 2010 Sysadmin 0

Sometimes, we need to do SAN maintenance — firmware upgrades, disruptive fabric changes, and the like. When these situations come up, it’s useful to know if anything is in a condition where it will break if it loses its connection to SAN storage, especially if you’re a lowly storage administrator without admin access to any of the Windows systems connected up to the SAN.

I poked around, and could not find one single utility or tool for monitoring the Windows MPIO framework, so I whipped up a quick script using VBScript and WMI. The script is called like so:

cscript.exe //NoLogo scripts\CheckMpioPaths.vbs /paths 4

(4 paths are used because the server is multipathed on two fabrics, and each of the active/passive controllers is also on each fabric — the server should see 2 controllers on 2 fabrics each, for 4 paths.)

This will cause the script to issue a Nagios CRITICAL if any multipath-registered LUN shows fewer than the given number of paths.

As usual, you can find the script in the GitHub repository for CheckMpioPaths.

Just use someone else’s coding convention already

Jeff May 24, 2010 Sysadmin 0

If there’s one thing that’s bugged me throughout my entire coding career, it’s the fact that I can’t seem to stick to a single coding style for a given language. Scope decorators, braces, spacing around parentheses, Hungarian notation, variable and method naming conventions — there’s so many stupid and trivial things to think about, with so many exceptions and gotchas, that after a while it seems like you end up putting half as much time into figuring out how you’re going to write your program as you actually do designing and coding it. Months later, I’ll have an epiphany, and change my coding style, until months after that, I’ll have another epiphany and change it back. This is an endless cycle.

Some languages are easier than others. Ironically, I have very little issue with Perl, but C++ gives me this headache every time I try to code something. It never resulted in bad code quality — I don’t think that any of the conventions, in and of themselves, were bad — but I occasionally sort of lost sight of what I was actually supposed to be doing.

Recently, I started work on a small C++ hobby project, wasted a ton of time, and got completely sick of this song and dance. I had again spent so long playing with my damn coding conventions that I failed to actually get work done.

I poked around for a little bit, and I ended up just going with Google’s C++ style guide. I didn’t love it; I didn’t even really like it. There’s a lot of things I completely hate about it. But Google is telling me to shut the fuck up and write some damn code, and it makes it easier to focus on what actually matters — writing (and finishing) a program that does what it’s supposed to.

Resilient infrastructures are only useful if they actually stay resilient

Jeff May 24, 2010 Sysadmin 2

Ask yourself a question: for every piece of resiliency you supposedly have in your network, are you really positive that it’s not running in a degraded state? Really, really sure?

Sometimes, it’s basic: are you being alerted when any disk array attached to any server suffers a disk failure?

Very often, it’s not: for your SAN-attached systems, are you positive that the multipathing is green? If you’re connected to two storage processors or controllers, can the server see two paths to each of them? Are you getting alerted if you can’t?

Are your port channels running over the number of links that they’re supposed to? How about the ISLs on your FC fabrics?

If you have failover clusters where services run on preferred nodes, are you sure they’re actually located where they’re supposed to be? Are you monitoring that services are all running on their preferred nodes?

If you have asymmetric fall-back connections, like a gigabit switch uplink used to back up a 10-gigabit switch uplink, are you notified when it’s using the backup connection, or do you rely on your users to tell you that things seem to be running slowly?

There’s a difference between things running, and things running smoothly: making sure that your “redundant” equipment and services are actually redundant is the key to keeping issues from turning into problems.

Charting performance data for IBM Midrange Storage Series SANs with PNP4Nagios

Jeff May 24, 2010 Sysadmin 0

If you’ve used IBM SAN products, particularly the DS4000, DS5000 and DS6000 series (which are rebranded LSI), one of the most obnoxious things about it is how you’re pretty much forced to roll your own monitoring tools. Compared to many mainstream vendors (and Sun/Oracle in particular), IBM’s performance monitoring and modelling tools have been lackluster at best and completely unsupplied at worst. The best tool you’ve got is the SMcli, which doesn’t supply a ton of good information, but at least provides you with a starting point for capacity planning.

I had originally wanted to make something like this for Cacti, which probably has a much broader install base than the pnp4nagios addon, but the Nagios way was just so easy, and I’d like to share it with anyone who doesn’t want to roll their own basic performance aggregator for it.

This tool gets the following statistics:

  • IOPS
  • Throughput
  • Read percentage
  • Cache hit percentage

It gets statistics at the following levels:

  • Logical Unit
  • Physical Array
  • Controller
  • Unit

It’s a little quick-and-dirty, but it works:

check_smcli_io

Like my other projects, it’s hosted on GitHub, so check out the GitHub project for check_smcli_io.