They’re probably going to kill me for calling this “hotfix firmware,” since I was assured this firmware was GA but not uploaded to the website because of some release engineering red tape. (Whatever, guys, I can’t download it without calling you, so it’s a hotfix as far as I’m concerned.)

Anyway, if you’re having this issue or are planning on replicating large LUNs with IBM Enhanced Remote Mirroring, contact your IBM support engineers and request that they send you firmware >=07.60.40.00.

SANs are important, mission-critical pieces of storage hardware, and as we all know, it’s important to manage change in the environment. However, sometimes mistakes happen — sooner or later, someone is going to delete the wrong LUN. IBM doesn’t really make clear how to recover this without technical support involved, and I can understand why — it’s an important thing to get right.

However, especially late at night when IBM’s Remote Support Center runs on a skeleton crew and can take a few hours to turn around a ticket, we can’t always rely on a timely response from IBM support in order to recover the disk. Since this is a largely undocumented procedure, I’m going to put it out there in the hopes that it helps someone else.

When doing advanced work, I tend to work from the command line using the SMcli utility. However, you can also run scripts in the graphical Storage Manager application. The functionality is oddly hidden in the root window of the DS Storage Manager 10 client, on the screen where you choose your SAN to manage. To access it, right-click your SAN and click “Execute Script.” The script editor window will open. (It would make a lot more sense to put this functionality into the Advanced menu of one of the managed SANs.)

The command-line reference guide for IBM Midrange Storage (LSI) SANs makes mention of the recover logicalDrive command:

recover logicalDrive (drive=(enclosureID,drawerID,slotID) |
Drives=(enclosureID1,drawerID1,slotID1 ... enclosureIDn,drawerIDn,slotIDn) |
array=ArrayName)
[newArray=arrayName]
userLabel="logicalDriveName"
capacity=logicalDriveCapacity
offset=offsetValue
raidLevel=(0 | 1 | 3 | 5 | 6)
segmentSize=segmentSizeValue
[owner=(a | b)
cacheReadPrefetch=(TRUE | FALSE)]

However, it doesn’t tell you where to get the LUN sizes, segment sizes, offsets and other numbers that you need to facilitate a successful recovery. Well, luckily, there’s a couple of places you can turn it up.

If you’ve collected support data recently, you can look inside the support bundle .zip and locate a file called recoveryProfile.csv. If you don’t have a support bundle handy, you might still be in luck — the DS Storage Manager application keeps a copy in its program directory, and you can usually find it at C:\Program Files\IBM_DS\client\data\recovery, ending in _Recovery_Profile.csv and named for the SAN you’re managing. Look at all the lines beginning with Volume, and locate the one that contains the LU name that you’re looking for. It should look like this:

Volume,600A0B80006E09620000BC914BF14835,My_LU,600A0B800047F5F20000BC914BF146C2,512,805306368000,393216000,65536,1,1

As far as I can tell, the fields are:

• Object type (volume, volume group, etc.)
• Volume NAA ID
• Volume name
• Owning array NAA ID
• Block size (typically 512; this might be 4096 on SSD or high-capacity disks with 4k blocks, but I have none of these to test with)
• LUN size in bytes
• Starting offset; on this LUN the unit appears to be (bytes / 2048) but I can’t figure out why
• Segment size in bytes
• Two integers/booleans I haven’t identified

You can take this information and feed it right back into that recover logicalDrive command from the guide:

SMcli -n My_SAN -p My_Password -c 'recover logicalDrive array=My_Array userLabel="My_LU" capacity=805306368000 offset=393216000 raidLevel=5 segmentSize=64;'

Note that the segment size needs to be converted from bytes into kilobytes.

One thing I haven’t figured out is how to preserve the old NAA ID on the LUNs, if this is at all possible. This generally isn’t important, but notably can cause problems with signaturing in VMware.

Expect a follow-up post on restoring an entire physical array.

This is the result of what was described to me as a regression in a bad firmware release, but it’s unclear to me from my discussions with IBM exactly how far back this issue goes. I’m grateful that we didn’t find this in the middle of a production DR failover, but it’s completely ridiculous that an enterprise storage vendor allows such a serious data loss issue into a real release.

This is supposed to be fixed in a firmware update already GA’d but not on the website yet, but I’m awfully hesitant to actually use these large LUNs until IBM hashes out their support for them a little further. I’m not looking to be burned with the exact same thing with a different premium feature.

This decision was made because most people run VMware off of Fibre Channel or iSCSI SAN, where you may do something like taking a copy-on-write snapshot of a VMFS volume and presenting it back to the host as a read-only volume. ESX compares the identifier presented by the LUN to the one written to the disk signature, the one that it expects to see. If they mismatch, it’s assumed that it’s not the original LUN and that it’s a copy. Sometimes this isn’t the case, and your underlying storage has actually changed, either because you made a copy to another LUN or because you’re trying to mount a replicated copy on another SAN at your disaster recovery site. In these cases, to mount the volume as a normal writable volume, you need to resignature it, which re-writes all of the above information to the disk. Since this information includes the UUID, and ESX uses that UUID to reference virtual machines in its inventory, you’ll need to manually re-add all of the virtual machines back to your cluster. This is one of those annoying things that Site Recovery Manager was designed to automate (see TechTarget article).

ESX 3.5 used to automatically mount and present the volume as a read-only snapshot with a new unique name. If you wanted to resignature the volume, you would set the advanced setting LVM.EnableResignature to 1, and you would rescan for volumes. This had the unfortunate consequence of re-signaturing all volumes, even if you only intended to resignature one of them. A new esxcfg-volume command was added to perform this operation, and VMware changed the default behavior in the GUI so that if a volume is detected as a snapshot, you have to manually add the storage, at which time you’ll be prompted about whether you want to mount the volume as a snapshot or if you want to resignature it.

Problem 1: Cloned boot LUNs don’t boot

As of ESX 4, the service console resides in a VMDK on a VMFS volume, so you can run into major issues if you boot from SAN and fail your boot LUNs over to your DR site, because the filesystem used to store your service console is subject to the exact same signaturing issues as other VMFS volumes. The boot LUNs on each server need to be manually resignatured, which is covered by Get VIRTical and VMware KB 1012874.

Problem 2: Non-contiguous extents don’t resignature (right now)

A couple of weeks ago, I ran into a rather significant and nasty regression in vSphere 4. I had taken a 500GB VMFS volume, added a second 500GB extent, and then I had grown each of the volumes by 250GB. As a result, the first LUN occupied 0-500GB and 1000-1250GB in the VMFS volume, while the second LUN occupied 500-1000GB and 1250-1500GB. Notice that the second disk’s start is before the first disk’s end (and this should be fine in any reasonable logical volume manager).

This worked fine for months. When I failed over to our DR site, the volume was detected as a snapshot and I couldn’t resignature because esxcfg-volume thought the LUNs overlapped:


[root@esx01 ~]# esxcfg-volume -l
VMFS3 UUID/label: 4bc639b4-21bbc059-d77b-e41f132c2a8a/shared-esxdev
Can mount: No (duplicate extents found)
Can resignature: No (duplicate extents found)
Extent name: naa.600a0b800047f5f20000bc934bf1480e:1 range: 0 - 1279487 (MB)
Extent name: naa.600a0b80006e09620000bc914bf14835:1 range: 511744 - 1535487 (MB)


After going back and forth with VMware for a very long time on this issue, they finally determined it to be a bug in 4.0 that prevents 4.0 from resignaturing the volume. Don’t extend any volumes defined as extents in a VMFS filesystem (VMware’s recommendation is to not use extents at all unless you absolutely need them to extend a VMFS volume beyond 2TB). If this issue bites you, you can get around it by presenting the volume to an ESX 3.5 host, setting LVM.EnableResignature to 1, rescanning/resignaturing, and then presenting the LUNs back to an ESX 4 host. This should hopefully be fixed by 4.1 U1.

As a final aside: it looks like the open-source VMFS driver has similar problems with non-contiguous extents (it throws back garbage data). I haven’t reported that as a bug yet.

(Before any Exchange people chime in: yes, I’m aware that continuous copy replication/log shipping has been available since Exchange 2007. It just wasn’t viable for larger environments because you couldn’t easily distribute where the databases got replicated, meaning you either ran a replication slave for each Exchange server or seriously overspecified/overcommitted your hardware.)

Microsoft minces words pretty frequently to save face with customers (as most corporations do), and they’re still pushing the opinion that high-end SAN storage is a good idea so as not to rock the boat and upset anyone who already shelled out for high-end SAN hardware to run Exchange. However, the truth of it as far as I can surmise is that Microsoft specifically redesigned Exchange to work on commodity hardware in order to cut operating expenses on their own hosted Exchange offering.

Many applications are seeing a major paradigm shift towards distributed processing like Hadoop and schemaless NoSQL distributed data stores like,  MongoDB and HBase, and proprietary software vendors are starting to take notice and move towards better use of commodity hardware. When there’s a lot of engineering effort involved, though, sometimes the best incentive for a company to improve the efficiency of their products is to try to make money on it themselves, and the results can benefit everybody.

–

For a sysadmin, testing software is really, really hard. We’re constantly stuck in a cycle where we either patch a piece of software and introduce unwanted bugs and regressions, or we leave it unpatched and often vulnerable, worse-performing and missing important new features. There are many tricks we’ve learned over the years to make the process easier, but it’s still a fundamentally difficult activity.

Since I came to system administration from the development side of the fence, I’ve always had a keen fascination with the similarities and differences with the way that software developers and system administrators (and the project managers who herd them) go about their jobs. In particular, I find it amazing that neither role generally has a good grasp of how the other functions, and how they can better work together.

I think that an interesting part of this problem is that software developers have a much easier time of testing things than system administrators do. For everyone to understand my viewpoint, I need to qualify it by saying that when a system administrator needs to test a new release of software before deploying it to a production system, it’s generally not to make sure that any new features introduced in the software are bug-free, because it’s simple enough to just document the problems and not use those features until they’ve stabilized. Rather, the issue is that we need to identify regressions in pieces of code that used to work fine and are now broken.

In the software industry, this is what unit testing is for. Unit testing allows developers to provide a comprehensive set of test cases for a particular function, and make sure that the method works properly for all of them and returns the expected result. Many agile developers believe in writing tests first, then code, and aiming for 100% test coverage to minimize unintended regressions from rapid code changes.

I’m not recommending that system administrators should automate testing other people’s software, because there’s no standardized model for business requirements. However, I do think that a little transparency into the development model of our upstream developers would help us to figure out where testing is and isn’t necessary.

While it’s not adopted across all of the software industry, unit testing is very popular in many rapid development scenarios, and has become more or less institutionalized in certain developer communities like CPAN. If you’re a developer, or at least, if you develop software without gluing together huge numbers of third-party libraries, it’s pretty simple for you to gauge regressions in your own software, because you know (or can easily find out) what the test coverage is for your own project. If you have really thorough unit test coverage, and your test cases are properly written, you shouldn’t have any function/method-level regressions slipping into production code when there’s an update. This doesn’t give the developers a ton of insight into the complex problems, like integration-level or system-level issues, but at least it provides a basic understanding that no minor and insidious issues are creeping up the chain and causing undetected problems.

The problem with unit testing is that the developers run the tests, and they run them on their own systems. This methodology can lead to some really bothersome problems for other people.

When you’re a system administrator, and especially if you’re a system administrator who deals with a lot of proprietary, closed-source software, it becomes very difficult to understand the development methodologies of every single piece of software you plan to update. There’s a certain amount of trust that goes into your Linux vendor’s ability to not break things like glibc that aren’t easily tested. I think the ability to trust a vendor’s stability track record is a wonderful thing, but it’s something that shouldn’t be necessary for system administrators. We should be able to validate the correctness of code on our systems, with our configurations, without fighting the developers for the right to do it.

There’s a constant impedance mismatch and a constant communication gap between developers and sysadmins that needs to be bridged. Software developers need to understand that most sysadmins aren’t developers, and we need an easy way to perform basic correctness validation on the software we install, especially if we install it from the distribution’s or developer’s packages and aren’t running a “make test” or similar during the install process. We need to understand what’s being tested, we need to understand the significance of the test coverage, and we need to be able to figure out what does and doesn’t warrant further testing. As it stands, all the validation that developers are (or aren’t) doing is lost on us, because we don’t get a warm-and-fuzzy from tests that someone else is running that we’ll probably never get to see.

I poked around, and could not find one single utility or tool for monitoring the Windows MPIO framework, so I whipped up a quick script using VBScript and WMI. The script is called like so:

cscript.exe //NoLogo scripts\CheckMpioPaths.vbs /paths 4

(4 paths are used because the server is multipathed on two fabrics, and each of the active/passive controllers is also on each fabric — the server should see 2 controllers on 2 fabrics each, for 4 paths.)

This will cause the script to issue a Nagios CRITICAL if any multipath-registered LUN shows fewer than the given number of paths.

As usual, you can find the script in the GitHub repository for CheckMpioPaths.

Some languages are easier than others. Ironically, I have very little issue with Perl, but C++ gives me this headache every time I try to code something. It never resulted in bad code quality — I don’t think that any of the conventions, in and of themselves, were bad — but I occasionally sort of lost sight of what I was actually supposed to be doing.

Recently, I started work on a small C++ hobby project, wasted a ton of time, and got completely sick of this song and dance. I had again spent so long playing with my damn coding conventions that I failed to actually get work done.

I poked around for a little bit, and I ended up just going with Google’s C++ style guide. I didn’t love it; I didn’t even really like it. There’s a lot of things I completely hate about it. But Google is telling me to shut the fuck up and write some damn code, and it makes it easier to focus on what actually matters — writing (and finishing) a program that does what it’s supposed to.

Sometimes, it’s basic: are you being alerted when any disk array attached to any server suffers a disk failure?

Very often, it’s not: for your SAN-attached systems, are you positive that the multipathing is green? If you’re connected to two storage processors or controllers, can the server see two paths to each of them? Are you getting alerted if you can’t?

Are your port channels running over the number of links that they’re supposed to? How about the ISLs on your FC fabrics?

If you have failover clusters where services run on preferred nodes, are you sure they’re actually located where they’re supposed to be? Are you monitoring that services are all running on their preferred nodes?

If you have asymmetric fall-back connections, like a gigabit switch uplink used to back up a 10-gigabit switch uplink, are you notified when it’s using the backup connection, or do you rely on your users to tell you that things seem to be running slowly?

There’s a difference between things running, and things running smoothly: making sure that your “redundant” equipment and services are actually redundant is the key to keeping issues from turning into problems.

I had originally wanted to make something like this for Cacti, which probably has a much broader install base than the pnp4nagios addon, but the Nagios way was just so easy, and I’d like to share it with anyone who doesn’t want to roll their own basic performance aggregator for it.

This tool gets the following statistics:

• IOPS
• Throughput
• Read percentage
• Cache hit percentage

It gets statistics at the following levels:

• Logical Unit
• Physical Array
• Controller
• Unit

It’s a little quick-and-dirty, but it works:

Like my other projects, it’s hosted on GitHub, so check out the GitHub project for check_smcli_io.