Update: IBM DS4000/5000 replication on big LUNs works again with hotfix firmware

A couple of weeks ago, I posted about my issues with replication of >2TB LUNs on IBM SANs not working correctly using Enhanced Remote Mirroring. Well, IBM got me to install some hotfix firmware (version 07.60.40.00), and the problem appears to be resolved, though I’m still having issues with Flash Copies of one of the affected mirror LUNs showing up to Windows as an empty, uninitialized disk. I’m getting married in a week and am too busy polishing documentation before I take 2 weeks off to open yet another case with IBM. C’est la vie. They’re probably going to kill … Continue Reading →

Recovering a deleted logical drive on an IBM Midrange Storage SAN

First, some keyword spam so this turns up to people who need it: this should apply to all IBM Midrange Series Storage SANs including the DS3200, DS3300, DS3400, DS3950, DS4000, DS4100, DS4200, DS4300, DS4400, DS4500, DS4700, DS4800, DS5020, DS5100, and DS5300. (Whew.) SANs are important, mission-critical pieces of storage hardware, and as we all know, it’s important to manage change in the environment. However, sometimes mistakes happen — sooner or later, someone is going to delete the wrong LUN. IBM doesn’t really make clear how to recover this without technical support involved, and I can understand why — it’s an … Continue Reading →

Replication of LUNs >2TB on IBM DS4000/DS5000 SANs flat-out doesn’t work

…but it says it does. It even reports that the mirroring completed successfully and that the volume status is “Synchronized” when the remote end in fact contains nothing but garbage data. This is the result of what was described to me as a regression in a bad firmware release, but it’s unclear to me from my discussions with IBM exactly how far back this issue goes. I’m grateful that we didn’t find this in the middle of a production DR failover, but it’s completely ridiculous that an enterprise storage vendor allows such a serious data loss issue into a real … Continue Reading →

Monitoring Windows MPIO through Nagios

Sometimes, we need to do SAN maintenance — firmware upgrades, disruptive fabric changes, and the like. When these situations come up, it’s useful to know if anything is in a condition where it will break if it loses its connection to SAN storage, especially if you’re a lowly storage administrator without admin access to any of the Windows systems connected up to the SAN. I poked around, and could not find one single utility or tool for monitoring the Windows MPIO framework, so I whipped up a quick script using VBScript and WMI. The script is called like so: cscript.exe … Continue Reading →

Charting performance data for IBM Midrange Storage Series SANs with PNP4Nagios

If you’ve used IBM SAN products, particularly the DS4000, DS5000 and DS6000 series (which are rebranded LSI), one of the most obnoxious things about it is how you’re pretty much forced to roll your own monitoring tools. Compared to many mainstream vendors (and Sun/Oracle in particular), IBM’s performance monitoring and modelling tools have been lackluster at best and completely unsupplied at worst. The best tool you’ve got is the SMcli, which doesn’t supply a ton of good information, but at least provides you with a starting point for capacity planning. I had originally wanted to make something like this for … Continue Reading →