(Before any Exchange people chime in: yes, I’m aware that continuous copy replication/log shipping has been available since Exchange 2007. It just wasn’t viable for larger environments because you couldn’t easily distribute where the databases got replicated, meaning you either ran a replication slave for each Exchange server or seriously overspecified/overcommitted your hardware.)

Microsoft minces words pretty frequently to save face with customers (as most corporations do), and they’re still pushing the opinion that high-end SAN storage is a good idea so as not to rock the boat and upset anyone who already shelled out for high-end SAN hardware to run Exchange. However, the truth of it as far as I can surmise is that Microsoft specifically redesigned Exchange to work on commodity hardware in order to cut operating expenses on their own hosted Exchange offering.

Many applications are seeing a major paradigm shift towards distributed processing like Hadoop and schemaless NoSQL distributed data stores like,  MongoDB and HBase, and proprietary software vendors are starting to take notice and move towards better use of commodity hardware. When there’s a lot of engineering effort involved, though, sometimes the best incentive for a company to improve the efficiency of their products is to try to make money on it themselves, and the results can benefit everybody.

…

The service wasn’t run according to Microsoft in-house standards at all, but users would not know this. They wouldn’t know that the Mobile brand and the Microsoft brand were just wrappers around a third-party service.

In the cloud it’s not just data that vanishes, it’s the ability to verify what is actually happening to it. Brands are surface things in the cloud with no guarantee at all that you can trust what goes on beyond them inside the cloud or verify it either.

…

Buy an online backup service from Mozy, Carbonite or a cloud storage service from Nirvanix, Google or Amazon, or from any of the myriad other local, regional and national services springing up, and you have no idea at all of the data centre infrastructure, products and processes involved. You just throw your data in and hope that they look after it properly. You can’t verify that they do. It’s a matter of blind faith.

…

The good news is that this isn’t rocket science. It’s what trade associations of professional service providers do already. They self-regulate by certifying members behave according to standards and carry sufficient insurance for the risks they run if they make mistakes. Look at dentists, lawyers, civil engineers or any other trade professional person or business – they all sport the distinction of their professional body and its standards.

What we need is a code of practice backed up by membership of a Cloud Storage Providers’ Association with certification for members. No business should contract for cloud storage services from suppliers who are not members of such a CSPA body, and the CSPA should rigorously enforce the creation of a minimum acceptable standard of service; and also rigorously police its members and throw out suppliers who fail to meet the standard.

Every cloud storage provider with a belief that they are an honest business providing a good and solid service should see the sense of this, and start making moves for a CSPA-type body to come into being. Without it cloud storage services will be offered by cowboys and incompetents, who lose users data, as SwissDisk, T-Mobile and Microsoft have.

Cloud storage needs open standards for the custodianship of users’ data, and only a reputable trade body can provide it. What is the industry waiting for? Do we need another SwissDisk, another Sidekick before it will act? ®

Now, in this regard, I don’t really think that a standards group for cloud storage in particular is necessarily the right approach, versus something more generalized that could apply to all outsourced cloud IT vendors. I don’t believe that storage, in this regard, is any different than any other IT service. After all, while your data can certainly be lost by a storage incident, whether local, remote or somewhere in between, it can also be lost by a logical failure of the IT software infrastructure leveraging that storage. That’s what happened in this Danger business with T-Mobile: the logical failure apparently occurred (if The Register’s writeup is to be believed) when a server in the cluster threw up and trashed the data.

Danger didn’t have appropriate backups. (Oops. More on online vs. offline backups, and physical vs. logical failures, in another post.) In my experience, there’s a certain threshold of reliability and competence where once your physical infrastructure is robust enough, your potential for logical failures far outweighs your potential for physical failures. (Most backup system failures are really logical failures.) It’s this mismatch that makes business continuity planning so difficult.

But they’re right that the industry needs accountability, and it’s probably going to take a major shakeup for that to happen. Right now, there’s just too many vendors, and they’re all too young for us as IT practitioners to figure out which ones are reliable enough for business needs, and which ones aren’t. Like with any fledgling industry, there will be new technologies, there will be acquisitions, and then there will be vendors who produce an enterprise-ready product.

For the time being, we refer to the ages-old axiom: if you want something done right, you’ve got to do it yourself.