Rounding out the top three is the startling finding that 44 percent of security products had inherent security problems. Security testing issues range from vulnerabilities that compromise the confidentiality or integrity of the system to random behavior that affects product availability. Even though it can be a demanding process, certification with a trusted, established third party is critical to verifying product quality, states the report. Product categories studied were: anti-virus, network firewall, Web application firewall, network IPS, IPSec VPN, SSL VPNs and custom testing.

The report has some caveats. For example:

Even the technology used to store and access test data has seen substantial change. We certainly cannot make the claim that a single, consistent data collection method was employed across all products throughout the timeframe of this study.

Check out the rest of the report; it’s a good read. I’ve long been of the belief that most high-end security products (beyond typical endpoint stuff) are snake oil and don’t provide any kind of real ROI; this report does nothing to change my opinion, especially in the IPS space, where a really remarkably huge portion of the sampled products failed to achieve certification.

…

The service wasn’t run according to Microsoft in-house standards at all, but users would not know this. They wouldn’t know that the Mobile brand and the Microsoft brand were just wrappers around a third-party service.

In the cloud it’s not just data that vanishes, it’s the ability to verify what is actually happening to it. Brands are surface things in the cloud with no guarantee at all that you can trust what goes on beyond them inside the cloud or verify it either.

…

Buy an online backup service from Mozy, Carbonite or a cloud storage service from Nirvanix, Google or Amazon, or from any of the myriad other local, regional and national services springing up, and you have no idea at all of the data centre infrastructure, products and processes involved. You just throw your data in and hope that they look after it properly. You can’t verify that they do. It’s a matter of blind faith.

…

The good news is that this isn’t rocket science. It’s what trade associations of professional service providers do already. They self-regulate by certifying members behave according to standards and carry sufficient insurance for the risks they run if they make mistakes. Look at dentists, lawyers, civil engineers or any other trade professional person or business – they all sport the distinction of their professional body and its standards.

What we need is a code of practice backed up by membership of a Cloud Storage Providers’ Association with certification for members. No business should contract for cloud storage services from suppliers who are not members of such a CSPA body, and the CSPA should rigorously enforce the creation of a minimum acceptable standard of service; and also rigorously police its members and throw out suppliers who fail to meet the standard.

Every cloud storage provider with a belief that they are an honest business providing a good and solid service should see the sense of this, and start making moves for a CSPA-type body to come into being. Without it cloud storage services will be offered by cowboys and incompetents, who lose users data, as SwissDisk, T-Mobile and Microsoft have.

Cloud storage needs open standards for the custodianship of users’ data, and only a reputable trade body can provide it. What is the industry waiting for? Do we need another SwissDisk, another Sidekick before it will act? ®

Now, in this regard, I don’t really think that a standards group for cloud storage in particular is necessarily the right approach, versus something more generalized that could apply to all outsourced cloud IT vendors. I don’t believe that storage, in this regard, is any different than any other IT service. After all, while your data can certainly be lost by a storage incident, whether local, remote or somewhere in between, it can also be lost by a logical failure of the IT software infrastructure leveraging that storage. That’s what happened in this Danger business with T-Mobile: the logical failure apparently occurred (if The Register’s writeup is to be believed) when a server in the cluster threw up and trashed the data.

Danger didn’t have appropriate backups. (Oops. More on online vs. offline backups, and physical vs. logical failures, in another post.) In my experience, there’s a certain threshold of reliability and competence where once your physical infrastructure is robust enough, your potential for logical failures far outweighs your potential for physical failures. (Most backup system failures are really logical failures.) It’s this mismatch that makes business continuity planning so difficult.

But they’re right that the industry needs accountability, and it’s probably going to take a major shakeup for that to happen. Right now, there’s just too many vendors, and they’re all too young for us as IT practitioners to figure out which ones are reliable enough for business needs, and which ones aren’t. Like with any fledgling industry, there will be new technologies, there will be acquisitions, and then there will be vendors who produce an enterprise-ready product.

For the time being, we refer to the ages-old axiom: if you want something done right, you’ve got to do it yourself.

I’m sorry. I know you probably paid a lot for that license, but if your infrastructure is relying on a machine’s ability to transition between VM hosts without rebooting as the crux of your high availability plan, you might want to reconsider.

Yesterday, Rational Survivability (a great all-over-the-place IT blog) had a post titled The Emotion of VMotion. It didn’t occur to me before reading this that my own previous search for a hypervisor that would do live migration was working directly against my own beliefs that uptime should only matter for services. Essentially, the infrastructure should be designed so that a single server down doesn’t contribute to the loss of availability.

That being said, live migration is a neat idea, and eventually it’s going to get to the point that it’s nearly instantaneous. When that happens, failovers will be next to invisible. Maybe we’ll have to reevaluate our approach in that case.

Until then, I read posts from people trying to rely on it to keep their infrastructures up and I worry that their approach is flawed.

Please, build your services for reliability, not just the underlying systems.

Now, I need to preface this by saying that I’m not missing the point of Matt’s post. There’s a lot of administrators out there who do treat live migration as a panacea for whatever ails your reliability problems. Anyone who has attempted to design real high-availability infrastructures is very aware that application-level clustering is more robust and typically more reliable than OS-level clustering, which is more robust than hypervisor-level clustering. But these features don’t compete with each other. They each function as a different piece of the datacenter puzzle. And as Matt implies, the cost savings aren’t right for everyone — but they are right for some people.

Absolutely, without a doubt, clustered services are a wonderful, great idea — that’s why people have been using them for decades, and continue to use them. And even though VMotion makes it very easy to add some server-level resiliency to any host or service, the application-level clusters are becoming much easier to configure and maintain at the same time, thanks to great configuration management tools like Puppet, Chef, and Cfengine.

But the big picture is an entire ecosystem around which VMotion thrives. The big cost driver for virtualization in large datacenter environments is consolidation, and being able to run multiple workloads on the same piece of physical hardware is only the first step. Consolidation ratios are improved substantially when you can transparently load-balance workloads in terms of network traffic, compute power and disk I/O — you don’t have to worry about a single bottleneck breaking your carefully-designed system. In addition to the raw server consolidation gains, you substantially save on engineering power, as there’s a lot less manual labor required to design a viable virtualized infrastructure, and a lot less things go wrong if you get it wrong. And if you require compute capacity on demand — say that the majority of your processing occurs during normal business hours and your servers stay mostly idle afterwards — a solution like DRS can actually completely power down your unused VMware hosts until your compute capacity is needed again.

Sure, this isn’t appropriate for everyone. In a pie-in-the-sky IT infrastructure, grid services would provide uniform access to compute capacity and storage on demand using commodity hardware, like Google or Facebook or other players who rely heavily on things like Hadoop or MapReduce in order to scale their operations. But for most real businesses, which have a real investment in commercial off-the-shelf software like databases, ERP systems, CRM and other necessities, we need hypervisors to abstract away the problem and do the work that the COTS vendors won’t, even if the result isn’t as elegant as it should be. And I’m sure that as the hypervisor marketplace matures and consolidates, VMware, Citrix, Microsoft, Red Hat and other vendors will begin to do things with their platforms that we haven’t even thought of yet. Maybe we’ll see cache-coherent shared-memory virtual infrastructures running over InfiniBand, removing the network overhead that was pointed to as a problem by Rational Survivability. The possibilities are endless.

It seems like in this instance, Matt is railing more against the idea of boot-from-SAN than he is about VMotion himself, as boot-from-SAN is another way of solving the same problem — it adds resiliency against hardware failure, but not a ton else. In various ways, he’s right: if you ignore maintenance of your systems documentation and proper server rebuild procedures in favor of a magical black box, your environment will become an unmaintainable mess as a result. It’s the same argument that Luke Kanies has been making about using Puppet or other configuration management systems versus golden master images. In this respect, I think Matt is right to want to know his systems well enough to rebuild them from scratch. It also makes upgrades and other migrations much simpler and smoother.

But every tool is just that: a tool. And they should be used as tools, and evaluated in terms of their effectiveness as a tool. You shouldn’t throw away a perfectly good tool because it doesn’t live up to the hype you were promised. You should use it if it delivers a real return on investment.

“An architect knows something about everything. An engineer knows everything about one thing.”

-Matthew Frederick, “101 Things I Learned in Architecture School”

This is making me think twice about my job title.

I forgot to mention another thought in my last post “Protect the Data” from Whom? Intruders are not mindly attacking systems to access data. Intruders direct their efforts toward the sources that are easiest and cheapest to exploit. This produces an interesting corollary.

Once other options have been eliminated, the ultimate point at which data will be attacked will be the point at which it is useful to an authorized user.

For example, if a file is only readable once it has been decrypted in front of a user, that is where the intruder will attack once his other options have been exhausted. This means that the only way to completely “protect data” is to make it unusable. If data is not usable then it doesn’t need to exist, so that means intruders will always be able to access data if they are sufficiently resourced and motivated, as explained in my first post on this subject.

This meshes pretty well with my philosophy on information security — it doesn’t matter how much security you layer onto the server side. For any sufficiently secure system, your weakest point of potential compromise is almost always going to be your clients. Banks and online payment services (such as Paypal) have learned this the hard way. For every breach, no matter how insignificant, there are millions of successful phishing attacks.

So many system administrators forget about the client side of things because it’s not their job. Big mistake.