You don’t have to be on the forefront of bleeding-edge server technology to know that x86 systems with 12 terabytes of RAM simply don’t exist. With RAM density as it is right now, there’s simply no way to fit that many DIMMs on a board. However, some inventive software steps up to the plate.

We’ve been meeting with a company called ScaleMP. ScaleMP is, in strict terms, a virtualization software vendor. However, unlike companies like VMware, ScaleMP specializes in using virtual machine monitors to aggregate CPU and memory resources among a number of InfiniBand-connected hosts, presenting a logical system with all of the combined CPU and memory resources of the aggregated physical machines. Through their black magic technology, they apparently do this without substantial overhead to the host, and the resulting virtual machine performs on par with a parallelized MPI solution utilizing operating systems running atop bare metal on the physical nodes. The difference, of course, is that you have a very large coherent block of memory to work with. If you’re familiar with Isilon’s storage architecture, the pattern should look familiar.

There’s an article from HPCwire written by Shai Fultheim, ScaleMP’s CEO, that sums up this approach a lot better than I could hope to. But the ten-cent version is that you can use this to virtualize compute, memory and I/O resources to present a very large system for single tasks that require tons of memory and extremely fast parallelism, or you can use it to aggregate an entire cluster into a single virtualized node that would completely eliminate the need for traditional cluster management tools.

I was thinking about this a little while ago when I posted VMotion/Live Migration is not an HA feature.

Maybe we’ll see cache-coherent shared-memory virtual infrastructures running over InfiniBand, removing the network overhead that was pointed to as a problem by Rational Survivability.

It started out as a sidenote, but it really got me thinking about the big picture. Why isn’t this a direction we’re seeing existing virtualization vendors moving in, vendors who currently embrace the partitioning approach? Storage vendors have historically worked from the idea that true virtualization involves both aggregation and partitioning. It’s not enough to simply present disks to multiple hosts. You aggregate them into storage pools, and then you carve up LUNs and present them to your storage network. Why aren’t we trying to make compute cycles commoditized for generalized workloads, instead of just specific programs written for message-passing interfaces?

Vendors have heavy investments in distributed infrastructures, using tools like VMotion and DRS to balance resource utilization and maximize consolidation ratios. But is this really the optimal approach to this problem? What if you didn’t need to dynamically balance workloads because the hypervisor’s SMP scheduler would do it automatically on an enormous aggregated system? For day-to-day operations (as opposed to offsite migrations, where VMotion can still be rather useful), what if you were able to move virtual machines across an InfiniBand fabric as a simple in-memory copy, rather than sending the entire contents of a virtual machine’s memory over the network? What if all of your virtual page sharing was completely coherent across your virtualized compute grid, and you really could have one single OS instance in memory running your entire infrastructure?

Certainly there’s a lot of complications and a lot of engineering in this approach. First, of course, is resiliency and failure isolation: how do you make sure that a single server failure doesn’t bring down every OS instance on the grid, which happen to be running tasks on that system’s CPUs? (There’s checkpointing approaches for existing large-scale SMP systems, which could probably be applied to the vSMP approach as well; however, this is pretty academic discussion, and I’m not going to pretend to know how viable it is.) With resiliency in mind, what’s the best way of allocating and distributing resources so that a minimal amount of recovery has to occur in the event of a failure? It’s not useful to recover in this way if it takes longer than a regular clean boot.

This kind of engineering will take a very long time, but I think it’s inevitable. Virtualization vendors have gotten the host resource partitioning part down to the point where I don’t know if anything new can even happen in that space, but there’s a lot more exciting things that can happen once the aggregation piece is layered underneath the hypervisor as we know it today.

I’m sorry. I know you probably paid a lot for that license, but if your infrastructure is relying on a machine’s ability to transition between VM hosts without rebooting as the crux of your high availability plan, you might want to reconsider.

Yesterday, Rational Survivability (a great all-over-the-place IT blog) had a post titled The Emotion of VMotion. It didn’t occur to me before reading this that my own previous search for a hypervisor that would do live migration was working directly against my own beliefs that uptime should only matter for services. Essentially, the infrastructure should be designed so that a single server down doesn’t contribute to the loss of availability.

That being said, live migration is a neat idea, and eventually it’s going to get to the point that it’s nearly instantaneous. When that happens, failovers will be next to invisible. Maybe we’ll have to reevaluate our approach in that case.

Until then, I read posts from people trying to rely on it to keep their infrastructures up and I worry that their approach is flawed.

Please, build your services for reliability, not just the underlying systems.

Now, I need to preface this by saying that I’m not missing the point of Matt’s post. There’s a lot of administrators out there who do treat live migration as a panacea for whatever ails your reliability problems. Anyone who has attempted to design real high-availability infrastructures is very aware that application-level clustering is more robust and typically more reliable than OS-level clustering, which is more robust than hypervisor-level clustering. But these features don’t compete with each other. They each function as a different piece of the datacenter puzzle. And as Matt implies, the cost savings aren’t right for everyone — but they are right for some people.

Absolutely, without a doubt, clustered services are a wonderful, great idea — that’s why people have been using them for decades, and continue to use them. And even though VMotion makes it very easy to add some server-level resiliency to any host or service, the application-level clusters are becoming much easier to configure and maintain at the same time, thanks to great configuration management tools like Puppet, Chef, and Cfengine.

But the big picture is an entire ecosystem around which VMotion thrives. The big cost driver for virtualization in large datacenter environments is consolidation, and being able to run multiple workloads on the same piece of physical hardware is only the first step. Consolidation ratios are improved substantially when you can transparently load-balance workloads in terms of network traffic, compute power and disk I/O — you don’t have to worry about a single bottleneck breaking your carefully-designed system. In addition to the raw server consolidation gains, you substantially save on engineering power, as there’s a lot less manual labor required to design a viable virtualized infrastructure, and a lot less things go wrong if you get it wrong. And if you require compute capacity on demand — say that the majority of your processing occurs during normal business hours and your servers stay mostly idle afterwards — a solution like DRS can actually completely power down your unused VMware hosts until your compute capacity is needed again.

Sure, this isn’t appropriate for everyone. In a pie-in-the-sky IT infrastructure, grid services would provide uniform access to compute capacity and storage on demand using commodity hardware, like Google or Facebook or other players who rely heavily on things like Hadoop or MapReduce in order to scale their operations. But for most real businesses, which have a real investment in commercial off-the-shelf software like databases, ERP systems, CRM and other necessities, we need hypervisors to abstract away the problem and do the work that the COTS vendors won’t, even if the result isn’t as elegant as it should be. And I’m sure that as the hypervisor marketplace matures and consolidates, VMware, Citrix, Microsoft, Red Hat and other vendors will begin to do things with their platforms that we haven’t even thought of yet. Maybe we’ll see cache-coherent shared-memory virtual infrastructures running over InfiniBand, removing the network overhead that was pointed to as a problem by Rational Survivability. The possibilities are endless.

It seems like in this instance, Matt is railing more against the idea of boot-from-SAN than he is about VMotion himself, as boot-from-SAN is another way of solving the same problem — it adds resiliency against hardware failure, but not a ton else. In various ways, he’s right: if you ignore maintenance of your systems documentation and proper server rebuild procedures in favor of a magical black box, your environment will become an unmaintainable mess as a result. It’s the same argument that Luke Kanies has been making about using Puppet or other configuration management systems versus golden master images. In this respect, I think Matt is right to want to know his systems well enough to rebuild them from scratch. It also makes upgrades and other migrations much simpler and smoother.

But every tool is just that: a tool. And they should be used as tools, and evaluated in terms of their effectiveness as a tool. You shouldn’t throw away a perfectly good tool because it doesn’t live up to the hype you were promised. You should use it if it delivers a real return on investment.