Home › Monthly Archives › October 2009

Google’s Broken Hiring Process

Jeff October 31, 2009 Sysadmin 1

Right as I was literally right in the middle of writing up a posting on conducting technical interviews, usually entertaining but useless magazine Valleywag has a really interesting article on Google’s broken hiring process:

Google strives to hire “the world’s best engineers,”and has crafted an “interminable” interview process dotted with puzzles and brainteasers to do so. One little problem: the process tends to give the worst scores to the best future employees.

That’s according to Peter Norvig (pictured), Google’s director of research, former Google director of search quality and former head of the Computational Sciences division at the NASA Ames research center. Here’s what Norvig tells Peter Seibel in a Q&A in the new book Coders at Work (emphasis added):

One of the interesting things we’ve found, when trying to predict how well somebody we’ve hired is going to perform when we evaluate them a year or two later, is one of the best indicators of success within the company was getting the worst possible score on one of your interviews. We rank people from one to four, and if you got a one on one of your interviews, that was a really good indicator of success.

Small suggestion: Maybe Google can take these genius employees and have them, hmmm, we dunno, debug the frickin’ broken interview process. Those who demanded they be hired should probably also be enlisted in the debugging effort. Writes Norvig:

Ninety-nine percent of the people who got a one in one of their interviews we didn’t hire. But the rest of them, in order for us to hire them somebody else had to be so passionate that they pounded on the table and said, “I have to hire this person because I see something in him…”

Unfortunately, Google’s had already done most of its hiring/rejecting and is now has been in layoff mode for much of this year. But, hey, there’s always the next bubble.

UPDATE: A Goolge spokesperson disputed that the company was “in layoff mode,” as we wrote, and stated: “To the contrary, we have been very explicit… that we are stepping our rate of hiring.” Indeed, CEO Eric Schmidt stated in a discussion of Q3 results that “we’re going to invest in people. We’re already stepping up our hiring.” That’s in contrast to earlier this year, when Google had three rounds of layoffs from January through the end of March.

Having never interviewed at Google (though having had the chance), and having no idea how their interviews are really conducted these days, I can’t say with Valleywag’s certainty that the entire interview process is broken. However, this is a great example of a completely broken metric. Obviously the score isn’t used to determine whether the person should be hired or not, so why use it at all? Either get rid of the metric, and admit that the interview process is what it is  — completely subjective, all the time (this isn’t a bad thing) — or come up with a new metric. If someone sees something special in a candidate, they’ll be arguing just as hard for them without a nonsense number being attached that doesn’t represent anything real.

awesome WM Appreciation Post

Jeff October 29, 2009 Sysadmin 3

Multiple workspaces always made me kind of sloppy as a user when they were supposed to make me more organized. I would open up dozens of tabbed terminals and lose track of some of them. Eventually I spent enough time looking for windows I should have had right in front of me, and closing windows that shouldn’t be closed, that I figured there had to be a more productive way out there.

I’ve flirted with tiling window managers before. Ion, wmii and xmonad always left a bad taste in my mouth. They were inflexible and annoying and difficult to use with any workflow that wasn’t 100% centered around the terminal. God help you if a program needed to open a dialog window.

awesome is a lightweight tiling window manager with very good support for floating windows, without having to mess with separate layers for tiled or floating windows. It has full support for multiple displays through XRandR, it seems to be a lot faster than Xmonad and other tiling WMs, and it’s freaking tiny — the full source, including wallpapers, is just over 300k. It is very flexible with tagging and per-screen workspaces, support for extensive configuration options through Lua, and a great user community. It also has a number of really nice mouse-driven features, like resizing your layout by holding the meta key and right-dragging. A lot of tiling WMs drive me nuts because you have to memorize 45 keyboard shortcuts in order to use them effectively.

I’ve been a KDE user since 2002, with occasional flirtations with other desktop environments. awesome made me switch straight-out. At the moment, I’m running a pared-down Xfce setup, with xfwm4 swapped out for awesome. The other utilities, like the Xfce session manager and panel, are still useful. (The one thing that drives me nuts is that awesome takes over my systray. I would really like it in my Xfce panel.)

Give it a shot. It’s worth the learning curve.

Obligatory screenshot is forthcoming.

El Reg: Cloud storage: It’s strictly for airheads

Jeff October 22, 2009 Sysadmin 0

The Register has an interesting take on the Sidekick/Microsoft-but-really-Sun-and-Oracle-but-really-really-Microsoft debacle. The good bits:

…

The service wasn’t run according to Microsoft in-house standards at all, but users would not know this. They wouldn’t know that the Mobile brand and the Microsoft brand were just wrappers around a third-party service.

In the cloud it’s not just data that vanishes, it’s the ability to verify what is actually happening to it. Brands are surface things in the cloud with no guarantee at all that you can trust what goes on beyond them inside the cloud or verify it either.

…

Buy an online backup service from Mozy, Carbonite or a cloud storage service from Nirvanix, Google or Amazon, or from any of the myriad other local, regional and national services springing up, and you have no idea at all of the data centre infrastructure, products and processes involved. You just throw your data in and hope that they look after it properly. You can’t verify that they do. It’s a matter of blind faith.

…

The good news is that this isn’t rocket science. It’s what trade associations of professional service providers do already. They self-regulate by certifying members behave according to standards and carry sufficient insurance for the risks they run if they make mistakes. Look at dentists, lawyers, civil engineers or any other trade professional person or business – they all sport the distinction of their professional body and its standards.

What we need is a code of practice backed up by membership of a Cloud Storage Providers’ Association with certification for members. No business should contract for cloud storage services from suppliers who are not members of such a CSPA body, and the CSPA should rigorously enforce the creation of a minimum acceptable standard of service; and also rigorously police its members and throw out suppliers who fail to meet the standard.

Every cloud storage provider with a belief that they are an honest business providing a good and solid service should see the sense of this, and start making moves for a CSPA-type body to come into being. Without it cloud storage services will be offered by cowboys and incompetents, who lose users data, as SwissDisk, T-Mobile and Microsoft have.

Cloud storage needs open standards for the custodianship of users’ data, and only a reputable trade body can provide it. What is the industry waiting for? Do we need another SwissDisk, another Sidekick before it will act? ®

Now, in this regard, I don’t really think that a standards group for cloud storage in particular is necessarily the right approach, versus something more generalized that could apply to all outsourced cloud IT vendors. I don’t believe that storage, in this regard, is any different than any other IT service. After all, while your data can certainly be lost by a storage incident, whether local, remote or somewhere in between, it can also be lost by a logical failure of the IT software infrastructure leveraging that storage. That’s what happened in this Danger business with T-Mobile: the logical failure apparently occurred (if The Register’s writeup is to be believed) when a server in the cluster threw up and trashed the data.

Danger didn’t have appropriate backups. (Oops. More on online vs. offline backups, and physical vs. logical failures, in another post.) In my experience, there’s a certain threshold of reliability and competence where once your physical infrastructure is robust enough, your potential for logical failures far outweighs your potential for physical failures. (Most backup system failures are really logical failures.) It’s this mismatch that makes business continuity planning so difficult.

But they’re right that the industry needs accountability, and it’s probably going to take a major shakeup for that to happen. Right now, there’s just too many vendors, and they’re all too young for us as IT practitioners to figure out which ones are reliable enough for business needs, and which ones aren’t. Like with any fledgling industry, there will be new technologies, there will be acquisitions, and then there will be vendors who produce an enterprise-ready product.

For the time being, we refer to the ages-old axiom: if you want something done right, you’ve got to do it yourself.

Bottom-Up Virtualization

Jeff October 22, 2009 Sysadmin 0

Working in the life sciences industry, I often deal with users who have requests that might be considered strange in other fields. For example, my organization has users asking for systems with 2 terabytes of RAM. We have other users asking for systems with 12 terabytes of RAM. To a normal system administrator who doesn’t run OLTP systems for a bank or brokerage, where you might find huge-memory database systems, this technical requirement seems silly. However, for gene sequence assembly and analysis, this much memory is really a requirement with longer sequences. Short read assemblers like Velvet can chew through this in as much time as it takes the system to allocate all that memory.

You don’t have to be on the forefront of bleeding-edge server technology to know that x86 systems with 12 terabytes of RAM simply don’t exist. With RAM density as it is right now, there’s simply no way to fit that many DIMMs on a board. However, some inventive software steps up to the plate.

We’ve been meeting with a company called ScaleMP. ScaleMP is, in strict terms, a virtualization software vendor. However, unlike companies like VMware, ScaleMP specializes in using virtual machine monitors to aggregate CPU and memory resources among a number of InfiniBand-connected hosts, presenting a logical system with all of the combined CPU and memory resources of the aggregated physical machines. Through their black magic technology, they apparently do this without substantial overhead to the host, and the resulting virtual machine performs on par with a parallelized MPI solution utilizing operating systems running atop bare metal on the physical nodes. The difference, of course, is that you have a very large coherent block of memory to work with. If you’re familiar with Isilon’s storage architecture, the pattern should look familiar.

There’s an article from HPCwire written by Shai Fultheim, ScaleMP’s CEO, that sums up this approach a lot better than I could hope to. But the ten-cent version is that you can use this to virtualize compute, memory and I/O resources to present a very large system for single tasks that require tons of memory and extremely fast parallelism, or you can use it to aggregate an entire cluster into a single virtualized node that would completely eliminate the need for traditional cluster management tools.

I was thinking about this a little while ago when I posted VMotion/Live Migration is not an HA feature.

Maybe we’ll see cache-coherent shared-memory virtual infrastructures running over InfiniBand, removing the network overhead that was pointed to as a problem by Rational Survivability.

It started out as a sidenote, but it really got me thinking about the big picture. Why isn’t this a direction we’re seeing existing virtualization vendors moving in, vendors who currently embrace the partitioning approach? Storage vendors have historically worked from the idea that true virtualization involves both aggregation and partitioning. It’s not enough to simply present disks to multiple hosts. You aggregate them into storage pools, and then you carve up LUNs and present them to your storage network. Why aren’t we trying to make compute cycles commoditized for generalized workloads, instead of just specific programs written for message-passing interfaces?

Vendors have heavy investments in distributed infrastructures, using tools like VMotion and DRS to balance resource utilization and maximize consolidation ratios. But is this really the optimal approach to this problem? What if you didn’t need to dynamically balance workloads because the hypervisor’s SMP scheduler would do it automatically on an enormous aggregated system? For day-to-day operations (as opposed to offsite migrations, where VMotion can still be rather useful), what if you were able to move virtual machines across an InfiniBand fabric as a simple in-memory copy, rather than sending the entire contents of a virtual machine’s memory over the network? What if all of your virtual page sharing was completely coherent across your virtualized compute grid, and you really could have one single OS instance in memory running your entire infrastructure?

Certainly there’s a lot of complications and a lot of engineering in this approach. First, of course, is resiliency and failure isolation: how do you make sure that a single server failure doesn’t bring down every OS instance on the grid, which happen to be running tasks on that system’s CPUs? (There’s checkpointing approaches for existing large-scale SMP systems, which could probably be applied to the vSMP approach as well; however, this is pretty academic discussion, and I’m not going to pretend to know how viable it is.) With resiliency in mind, what’s the best way of allocating and distributing resources so that a minimal amount of recovery has to occur in the event of a failure? It’s not useful to recover in this way if it takes longer than a regular clean boot.

This kind of engineering will take a very long time, but I think it’s inevitable. Virtualization vendors have gotten the host resource partitioning part down to the point where I don’t know if anything new can even happen in that space, but there’s a lot more exciting things that can happen once the aggregation piece is layered underneath the hypervisor as we know it today.

New Malware Re-Writes Online Bank Statements to Cover Fraud

Jeff October 20, 2009 Sysadmin 1

A few weeks behind the ball on this one too, but I ran across it in a security discussion and it was way too cool (in a perverse sense) not to share:

From Wired.com:

New Malware Re-Writes Online Bank Statements to Cover Fraud

New malware being used by cybercrooks does more than let hackers loot a bank account; it hides evidence of a victim’s dwindling balance by rewriting online bank statements on the fly, according to a new report.

The sophisticated hack uses a Trojan horse program installed on the victim’s machine that alters html coding before it’s displayed in the user’s browser, to either erase evidence of a money transfer transaction entirely from a bank statement, or alter the amount of money transfers and balances.

The ruse buys the crooks time before a victim discovers the fraud, though won’t work if a victim uses an uninfected machine to check his or her bank balance.

The novel technique was employed in August by a gang who targeted customers of leading German banks and stole Euro 300,000 in three weeks, according to Yuval Ben-Itzhak, chief technology officer of computer security firm Finjan.

“The Trojan is hooked into your browser and dynamically modifies the text in the html,” Ben-Itzhak says. “It’s a very sophisticated technique.”

The information appears in a cybercrime intelligence report (.pdf) written by Finjan’s Malicious Code Research Center.

The victims’ computers are infected with the Trojan, known as URLZone, after visiting compromised legitimate web sites or rogue sites set up by the hackers.

Once a victim is infected, the malware grabs the consumer’s log in credentials to their bank account, then contacts a control center hosted on a machine in Ukraine for further instructions. The control center tells the Trojan how much money to wire transfer, and where to send it. To avoid tripping a bank’s automated anti-fraud detectors, the malware will withdraw random amounts, and check to make sure the withdrawal doesn’t exceed the victim’s balance.

The money gets transferred to the legitimate accounts of unsuspecting money mules who’ve been recruited online for work-at-home gigs, never suspecting that the money they’re allowing to flow through their account is being laundered. The mule transfers the money to the crook’s chosen account. The cyber gang Finjan tracked used each mule only twice, to avoid fraud pattern detection.

“They instruct the Trojan that the next time you log into your online banking account, they actually modify and change the statement you see there,” says Ben-Itzhak. “If you don’t know it, you won’t report it to the bank so they have more time to cash out.”

The researchers were able to capture screen shots showing the rogue bank statements in action, disguising, for example, a transfer of Euro 8,576.31 as Euro 53,94.

The researchers also found statistics in the command tool showing that out of 90,000 visitors to the gang’s rogue and compromised websites, 6,400 were infected with the URLZone trojan. Most of the attacks Finjan observed affected people using Internet Explorer browsers, but Ben-Itzhak says other browsers are vulnerable too.

Finjan provided law enforcement officials with details about the gang’s activities and says the hosting company for the Ukraine server has since suspended the domain for the command and control center. But Finjan estimates that a gang using the scheme unimpeded could rake in about $7.3 million annually.

“The example we found relates to German banks,” Ben-Itzhak says. “But we believe this will increase to other countries.”

Book Review: Network Warrior

Jeff October 19, 2009 Sysadmin 0

Network Warrior, by Gary A. Donahue

Network Warrior book cover

This book bills itself as “everything you need to know what wasn’t on the CCNA exam.”

I never took the CCNA exam (though I plan on it soon). In fact, when I ordered this book, I really didn’t know very much about networking at all. I was of course familiar with the basic concepts from the sysadmin side – IP addresses, broadcast domains, gateways, DNS, stateful and stateless protocols, ARP and all that jazz – but I really didn’t understand much about how it worked from the side of the network infrastructure. I was pretty ignorant to STP, routing protocols, the IOS CLI, and other facets of a network engineer’s day-to-day job. I picked it up because I was in the middle of designing a development environment based on ESX and iSCSI and it helps to optimize network storage protocols if you actually understand the network in addition to the storage protocol. So, a few days later, here I was.

Contrary to what the tagline might imply, this book does not make any assumptions about your Cisco knowledge. Not only do you not need to be a CCNA, you really don’t need much knowledge at all of CatOS or IOS (both are covered) in order to absorb most of the book. (I did find myself flipping to the glossary from time to time to look up a term that hadn’t been introduced yet. Every term I wanted to look up was in the glossary.) The book makes few assumptions, and tends to stick with theory more than commands. The CatOS bits are a nice touch, though.

Instead, what’s delivered is a concise, easy read that’s probably of more use to people just looking into a career of network administration than those who are already entrenched in it. This isn’t to say it’s a niche book, or a book geared towards sysadmins, or that it’s not useful for a moderately experienced network administrator. Rather, the material is incredibly accessible to anyone of a moderate technical inclination. Donahue is extremely knowledgeable and his tone is technical and understandable, but not overwhelming, throughout the entire book.

The book focuses on breadth rather than depth; if you’re planning on studying for a CCNA, I recommend picking this up along with a good CCNA study guide (I used McGraw-Hill’s) and the CCNA Portable Command Guide. But what is covered in the book’s breadth both is useful and makes an impression: his section on dealing with management in a crisis is every bit as useful as the bits on QoS and VoIP or his subnetting tips.

While the book is somewhat weak on routing, paying only lip service to protocols other than EIGRP and OSPF, it can be easily supplemented with a book that’s more detail-oriented. This book prompted me to pick up Halabi’s second edition Internet Routing Architectures. I think my CCNA practice book covered the basics of other routing protocols pretty well.

If Limoncelli’s “The Practice of System and Network Administration” is the canonical tome on what it means to be a sysadmin, this ought to be the canonical tome on administering Cisco networks.

Rating: 5/5

SaaS: Who needs release management?

Jeff October 19, 2009 Sysadmin 0

Google shows us the importance of release management:

Google Docs users are having trouble printing, exporting, and importing files, following a recent update to the alleged Microsoft killer.

As reported by the IDG News Service, a Google employee has acknowledged the problems with a pair of posts to the company’s Google Docs Help discussion forum. “We’re currently looking into this issue,” said a Google person identifying herself only as Marie. “Thanks a lot for your patience. I’ll be providing any relevant updates.”

…

Google acknowledged problems with a suite a day after it battled serious delays in the delivery of business email on its Postini message security and anti-spam system, and over the past several months, Google has taken more than a little heat over outages on its Gmail and News services. ®

What businesses are slowly finding is that giving up control over updates and patches isn’t necessarily a good thing.

While SaaS/cloud computing/outsourcing allows businesses to leverage economies of scale, it leads to problems when the vendor’s objectives don’t necessarily align with the customer’s. In this case, an unannounced patch broke the service for a number of businesses.

Software deployment is a difficult thing to manage. IT departments are constantly faced with this problem whether they’re working on server-side or client-side software. But regardless of what’s being deployed, most organizations with heavy investments in IT infrastructure go through great pains to ensure that new rollouts align with business objectives. On the client side, this means that new versions of the software are tested, heavily QAed, and documented before anything is pushed into production. Desktop support should be familiar with the new version of the software, especially when major interface changes occur (a la Office 2003 to 2007). On the server side, it means rigidly enforcing change control practices so that, for example, no email system maintenance occurs on the day that executives are closing a big deal with a major business partner.

Outsourcing critical infrastructure services and applications may gain you the ability to offload your IT support to people who focus on a single core competency, a single set of applications. Theoretically, this should increase your availability, as people specialize in making this one system work as smoothly as possible. This allows amateur maintenance mistakes to be avoided, and the technical staff end up possessing a much greater understanding of the platform they support.

The cloud availability argument is a fallacy because not all availability is equal. Some periods are more important than others. It’s why IT systems have maintenance windows. When you know something especially important is happening from a business perspective, you don’t roll out new changes.

When switching to the cloud, you lose all of your release management capability. It’s the vendor’s responsibility to bring your IT staff up to speed on changes being made to a product so they can be prepared and ready. It’s the vendor’s responsibility to ensure that your services are not touched during periods of peak or critical usage. When possible, it’s the vendor’s responsibility to allow you to push out changes incrementally to groups of users. It’s the vendor’s responsibility to let you back out of an upgrade if something is wrong, no questions asked, and preferably without any manual intervention required.

The vendors all suck at this. They all know how to run a platform. They don’t know how to run someone else’s business, a business that uses technology as a tool instead of a core competency. And from the looks of cloud computing at the end of the year 2009, they don’t care.

Hopefully 2010 looks better. I’ll trust this cloud computing thing a lot more when they start treating businesses like businesses.

VMotion/Live Migration is not an HA feature

Jeff October 19, 2009 Sysadmin 0

I’m a couple of weeks behind the ball here, but I was a bit inspired by this (somewhat controversial) post over at Standalone Sysadmin:

I’m sorry. I know you probably paid a lot for that license, but if your infrastructure is relying on a machine’s ability to transition between VM hosts without rebooting as the crux of your high availability plan, you might want to reconsider.

Yesterday, Rational Survivability (a great all-over-the-place IT blog) had a post titled The Emotion of VMotion. It didn’t occur to me before reading this that my own previous search for a hypervisor that would do live migration was working directly against my own beliefs that uptime should only matter for services. Essentially, the infrastructure should be designed so that a single server down doesn’t contribute to the loss of availability.

That being said, live migration is a neat idea, and eventually it’s going to get to the point that it’s nearly instantaneous. When that happens, failovers will be next to invisible. Maybe we’ll have to reevaluate our approach in that case.

Until then, I read posts from people trying to rely on it to keep their infrastructures up and I worry that their approach is flawed.

Please, build your services for reliability, not just the underlying systems.

Now, I need to preface this by saying that I’m not missing the point of Matt’s post. There’s a lot of administrators out there who do treat live migration as a panacea for whatever ails your reliability problems. Anyone who has attempted to design real high-availability infrastructures is very aware that application-level clustering is more robust and typically more reliable than OS-level clustering, which is more robust than hypervisor-level clustering. But these features don’t compete with each other. They each function as a different piece of the datacenter puzzle. And as Matt implies, the cost savings aren’t right for everyone — but they are right for some people.

Absolutely, without a doubt, clustered services are a wonderful, great idea — that’s why people have been using them for decades, and continue to use them. And even though VMotion makes it very easy to add some server-level resiliency to any host or service, the application-level clusters are becoming much easier to configure and maintain at the same time, thanks to great configuration management tools like Puppet, Chef, and Cfengine.

But the big picture is an entire ecosystem around which VMotion thrives. The big cost driver for virtualization in large datacenter environments is consolidation, and being able to run multiple workloads on the same piece of physical hardware is only the first step. Consolidation ratios are improved substantially when you can transparently load-balance workloads in terms of network traffic, compute power and disk I/O — you don’t have to worry about a single bottleneck breaking your carefully-designed system. In addition to the raw server consolidation gains, you substantially save on engineering power, as there’s a lot less manual labor required to design a viable virtualized infrastructure, and a lot less things go wrong if you get it wrong. And if you require compute capacity on demand — say that the majority of your processing occurs during normal business hours and your servers stay mostly idle afterwards — a solution like DRS can actually completely power down your unused VMware hosts until your compute capacity is needed again.

Sure, this isn’t appropriate for everyone. In a pie-in-the-sky IT infrastructure, grid services would provide uniform access to compute capacity and storage on demand using commodity hardware, like Google or Facebook or other players who rely heavily on things like Hadoop or MapReduce in order to scale their operations. But for most real businesses, which have a real investment in commercial off-the-shelf software like databases, ERP systems, CRM and other necessities, we need hypervisors to abstract away the problem and do the work that the COTS vendors won’t, even if the result isn’t as elegant as it should be. And I’m sure that as the hypervisor marketplace matures and consolidates, VMware, Citrix, Microsoft, Red Hat and other vendors will begin to do things with their platforms that we haven’t even thought of yet. Maybe we’ll see cache-coherent shared-memory virtual infrastructures running over InfiniBand, removing the network overhead that was pointed to as a problem by Rational Survivability. The possibilities are endless.

It seems like in this instance, Matt is railing more against the idea of boot-from-SAN than he is about VMotion himself, as boot-from-SAN is another way of solving the same problem — it adds resiliency against hardware failure, but not a ton else. In various ways, he’s right: if you ignore maintenance of your systems documentation and proper server rebuild procedures in favor of a magical black box, your environment will become an unmaintainable mess as a result. It’s the same argument that Luke Kanies has been making about using Puppet or other configuration management systems versus golden master images. In this respect, I think Matt is right to want to know his systems well enough to rebuild them from scratch. It also makes upgrades and other migrations much simpler and smoother.

But every tool is just that: a tool. And they should be used as tools, and evaluated in terms of their effectiveness as a tool. You shouldn’t throw away a perfectly good tool because it doesn’t live up to the hype you were promised. You should use it if it delivers a real return on investment.

Architects vs. Engineers

Jeff October 15, 2009 Sysadmin 0

From The Lone Sysadmin‘s post, Architects vs. Engineers:

“An architect knows something about everything. An engineer knows everything about one thing.”

-Matthew Frederick, “101 Things I Learned in Architecture School”

This is making me think twice about my job title.

TaoSecurity: “Protect the Data” Where?

Jeff October 11, 2009 Sysadmin 0

From one of my favorite blogs, TaoSecurity:

I forgot to mention another thought in my last post “Protect the Data” from Whom? Intruders are not mindly attacking systems to access data. Intruders direct their efforts toward the sources that are easiest and cheapest to exploit. This produces an interesting corollary.

Once other options have been eliminated, the ultimate point at which data will be attacked will be the point at which it is useful to an authorized user.

For example, if a file is only readable once it has been decrypted in front of a user, that is where the intruder will attack once his other options have been exhausted. This means that the only way to completely “protect data” is to make it unusable. If data is not usable then it doesn’t need to exist, so that means intruders will always be able to access data if they are sufficiently resourced and motivated, as explained in my first post on this subject.

This meshes pretty well with my philosophy on information security — it doesn’t matter how much security you layer onto the server side. For any sufficiently secure system, your weakest point of potential compromise is almost always going to be your clients. Banks and online payment services (such as Paypal) have learned this the hard way. For every breach, no matter how insignificant, there are millions of successful phishing attacks.

So many system administrators forget about the client side of things because it’s not their job. Big mistake.