Technical interviews are hard. Really, really hard. This is why a lot of big corporations continue to hire IT employees on a six-month contract, followed by an offer for continued employment if they work out: many of them don’t have adequate confidence in their hiring processes.

Deciding who to hire and who not to hire is one of the most difficult parts of trying to run a team, and it’s even harder in a profession where people overstate, exaggerate and lie in order to get through an incompetent HR-driven resume-screening process. It’s a zero-sum game of escalation: jobless workers fire scattershot resumes to all kinds of positions they aren’t qualified for, and large corporations are left with little choice but to do the same thing we as IT people do: apply some remarkably naive filtering to stop the spam. This generally consists of awful keyword filters.

By now, I’ve sat in on, and helped conduct, a ton of technical interviews. Whereas most IT organizations will have a manager or two interview a candidate, along with the head of the department, my group decided a long time ago that whenever we had a position to fill, we would fill that position by conducting a group interview — the candidate would interview with the entire team they would be working with. While this is very time-consuming, and most technical people hate meetings, everyone in our group was able to see the importance of getting the job done right the first time. Like most professions, it’s often more harmful to have someone who does an IT job incorrectly than it is not to hire anyone at all.

I’m not going to zero in on things that occur above IT’s head, like HR’s screening and review processes; there’s nothing that most technical people can do about that besides complain. Instead, I’m going to focus on the technical side of the interviewing process, which lots of companies get just as wrong. I’m not going to attempt to paraphrase the entirety of Joel Spolsky’s famous guide to interviewing, which has some great ideas and some ideas that I don’t always agree with. I will, however, rant for a very long time.

In understanding why I think so many people get the interviewing process wrong, you should familiarize yourself with the concept of illusory superiority. In particular, read the bit about the Kruger and Dunning experiments:

All groups put themselves above average. This meant that the lowest-scoring group (the bottom 25%) showed a very large illusory superiority. Although their test scores were in the 12.5^th percentile on average, they estimated themselves to be in the 62^nd. Kruger and Dunning explained that those who were worst at the tasks were also worst at recognising skill in those tasks. This was supported by the fact that, given training, the worst subjects improved their estimate of their rank as well as getting better at the tasks.^[2]

Curiously, I’m not bringing this up because I think most candidates think they’re much better than they are. I’m bringing this up because a lot of technical interviewers think they’re really clever, and this trait gets them into trouble. What gets them into even more trouble is their eagerness to prove it.

Don’t be a jerk.

This was last, but I just put it first because it’s probably the single biggest issue I’ve seen with most engineer-driven technical interviews.

Technical professions, particularly programming and systems and network administration/engineering jobs, have this really cool personal habit: whenever someone finds themselves in a new position, they spend a couple of months cursing the stupid decisions made by the idiots that came before them. I’ve been guilty of it before, and I’m still guilty of it. Sometimes they’re right, and sometimes they’re wrong.

But most technical people hiring people to work with them seem to be very insecure, and very afraid of people saying the same thing about their work. As a result, they make a habit of being as rude and condescending to the candidate as they can in order to prove their technical prowess, and establish themselves as alpha wolf before the person ever gets hired. This is a good way to drive great candidates away.

As an interviewer, you have to be very perceptive of your tone, and very open to accepting different solutions to a problem. As a manager, especially one conducting a group interview, you have to know where to step in and tell your people when to knock it off.

This isn’t to say that putting pressure on a candidate is a bad thing. I’m not even saying that being a jerk to an applicant is always a bad thing. I’ve known lots of people with other companies who would interview for heavily customer-facing positions, whether helpdesk or directorship positions, and intentionally try to get a rise out of the interviewee. In these cases, if the employee overreacts or blows up, it’s a pretty good indicator that they don’t have the professional rapport and demeanor to be a good choice for the position, because if they can’t take it from the interviewer, they probably can’t take it from the executives above them when something goes wrong.

The ability to memorize useless crap doesn’t make somebody valuable.

If you’re reading this post, you almost certainly know this, so I’m not going to dwell on it: asking someone what the “-q” option to GNU grep does simply isn’t a good interview question at any level. Neither is asking about some random detail of some random programming API, or asking about some Gang of Four design pattern that nobody’s ever consciously related to that name.

You know how it was always a lot harder to BS the right answer on an essay question than on a multiple choice question? That’s because it requires you to actually demonstrate comprehension and understanding of the topic and a whole. Beyond the initial screening process, there’s very little value to questions which have a simple right or wrong answer.

Your questions probably aren’t as clear as you think they are.

Someone was ranting about a candidate once that did horribly in their interview. He asked the candidate to guess the state he was born in. The candidate asked, “Tennessee?” to which the interviewer replied that the candidate should think bigger. The candidate asked, “Texas?”

The interviewer was flabbergasted. There were so many clever ways to approach the problem! This guy was doing an O(n) comparison! If he was pragmatic, he could have tried doing a binary search, dividing the search area into regions. If he was a smart aleck, and had a different idea of what the interviewer meant by the word “state,” he could have responded with, “Naked?”

But the problem was one of connotation: that simple, innocuous word “guess” was wrong, and the question wasn’t half as clear as he thought it was. A better way to phrase the scenario might have been something like: “I was born in a certain state, and I want you to figure out what it is. To do this, you’re going to ask me a series of questions; I will answer ‘yes’ or ‘no’ to each question.”

“But Jeff,” you may say, “this takes all of the imagination out of the interview process.” I disagree. I think that if your problem ceases to become difficult once you’ve explained the problem clearly, the issue might just be with the questions you’re asking. In most cases, a genuinely complex question with a number of genuinely complex answers will yield much more fruitful results for you than asking vague riddles.

It’s not merely a question of open-mindedness. The questioning methodology exposes a number of cultural biases which can be exclusionary to candidates who are completely qualified for the position otherwise.

Complex questions aren’t always relevant questions.

Large software companies like Google and Microsoft have attracted a lot of attention over the years by having some interview questions that are, by traditional standards, completely outlandish. Some of these questions have included:

• How many piano tuners are there in the entire world?
• Why are manhole covers round?
• You have to get from point A to point B. You don’t know if you can get there. What would you do?

What doesn’t get the attention are questions like these, which are all for different positions:

• What’s a creative way of marketing Google’s brand name and product?
• How would you boost the GMail subscription base?
• What is multithreaded programming? What is a deadlock?

Even at Google and Microsoft, the interview process is dry. They still vet the candidates on the basis of their technical merits and their applicability to the position. The riddle-type questions are ancillary to the other types of questioning, and only really apply to people who really need to be delivering clever solutions to problems as part of their day-to-day job responsibilities — project managers, architects, lead programmers and the like.

If there’s anything we’ve learned from Steve McConnell, Andy Hunt, Dave Thomas and the like, it’s that there’s a fine balance that needs to be struck between cleverness and clarity, and most of the time, that balance leans much more towards clarity than cleverness. Cleverness in technical fields has an awful tendency to be identical to obtuseness, and you should apply cleverness really sparingly where it will actually help you out.

I’ll close out this section with a quote from an ex-Google employee, from Michael Arrington’s TechCrunch article from January:

I left Microsoft to work for Google in 2005. I stayed 10 months. I was demoralized. I shouldn’t have ever taken that job. I was disenchanted the whole time, and yes, like you, my regret over the poor bargain I’d made affected my performance.

As I was saying. Google actually celebrates its hiring process, as if its ruthless inefficiency and interminable duration were a sure proof of thoroughness, a badge of honor. Perhaps it is thorough. But I would be willing to wager that Microsoft’s hiring process, which takes a fraction of the time,  does not result in a lower-skilled workforce or result in a higher rate of attrition. And let me say this: if Larry Page is still reviewing resumes, shareholders should organize a rebellion. That is a scandalous waste of time for someone at that level, and the fact that it’s “quirky” is no mitigation.

The person you’re interviewing would like to do a job.

They may even have a job already. They may be busy people who don’t have the time or inclination to jump through needless hoops.

It’s important to make sure that the person you’re interviewing is a good fit for the team. This is why my team will almost always conduct group interviews when filling a position, and we all take the time to ask our own personal questions to try to get a feel for the interviewee’s personality. Some people, however, take the interpersonal aspect a bit too far.

Somebody on a thread I was following a couple of weeks ago was posting about how their interviewing process involved the candidate burning an 80 minute CD of their favorite music, watching the movie Idiocracy and being prepared to discuss it at the interview. Job interviews aren’t high school book reports or college entrance exams. The people applying for positions have jobs and families and other obligations that prevent them from dumping endless hours on interview prerequisites. Ultimately, if you’re going to test your candidates or give them assignments outside of the interview room, keep it really short or make sure it’s restricted to people who are already in a tight race for the job. It’s just a waste of everybody’s time otherwise.

Management is about using the right metrics.

In most fields, management is all about numbers. Units are shipped, dollars are earned, money is saved. In most organizations, IT is a cost center and an internal customer service organization. That makes it really difficult to figure out metrics. Lots of people in IT hate the idea of metrics as a whole, because in many cases they emphasize taking the quick way out instead of finding the right solution to a problem. This leads to systemic issues that are not easy to undo later. Metrics are only valuable if they’re both relevant and useful. The length of time spent working on a ticket is not a useful metric: some tickets may appear to be simple issues but may reflect systemic problems. The number of tickets closed in a reporting period is not a useful metric either, because the issues may not be resolved in a satisfactory way.

The interviewing process is the same way: if you get it wrong, you will introduce systemic issues. And in nearly all cases, it’s better to be using no metrics at all than to be using the wrong metrics.

Last week, Valleywag wrote of Google:

Google strives to hire “the world’s best engineers,”and has crafted an “interminable” interview process dotted with puzzles and brainteasers to do so. One little problem: the process tends to give the worst scores to the best future employees.

That’s according to Peter Norvig (pictured), Google’s director of research, former Google director of search quality and former head of the Computational Sciences division at the NASA Ames research center. Here’s what Norvig tells Peter Seibel in a Q&A in the new book Coders at Work (emphasis added):

One of the interesting things we’ve found, when trying to predict how well somebody we’ve hired is going to perform when we evaluate them a year or two later, is one of the best indicators of success within the company was getting the worst possible score on one of your interviews. We rank people from one to four, and if you got a one on one of your interviews, that was a really good indicator of success.

Small suggestion: Maybe Google can take these genius employees and have them, hmmm, we dunno, debug the frickin’ broken interview process. Those who demanded they be hired should probably also be enlisted in the debugging effort. Writes Norvig:

Ninety-nine percent of the people who got a one in one of their interviews we didn’t hire. But the rest of them, in order for us to hire them somebody else had to be so passionate that they pounded on the table and said, “I have to hire this person because I see something in him…”

That last line is the key, and it leads me to my last point:

The interview process is completely subjective, even when you think it isn’t.

Especially when you think it isn’t.

There is no magic mathematical formula that will tell you whether or not it’s a good idea to hire somebody, and one of the most dangerous things that you can do is convince yourself that there’s a rubric you should be following in terms of scoring your candidates. Google manages to hire great employees, but they do so exactly because they ignore their own metrics and certain staff members pound on the table to give people a chance. Interviewing takes, above all else, a critical eye, and no criteria on paper will ever replace that.

I’d love to hear some discussion of your own experiences on either side of the table.

Google strives to hire “the world’s best engineers,”and has crafted an “interminable” interview process dotted with puzzles and brainteasers to do so. One little problem: the process tends to give the worst scores to the best future employees.

That’s according to Peter Norvig (pictured), Google’s director of research, former Google director of search quality and former head of the Computational Sciences division at the NASA Ames research center. Here’s what Norvig tells Peter Seibel in a Q&A in the new book Coders at Work (emphasis added):

One of the interesting things we’ve found, when trying to predict how well somebody we’ve hired is going to perform when we evaluate them a year or two later, is one of the best indicators of success within the company was getting the worst possible score on one of your interviews. We rank people from one to four, and if you got a one on one of your interviews, that was a really good indicator of success.

Small suggestion: Maybe Google can take these genius employees and have them, hmmm, we dunno, debug the frickin’ broken interview process. Those who demanded they be hired should probably also be enlisted in the debugging effort. Writes Norvig:

Ninety-nine percent of the people who got a one in one of their interviews we didn’t hire. But the rest of them, in order for us to hire them somebody else had to be so passionate that they pounded on the table and said, “I have to hire this person because I see something in him…”

Unfortunately, Google’s had already done most of its hiring/rejecting and is now has been in layoff mode for much of this year. But, hey, there’s always the next bubble.

UPDATE: A Goolge spokesperson disputed that the company was “in layoff mode,” as we wrote, and stated: “To the contrary, we have been very explicit… that we are stepping our rate of hiring.” Indeed, CEO Eric Schmidt stated in a discussion of Q3 results that “we’re going to invest in people. We’re already stepping up our hiring.” That’s in contrast to earlier this year, when Google had three rounds of layoffs from January through the end of March.

Having never interviewed at Google (though having had the chance), and having no idea how their interviews are really conducted these days, I can’t say with Valleywag’s certainty that the entire interview process is broken. However, this is a great example of a completely broken metric. Obviously the score isn’t used to determine whether the person should be hired or not, so why use it at all? Either get rid of the metric, and admit that the interview process is what it is  — completely subjective, all the time (this isn’t a bad thing) — or come up with a new metric. If someone sees something special in a candidate, they’ll be arguing just as hard for them without a nonsense number being attached that doesn’t represent anything real.