A Recipe for an Interview

Published in

jocelyngoldfein

8 min readOct 4, 2016

or, Hiring Engineers part 4: Interview Assessment

The tech industry is a bit in flux right now on how to deliver “a good comprehensive interview for a software engineer.”

We jettisoned brainteasers in the 2000’s, and as I write this in 2016, we are jettisoning whiteboard coding interviews for about the same reasons: they don’t seem to actually predict performance on the job. Google says so, and they have more data than anyone. My own experience backs it up: it’s hard to convince yourself you’re measuring ability when candidate performance declines as engineers get more experienced.

Early stage startups have already moved on, but there doesn’t seem to be a consensus pick on a replacement yet. I see seed and series A startups using a mix of take home puzzles, pair programming, laptop exercises, etc. It’s going to take some time for this to work through the system and a new industry consensus to form. In the meantime, my advice to you is to forget the whiteboard and experiment with different types of coding assessments.

But don’t stop with assessing coding — one thing we know does predict performance, and that’s a high quality behavioral interview. Too many startups skip them — we focus on coding, discuss experience, maybe a get-to-know-you lunch, aka “beer test.” But behavior is much different than likability, and it can apply to any number of job skills.

The problem with asking people about their experience is that they know much more about it than you do, and almost anyone can sound good with an expertise asymmetry on their side. The key to a great behavioral interview is to level that playing field. After many years of closely guarding my formula, I’m going to share my secrets.

Pick a functional skill you know well that makes up an important and non-trivial part of the job — let’s call it <X>. Whether <X> represents design skills, detecting bugs before shipping, project management, or conflict resolution: this series of questions will give you insight into the person on the other side of the table. You can use it on engineers, managers, product managers, execs — really, anyone your startup wants to hire.

This is a series of questions; expect to spend 15–25 minutes on it, although you can cut it short at #3 and still get some value.

About how much of <X> have you done?
What does it mean to do <X> well?
What do you do when you do <X>?
Have you made any mistakes doing <X>? Pick one, and tell me the story.
How did you ultimately deal with the situation?
Do you think you could have avoided the mistake?
Would you change the way you do <X> to avoid that mistake in the future?

Question #1: You’ve been doing <X> for a while. About how much of <X> have you personally done?

Good/bad answer: This is just a ranging shot so you have context to gauge their answers. You’re looking for them to have done enough that it’s fair for you to evaluate their expertise. If they haven’t done much, maybe switch gears and ask about something else.

Question #2: What does it mean to do <X> well?

Good answer: values that match yours! Bonus points for being opinionated about this (if you agree with their opinions), bonus points for demonstrating they’ve given this thought. I expect senior candidates to have a point of view, or even to have spent time teaching others how to do it.
Neutral answer (doesn’t help, don’t deduct): elements of <X> you can agree with, but don’t seem like the high priority.
Bad answer: any of the neutral answers as their primary focus. Coming up with an opinion on the spot rather than having thought about it before.
Remember: do NOT feed them the answer to this question. Establishing up front what they think matters is critical to assessing answer quality in later questions in the series.

Question #3: What do you personally do when you do <X>?

Note: Candidates will have a tendency to stay general and talk about people doing <X> in the abstract. With this question, you want to know specifically what they do when they are running a one on one or giving a code review or whatever <X> may be. It’s fine for them to tell you about overall process or other people’s roles, but you must walk away with a clear picture of what part they do and how they do it.
Good answer: They can tell you the brass tacks: concrete details and examples of the work they do when they are doing <X>. If <X> is systems design, that might relate to their process for generating ideas or getting feedback. If it’s conflict resolution, maybe it’s pulling people aside or letting them feel heard. You should end up with a decent picture of what they do personally and what they rely on others for. They need to convince you that what they do sets them up to have a successful outcome.
Mediocre answer: They have decent explanations of what they do, but they struggle to relate it to their answer to question #2 (what it means to succeed.) You want someone who connects their value system to their actions.
Bad answer: They can’t tell you what they do. Either they don’t really do it much, or they don’t really have a system and they’re winging it each time.
Remember: #2 sets you up to ask this one; they should be able to connect the dots between “what I value” and “what I do” with this answer.

Question #4: Have you made any <X> mistakes? Pick one and tell me the story.

Good answer: I like it when people are judgmental. That suggests to me that they’ll learn from mistakes and move forward from them. I also like people who own their mistakes, don’t tell me “I was too trusting of other people who screwed up.” Great candidates relate their mistakes back to their answer to #2 (what is important). You’re looking for a real mistake, not a fake one, but not a mistake so glaring you can’t relate.
Bad answers: “I relied on other people and they screwed up.” “Circumstances were outside my control.” I dislike this because it’s self-exculpating. People who take accountability and are resilient to obstacles would reframe it in terms of the pieces that are under their control. “I knew he hadn’t done something like that before, so I should have been keeping closer track of his tasks.”
Wrong answer: Lots of justification about how it’s not their fault, they couldn’t have known, or they were tricked into taking the fall by someone else. We want to hire people whose top priority when things go wrong is to solve the problem, not to evade blame.
Wrong answer (surprisingly rare!): “I have never made a mistake.” Question 1 established that they’ve done a significant amount of this kind of work. If they have no mistakes to relate, either their self-assessment is off or they’re deliberately being disingenuous and don’t know how to admit failure. Both are grounds for a “no hire” decision.

Question #5: How did you ultimately deal with the situation?

Good answer: They took responsibility, identified the problem in a short period of time, took quick action, made thoughtful attempts to solve the problem, either those attempts worked, or they got help. Bonus points if they were mature about it, have solved similar problems more than once, and the fix went smoothly. Alternatively, bonus points if it was their first time and they can articulate what they learned from a rocky situation.
Neutral answer: Took a long time to solve the problem but eventually acted. Or, couldn’t bring themselves to confront the problem head on, but tried to change things in the background or sweep it under the carpet in a way that actually worked. Another form of neutral answer is “the situation changed so it was no longer an issue.” In this case it’s hard to hold it against them — but there’s a caveat about speed — should they reasonably have acted before the change in circumstances?
Bad answer: Your candidate was passive, and waited around hoping the problem would go away. They can’t articulate clear steps that could have turned the situation around. Never made a tough call.
WRONG ANSWER, INSTA NO-HIRE: Got rid of problem by leaving someone else holding the bag (transferred a difficult employee to another team, walked off a project.) Sometimes a change of scenery can be a good solution when things aren’t working out. The difference between this and “ditched my problem” is the candidate clearly explaining why it was a bad fit one place and a good fit in the other team, and it’s obvious they clearly communicated what they were handing off.

Question #6: Do you think you could have avoided the mistake?

Good answer: “No, and here’s a good reason why.” Or, “Yes, and here’s a good reason why.” In other words, they’ve already thought about it and have a point of view that seems sane. It doesn’t have to be profound.
Bad answer: Has never thought about it and has to think about it for the first time on the spot with you. Also a bad answer if your candidate thinks the mistake was unavoidable and you think it was totally avoidable. Or they say “it was avoidable” because they see that’s the answer you’re looking for, but they can’t articulate why.

Question #7: Would you do anything differently in the way you do <X> to avoid that mistake?

Note: Good candidates will have anticipated and already answered this in response to #6 or even #5.
Good answer: “Here’s what I already changed/in process of changing, remember <specific detail from answer to #1/#2/#3>? That was because of this situation.” (No credit if you don’t think it’s a good change or wouldn’t have solved the problem.)
OK answer: Some mistakes are not avoidable; it’s not a bad answer if the candidate thinks there was nothing to change and can give you a good explanation why.
OK answer: “Oh, I didn’t tell you about this before, but <specific detail> is the change I made.” (This is surprisingly rare.)
Bad answer: Never thought of improving their process as a result of the mistake. Thinking about it for the first time in the interview with you. Can’t explain why they didn’t change anything. The change they come up with doesn’t give you confidence it would prevent future mistakes.
Note: If you did a good job in #1/#2/#3, they are pretty constrained. They KNOW the right answer is to have improved their process, but they have little room to make up answers because they already told you what their process is.

When you’re done asking this question, you will have a strong read on how judgmental they are; how smart they are; what they value; how proactive they are; how they deal with tough situations; whether they can express vulnerability and admit weakness; how they introspect, take accountability, and improve from mistakes. You’ll even have a good idea how skilled and experienced they are at <X>.

How people handle mistakes (and reflect on and learn from them) is pretty telling for you as an interviewer. Operating in a startup environment, there is no textbook, no map, and there’s going to be lots of adversity as you go. So choose teammates who will have your back along the way.