Interviewing: The Email Challenge

I’ve interviewed a significant number of candidates recently for a developer position at my current customer. I’ve never boiled down interviewing to an exact science, but I think I’ve managed to get a lot of bugs out of the process. The up-front work is where I’ve been concentrating most of my efforts, due to the large number of resumes. By “up-front,” I mean the efforts leading up to the decision as to whether or not to bring in a candidate for face-to-face interviews.

Prior to even looking at resumes, putting out a good job description is an important step. That’s a topic for another blog entry. Once you have a stack of resumes, your job is to whittle them down to a small enough number. My recommendation is to set aside two weeks for screening candidates. Over that two weeks, expect to run an average of 2 phone screens a day. (Don’t try to do more than a couple a day, unless you want to go insane!) This allows you to cover up to twenty candidates if necessary.

Each phone screen should last from 30 minutes to an hour. This is a feeling-out process for both of us. I try to give the candidate all the negatives up front and give them the opportunity to beg off. Most don’t. I ask a number of conceptual questions, mostly design or process related. I look for the ability to carry a technical conversation. Does the candidate have knowledge of significant development concepts (patterns, OO, unit testing, etc.) that is commensurate with their claimed experience?

For a while, I would ask technical questions over the phone. It’s awkward, not very effective, and ultimately frustrating. I brought in a few people for face-to-face interviews; these people ended up being good at answering technical questions over the phone but had little additional value.

Recently I began a practice of sending out an email challenge to each candidate that I was still interested in. The email has two parts. The first part is a number of simple questions and answers regarding the language of choice (Java here). Answers to these few simple questions can reveal a lot about the core knowledge of each candidate. You’d be surprised, for example, how many candidates have a tough time explaining the difference between .equals and ==. Even with Google at their disposal!

For the second part, I grab a prototypical sludge method from the system. The candidate’s job is to figure out what the messy code does. They can define a contract for the method, write a unit test, add comments, or do whatever they choose. I also suggest they rewrite the method. The nice part is that they can cheat as much as they want–they still have to be the ultimate arbiter of what is quality code.

I give the candidate two days to return the email. From there, it’s a dialogue. I challenge the candidate as to their answers and as to why they coded things a certain way. A candidate might get a number of questions wrong, or have questionable programming style, but that doesn’t preclude them from further consideration. I look for their ability to think, to justify their answers, to perhaps open their mind up to different ways of looking at things. I bring in candidates for face-to-face interviews based upon the results of this dialogue.

I used this technique many years ago when doing a slew of interviews. It worked well then, and it still works well now. One other benefit is that it gives candidates an opportunity to shine if their phone chat didn’t go so well. And most candidates actually enjoy it, since most good developers are geeks who like a challenge.

The email challenge also does a good job of ferreting out the troublemakers. One recent candidate (I’ll call him Joe Blow) interviewed for a lead developer/architect position on the team. All went well on the phone. But answers in Joe’s email response were commensurate with a mid-level developer. A couple of answers to the Java questions were dead wrong: Joe demonstrated complete misunderstanding of the difference between instanceof and getClass(), for example. The code he sent back was pretty confusing in one spot, and only adequate elsewhere. Further, Joe hadn’t bothered to compile it, which I had explicitly required per the instructions. I pointed out the incorrect answers, as well as the code issues, including items for which there is no “right” answer, just opinion.

All of the other candidates were able to cope with me picking at their code. Some of them challenged me back–good!–or acknowledged that they could have done things differently, and why or why not it might be a good idea to do so. Unfortunately, Joe Blow couldn’t handle the fact that someone might actually see things a different way. He ended up withdrawing from consideration for the position immediately after providing his very defensive responses. Apparently he couldn’t take the heat. He wrote a lengthy email to his recruiter, and also accused me of looking for people who coded in a specific Java “dialect.”

Touchy and perhaps even whiny: this is not the kind of candidate you want for your lead developer! Still, I learned from the process. I added a few more disclaimers to my comments (“these are not intended to suggest there is one right answer,” for example) and tried to show a bit more compassion in my already-sensitive answers.

Ultimately, when you’re looking for a developer, you want to know if they can write maintainable code. Asking technical questions rarely gives you the answer. The only way to find out is to get them to write some code. Sitting and pairing in a live interview accomplishes this effectively. But if you want to save the time and expense of bringing in candidates for face-to-face interviews, the email challenge is a great tool.

Pair Programming Observations

Pair programming is one of the most contentious practices of extreme programming (XP). The basic concept of pair programming, or “pairing,” is two developers actively working together to build code. In XP, the rule is that you must produce all production code by virtue of pairing. The chief benefit touted by pairing proponents is improved code quality. Two heads are better than one. Note that pairing is a practice that you can use exclusively of XP.

However, a large number of developers despise the notion of having to sit next to someone for the better part of their work day. Reasons for resistance can include:

  • the belief that pairing is a waste of time and ineffective
  • the belief that pairing should only be used occasionally
  • fear of exposing personal weaknesses
  • personality issues, including introversion

Pairing is not for everybody. But too many people resist pairing based on a knee-jerk reaction to what they understand it to be. Usually, the practice and its benefits are not fully understood.

Pairing Technique

There are only a few simple rules to follow if you choose to do pair programming. First and foremost is that all production code must be developed by a pair. Conversely, this means that are plenty of other activities that you can undertake in the absence of a pair:

  • work on any non-production code: the acceptance test framework, other tools, build scripts, etc. In most shops, these are in dire need of attention.
  • work on documentation
  • work on spikes for future stories
  • learn how to use a new third-party product; learn a new coding technique; and so on
  • identify problem spots in the production code that need refactoring
  • refactor tests
  • improve the existing test coverage if necessary

Any of the above could be done better if a pair were available, but they are usually lower risk activities. If time is absolutely a factor, the non-pairing developer can work on production code, but with the insistence that such code is peer-reviewed after the fact.

Having a non-pairing developer work on production code should be the exception, not the rule. As such, it should be justified and treated as a high risk activity.

The second pairing rule: it’s not one person doing all the work and another watching. During a good pairing session, the keyboard should be moving back and forth between the two participants several times an hour. The person without the keyboard should be thinking about the bigger picture and should be providing strategic direction. They should also be helping to ensure maximal code quality and minimal defects.

Third, don’t pair more than 75% of your work day. A good, productive run of 6 hours of software development is mentally exhausting. Make sure you take breaks! Get up and walk around for a few minutes at least once an hour.

Finally, you need to switch pairs frequently. Working with any one person for any extended duration will not only drive you nuts, but you will begin to lose the benefit of getting fresh outlook on problems. Minimally, switch pairs at least once a day. In fact, I promote switching pairs once in the morning and once in the afternoon. In addition to getting new insight on solving a problem, there is another, less obvious, benefit to frequent pair switching.

Context Switching

When you sit down to work with a new pair, you must switch mental contexts from the task you were working on to a whole new problem. Context-switching is difficult. Work for only 55 minutes then switch tasks? Seems outrageous. One might think, “It’ll take almost that amount of time for me to come up to speed on what you’ve just developed!”

An XP rule of thumb is: when something is difficult or painful, do it more often until it becomes easier. If integrating is a royal pain, do it more often until you learn how to do it better, or at least until it’s apparent you’ve hit the point of diminishing returns.

If context-switching takes too much time, do it more often. In theory, what this should do is force developers to write better code. By “better” I mean code that accommodates cheap maintenance without adverse impacts on the system.

If it takes me thirty or even ten minutes to come up to speed on a task, yes, there is a thrashing problem. If instead the other developer has followed good design/coding guidelines (small, composed methods, no duplication, appropriate naming, and basic OO design principles go a long way), context switching should take only a couple minutes. Couple that with a test-driven approach, and I can quickly focus on a small amount of detail.

The Numbers

So how many people actively resist pairing? At a large shop of ~300 developers (divided among several teams), about fifteen (5%) developers actively resisted pairing when they embarked upon XP. The rest of the people fell into one of three groups: skeptics, interested adopters, and cows, divided fairly evenly in terms of numbers. After pairing for a few iterations, perhaps five of the fifteen resisters learned to enjoy it. Another five didn’t mind it enough to complain any more, and another five hated it even more than before.

Having consulted in a good number of XP shops, my personal experience shows these percentages to be pretty consistent. Based on what I’ve seen, you’ll end up with from one to five percent of developers who can’t or won’t pair. For most shops, that’s one or two people.

Obviously there are always people who don’t voice their objection and just go along with whatever is tossed their way. You do want to ensure everyone has a forum for feedback. I’ve solicited feedback via anonymous 3×5 cards, email, public forums, one-on-one discussions, however I could. Beyond that, if someone isn’t going to be honest enough to complain, I suspect it says something about the caliber of that employee.

What To Do With These People

Any shop embarking on XP, RUP, Scrum, or whatever, needs a coach to steer people in the right directions. Engaging in a new process without a coach is worse than trying to play a football game without a coach–at least the football players have done it all before and know some of the things to watch for.

It’s a coaching failure if I’m unable to turn some of the pairing resisters around. And usually I can, by working directly with them, by demonstrating the benefits firsthand, and by ensuring that the process otherwise goes smoothly. From my own exposure to pairing, I initially thought it was a bad idea. After practicing it a bit, I didn’t necessarily love it, but recognized the benefits and was willing to do it. Shortly thereafter, I began to love what I was able to get out of it.

Most sizeable shops have small adjunct efforts. For valuable resources, one-offs are a great place. There’s also the possibility of working on non-production code, such as tools for internal use.

Resisters might still be able to work within a team, as long as they put up with their end of the bargain. Remember that pairing is initially a way of doing continuous review. In lieu of pairing, the resister must initiate Fagan inspections or some other sort of formal review. One way or another, the code must be reviewed–otherwise you’ll get pricy consultants (like myself) or unguided novices, producing unmaintainable garbage. Often, people find that the evil of pairing is preferable to the evil of group review.

Ultimately the rules should be up to the team, as long as the rules satisfy the requirement that code is reviewed. If the entire team revolts and insists upon no pairing, then they can all do inspections or whatever review form acts as a second-best choice. If 95% of the team insists upon pairing, and the sole remaining developer can’t deal with it at all, the organization should help them find something else to work on.

While this may sound intolerant, remember that software development is a team effort. Suppose I go to your shop and find out that you value RAD. You sit in a board room with 25 other people for two days and hash out every detail of the project. You then produce 200 pages of design documents. But I have a personality disorder that prevents me from contributing in such an environment. I can’t stand sitting in a room for days on end slogging through stuff, 95% of which is useless to my role. And I also have a disorder that prevents me from understanding design document doublespeak.

Do I belong in this organization? Probably not. If the organization is successful with this culture, why should they waste time and money trying to accommodate my obstinance?

I’m not trying to be clever or obtuse here. The point is that organizations should grow the cultures that they value, and that those cultures may not be appropriate for everyone. Anyone who says every shop should do XP or RUP or whatever is insane. The reality is that there are always other shops to choose from.

Other Pairing Benefits

The book Pair Programming Illuminated goes into much further depth on the costs and benefits of pair programming, in addition to many other related topics. I highly recommend it. Over the years, I’ve built my own brief list of pairing benefits.

General benefits:

  • Produces better code coverage. By switching pairs, developers understand more of the system. Many benefits can result from this increased knowledge.
  • Minimizes dependencies upon personnel. Everyone worries less about buses and trucks.
  • Results in a more evenly paced, sustainable development rhythm.
  • Can produce solutions more rapidly.
  • Moves all team members to a higher level of skills and system understanding.
  • Helps build a true team.

Specific benefits from a management standpoint:

  • Reduces risk
  • Shorter learning curve for new hires
  • Can be used as interviewing criteria (“can we work with this guy?”)
  • Problems are far less hidden
  • Helps ensure adherence to standards
  • Cross-pollination/resource fluidity. Allows you to swap members from two or more teams on occasion, with minimal downtime. Usually each person will bring immediate value to the new team (“you guys should use this utility class that we built here…”).

Specific benefits from an employee perspective:

  • Awareness of other parts of the system
  • Resume building
  • Decreases time spent in review meetings
  • Continuous education. As someone who thinks he is a pretty hot programmer, I still learn new things every day from even the most junior programmers.
  • Provides the ability to move between teams. (“this team is boring,” “I can’t stand working with him,” “that stuff they’re doing looks cool.”). Since this can be a benefit for management as well, they don’t have to clamp down on their resources.
  • More rapid learning as a new hire. You don’t sit and read out-of-date manuals for a week, or worry that you’re going to be fired because the system looks indecipherable.

These benefits come about from monitoring the process, making sure the technique is executed well, and fixing problems. Don’t forget a coach!

Skill Level Gaps

An experienced developer can outperform an inexperienced developer by two, three, five, ten or even twenty times. Sitting with a novice can be excruciatingly painful. But I’d rather burn a little time bringing other guys up to speed as soon as possible. It pays off in spades.

Pairing allows me to keep tabs on their work and make sure they are not producing junk that will have to be rewritten. It also prevents them from holding up the project. Several years ago I was on a project that ultimately got cancelled, largely due to a schmuck that couldn’t get his work done in time, and when it did get done, it was garbage. With a pair, incompetence that can destroy a project surfaces far more quickly.

I still learn some very interesting things from working with novice developers. And they learn a wealth of things that they would never learn were they to be left to their own devices.

Leaving an inexperienced developer alone to suffer through the system and other issues presents them with a steep learning curve. Through pairing, this learning curve begins to flatten more early on in the project. The more time I spend up front with a novice developer, the more we can depend upon their contributions later in the project–when it matters far more.

Of course, the novice must be capable of growing. Pairing lets you find out quickly who’s worth keeping versus who will always be a drag on the team. Management can get involved in the first month or so of a project, as opposed to late in the project when it’s crisis time.

Final Comments

I’ve been in the position of promoting XP and hence pairing as a consultant for a while now. Until I did a longer, six-month consulting stint, my pairing experience was more sporadic. Sure, I saw the benefits and the negatives, and did it enough to know how to make it work. Plus I learned plenty more about it from other sources. But until I sat there and actively paired for a longer duration, some of its nuances weren’t as evident.

In fact, pairing seemed to me like a nice thing to promote, but maybe pairing wasn’t something that I needed to worry so much about. Not drinking my own kool-aid!

What I realized is that pairing after a while becomes a dependency, in both a good way and a bad way. I learned to look forward to most pairing sessions (there are always some difficult people). I also felt naked when not pairing, and begin to question more what I produced by my lonesome self. Pairing became assuring and thus relaxing.

Before, I would be overly confident that I was a great programmer and that I produced code that was just fine. Maybe not! A few pairings with some sharp people and I learned a few cool new techniques. I took on more humility.

Dependencies in code can be bad, as can dependencies in life. The secret is in managing these dependencies well. Learn to use pairing as a tool to help you do better the next time you aren’t

Atom