I just read an interesting article on Slashdot (made moreso by the fact that I have an interview with a company looking for the very best developers later today) about how a company can go about finding superstar programmers. As I expected, the discussion veered quickly into what I now call “the scientist bias”. That is, people began talking about how to set up tougher interviews, screening more rigorously, not checking past work, etc. etc. It’s the same sort of closed-mindedness that I lamented about when discussing peer review (and I still lament, even though 4/4 of my recently submitted papers were accepted). Just as you cannot hope to discover great ideas by being closed-minded, you can’t hope to discover great people by being closed minded.

The easiest way to find good programmers, I would think, would be to first admit that, like peer reviewers, you don’t know precisely what you’re looking for. Just as one scientist unfamiliar with the work of another can’t objectively judge the full ramifications of the other’s science, one programmer cannot objectively judge the programming ability of another. Pretending that you can introduces all sorts of ascertainment biases into the loop, not the least of which being that some people just don’t interview well. I know – I’m among them.

Once you admit that, look at the candidate’s past work. Run it. Try to break it. If you’re feeling adventurous, sic QA on it and see what they find. See (or ask) how long it took to make, how much maintenance had to go into it, and how efficient it is. Don’t neglect the circumstances under which it was written, either – I personally wrote some very professional code when I was 12 with no formal CS education, which I think is quite an achievement. I couldn’t have used knowledge I did not possess, but I made up for it with sheer ability. Not many people independently rediscover alpha-beta pruning at 14, for instance 🙂

Next, if you want to see how the person can actually code, give him a short assignment and ask him to hand it in the next day, for example. This models a real workplace condition – the skills demonstrated in the assignment are the same that will be demonstrated on the job.

By the time you bring someone in for an interview, you should basically have made a decision to hire already, and the final interview would then consist of making sure the company and candidates were fits for each other. If you ask any sort of technical question, ask it in full and let the candidate work. If you jump in every 5 seconds to change the requirements, it’s guaranteed to screw up the candidate. This happened to me during my final interview at Google and really cast doubt on my view of the whole process. For example:

“Write an algorithm to reverse a string”
*Writes about 80% of it, when suddenly…*
“Do it with just one byte of storage”
*Erases and starts over, getting about 50% of the way*
“Making just one pass over the array”
*Erases and goes again*

etc.

I think that interview actually went well, as I was able to perform everything the interviewer asked without hesitation, but it was a very frustrating process, and prevented me from forming any lasting “big picture” image of the problem.

Anyway, those are just my thoughts on the process. Right now I have to put up with whatever techniques people decide to use, but when I hire people, I’m going to do it my own way.

(Update: And I am completely amazed, but the average programmer apparently only codes 76.7 lines of code per hour. This isn’t a measure of quality, just speed, but now I understand what the mythical man month means about the best developers being orders of magnitude better than the worst.)