by Serguey Shinder
I had a question I liked. I used it in first-round interviews for about a decade, and I could usually tell within ten minutes of asking it whether I wanted to work with the person. It involved parsing a moderately unpleasant text format by hand, no libraries, and talking through the edge cases as they appeared.
It was a good question in 2010. It separated people who had actually written code from people who had read about writing code, which was, at the time, most of what a first round needed to do. I have hired people off the back of it who turned out to be excellent, and I felt entitled to my confidence in it.
A couple of years ago I sat in a debrief and realised I could not defend it any more, and probably had not been able to for some time.
The reason was not that candidates could look the answer up. It was that the question had stopped describing the job. Nobody on my team had hand-parsed anything in years. What they did all day was read code they had not written, decide whether it was correct, reason about a system with six moving parts they could not see at once, and choose which of four plausible approaches would still be maintainable in a year. My question tested none of that. It tested production, and production had become the cheap part.
What made it hard to give up was not attachment to the question itself. It was that the question was calibrated. I had a decade of signal on it. I knew precisely what a strong answer looked like and what a mediocre one sounded like, and that calibration was real and valuable and pointed at something that no longer mattered. Replacing it meant going back to being bad at interviewing, deliberately, with real candidates and real consequences, for a year or more.
That, I think, is the actual mechanism by which assessment lags the work. Not stubbornness. The people best at judging a skill are the ones who have judged it longest, and their expertise is denominated in the old currency. The cost of switching falls hardest on exactly the people qualified to notice that switching is necessary.
What I ask now is closer to the work. I hand people a piece of real code containing a real bug, and something plausible-looking that is quietly wrong, and I watch what they check and what they take on trust. I am considerably worse at reading those answers than I was at reading the old ones.
I have decided that is the correct trade. A well-calibrated instrument pointed at the wrong quantity is not a measurement. It is just a number you trust.
– Serguey Asael Shinder
Leave a Reply