> For example, to assess debugging you can give your candidate some pre-prepared code with failing test cases and see how many bugs they can fix within 30 minutes or so. But that requires preparation and test calibration.
The tricky thing is that no matter what test you contrive, it's more likely to say something about the developer's recent experience than about their competency in general.
For example, I'd say I have pretty good intuition for when to just read code or sprinkle printfs or fire up valgrind/gdb/asan when debugging C. Which I guess is to be expected given that I've been doing C almost exclusively for many many years. I'd do pretty bad with Haskell; the last time I really used it was around 13 years ago. The next guy might be a bit lost with gcc's error messages since the last time they used C in anger was 5+ years ago for a small project, but they'd do well if you hand them Python code that uses a well known unit test framework or whatever. I guess that's fine if you're a run of the mill crud company looking for "senior <foo-language> developer" but not if you're after general competency.
You can try hard to make the debugging be more about the system than about the implementation but it's not easy to separate the two. You can make different tests for people with different backgrounds but that only makes calibration harder.
One trick I've seen a company do is deliberately pick a very obscure language that most people have never heard of. That can eliminate some variables but not all of them (I took the test and did well but I also spent a fair amount of time studying the language to figure out if it's suitable for a purely functional solution before handing in a very boring piece of imperative code). Ultimately it wasn't much more than a fizzbuzz.
And if there's puzzling involved, I'd say there's an element of luck involved. At least that's how I perceive the subconscious mind to work when you're thinking about a problem that isn't immediately obvious or known to you beforehand. Which path does your mind set you on? Are you stupid or incompetent if it happened to pick the wrong one today and you spent 10 minutes thinking about it too hard? Are you super smart if the first thing that came to mind just happened to be the right one and you didn't doubt yourself before blurting out an answer?
That is to say, test calibration is hard and there are so many variables involved. It follows that there's no obvious right way to conduct interviews. And I guess it follows that companies who need people (and aren't necessarily experts at interviewing) effectively outsource the problem by conducting the same type of interviews they've seen elsewhere. Maybe that's less cargo culting and more just doing whatever seems popular and good enough?
The best solution I’ve seen to this (if you have the time, and are interviewing for a variety of roles) is to have the same code (& same bugs) in a variety of languages. And let the candidate use their own computer and their own tools to work on the problem. If you’re hiring for a python role, get the candidate to debug python code!
I’ve done hundreds of interviews like this, and it’s fascinating watching what people do. Do they read the code first? Fire up a debugger? Add print statements and binary search by hand? I had one candidate once add more unit tests, to isolate a bug more explicitly.
After hundreds of interviews I still couldn’t tell you which approach is best. But if there’s one trend I noticed it’s that more senior people (& older people) seem to do better at this assessment. Which is fascinating. And that implies it’s not simply a test of what tools the person is familiar with most recently.
As for luck, I agree this is a problem. It’s mitigated somewhat by having a bunch of easy bugs to fix instead of one hard one. But even a debugging problem like this should be one of a set of assessments you perform. If you fail 5 small assessments in a row it’s probably not luck.
The tricky thing is that no matter what test you contrive, it's more likely to say something about the developer's recent experience than about their competency in general.
For example, I'd say I have pretty good intuition for when to just read code or sprinkle printfs or fire up valgrind/gdb/asan when debugging C. Which I guess is to be expected given that I've been doing C almost exclusively for many many years. I'd do pretty bad with Haskell; the last time I really used it was around 13 years ago. The next guy might be a bit lost with gcc's error messages since the last time they used C in anger was 5+ years ago for a small project, but they'd do well if you hand them Python code that uses a well known unit test framework or whatever. I guess that's fine if you're a run of the mill crud company looking for "senior <foo-language> developer" but not if you're after general competency.
You can try hard to make the debugging be more about the system than about the implementation but it's not easy to separate the two. You can make different tests for people with different backgrounds but that only makes calibration harder.
One trick I've seen a company do is deliberately pick a very obscure language that most people have never heard of. That can eliminate some variables but not all of them (I took the test and did well but I also spent a fair amount of time studying the language to figure out if it's suitable for a purely functional solution before handing in a very boring piece of imperative code). Ultimately it wasn't much more than a fizzbuzz.
And if there's puzzling involved, I'd say there's an element of luck involved. At least that's how I perceive the subconscious mind to work when you're thinking about a problem that isn't immediately obvious or known to you beforehand. Which path does your mind set you on? Are you stupid or incompetent if it happened to pick the wrong one today and you spent 10 minutes thinking about it too hard? Are you super smart if the first thing that came to mind just happened to be the right one and you didn't doubt yourself before blurting out an answer?
If you're lucky and know the problem beforehand, you can always fake brilliance: https://news.ycombinator.com/item?id=17106291
That is to say, test calibration is hard and there are so many variables involved. It follows that there's no obvious right way to conduct interviews. And I guess it follows that companies who need people (and aren't necessarily experts at interviewing) effectively outsource the problem by conducting the same type of interviews they've seen elsewhere. Maybe that's less cargo culting and more just doing whatever seems popular and good enough?