by Serguey Shinder
There was a failure in one of our services that behaved like it had a personality. It showed up in production, reliably, a few times a day, but the moment I attached a debugger or added logging around it, it vanished. I would spend an afternoon instrumenting the code, run it a hundred times, and watch it pass a hundred times. The instant I removed my instrumentation and shipped, the failure came back, smug, as if it had been waiting for me to look away.
For a while I treated this as bad luck, or as the machine being contrary. That is a comforting story because it makes the bug someone else's fault, some gremlin in the runtime. But a bug that hides when you observe it is not being contrary. It is telling you something precise about itself, if you are willing to hear it. It is telling you that the act of observing changes the timing, and if changing the timing changes the outcome, then timing is the bug.
Once I framed it that way, the mystery collapsed into an ordinary race condition. Two operations that I had assumed happened in a fixed order were actually racing, and most of the time one won by a comfortable margin. My logging slowed one of them down just enough to always let the other finish first, which is why the failure politely disappeared whenever I watched. I had been papering over the bug with my own attention.
What stayed with me was not the fix, which was small, but how long I had spent looking directly at the thing while it hid from me. The failure was never random. It was conditional on something I was not measuring, and my instinct, to look closer, was the exact instinct that destroyed the evidence. I was standing in the crime scene, contaminating it, and calling the crime unsolvable.
I have since learned to be suspicious of any bug that improves when I observe it. That improvement is not progress. It is a clue, and usually the most important clue available, because it tells me the fault lives in the space between events rather than inside any single one of them. The disappearance is the diagnosis. I just had to stop reading it as failure to reproduce and start reading it as a fact about the system.
Now when something hides from my debugger, I feel a small, grim gratitude. It has narrowed the search for me. It has told me the problem is not in the logic I can freeze and inspect but in the choreography I cannot, and that the ground I need to dig in is time itself. The bugs that run from the light are the ones pointing straight at where they live.
– Serguey Asael Shinder
Leave a Reply