by Serguey Shinder
In the autumn of 2021 I stood up at a department meeting of about ninety people and explained a two day outage in our order platform. I said, with the confidence of somebody who had been awake for most of both days, that the cause was a change our hosting provider had made to their load balancers without telling us. I had a timeline on a slide. People nodded. It was a satisfying explanation, because it was clear, it was external, and it was nobody in the room.
Nine days later one of our engineers found that the load balancer change had been ours. A configuration update from our own pipeline, applied by a scheduled job that nobody had connected with the incident because it ran every week and had never caused any trouble.
I told my manager that afternoon. I told the two engineers who had worked the incident with me. I added a paragraph to the bottom of the incident document. I considered it handled, and if anything I felt rather good about how quickly I had owned up.
Fourteen months later we were renegotiating the hosting contract, and a colleague from procurement who had been at that department meeting used the outage as leverage. It was in her notes. The provider's account manager, perfectly politely, sent her the ticket history. I heard about it from her in a corridor, and she was right to be annoyed, because I had given ninety people a fact and quietly taken it back from three.
What I had misjudged was the shape of a correction. I had treated it as a confession, something owed to the people who might hold the mistake against me, and delivered promptly it had felt like integrity. But the error had not been told to those people. It had been told to a room, and some of that room had repeated it, built on it and, in one case, typed it into a negotiation. A correction that reaches fewer people than the error, to most of the people who heard the error, is indistinguishable from no correction at all.
It also grows more embarrassing the longer it waits, which is precisely the pressure that keeps it small. Standing up at the following month's meeting and saying I had been wrong about the cause would have cost me two uncomfortable minutes. Not doing it cost a colleague her credibility in front of a supplier.
So when I get something wrong now, the correction goes wherever the original went, in the same channel, to the same list, at roughly the same size. If I said it on a slide, I retract it on a slide. The paragraph at the bottom of a document is for the record. It does nothing for the ninety people who will never open it.
– Serguey Asael Shinder
Leave a Reply