by Serguey Shinder
In 2015 a message broker lost roughly four hours of messages during a failover on a project I was leading, and I spent the following week rebuilding the missing work from the other side's logs. It was my first genuinely bad week as a technical lead, and I did not forget it.
For the next decade, whenever anybody proposed that broker, I said no. I never had to say much. I had a story, and a story from someone who was there beats an argument from someone who has read the documentation, in every room, every time. Two projects were then designed around my no. One of them took an extra month and carried a queue implemented in a database table that we maintained ourselves, which I defended as the pragmatic option.
A new engineer asked me about it in the spring. Not to challenge me, as far as I could tell. She wanted to know which version it had been and what the configuration looked like, and I discovered in front of her that I could not answer either question. I knew the feeling of that week in 2015 in considerable detail and I knew almost nothing about what had actually happened.
So I went and looked. The failure I remembered was a known defect, documented, fixed in a release about five months after our incident. The default that made it possible had been changed twice since. The broker had been through three major versions and now runs under loads that dwarf ours. The evidence I had been quoting for ten years expired years ago, and there was nothing anywhere that was ever going to tell me so.
That is the part worth sitting with. A veto generates no new evidence. If you say no to something, you spend the following years not using it, which produces no information at all, while the confidence you formed sits exactly where it was, wrapped in a story that gets a little smoother each time you tell it. Everything else I believe is tested by the work, without my permission. My refusals are the only opinions I hold that are protected from ever being found wrong.
Two things came out of that conversation. When I say no to a technology I now write down which version I am talking about and what would change my mind, in the ticket, where somebody can hold me to it. And once or twice a year I take one thing I am certain about, something I would say in a meeting without pausing, and establish when I last checked it. The results have been mixed in a useful direction.
The opinions worth auditing are not the ones you argue about. They are the ones nobody argues with any more, because everybody around you learned years ago what your answer is going to be.
– Serguey Asael Shinder
Leave a Reply