by Serguey Shinder
I needed to reconcile two systems that held overlapping records and disagreed about them often enough to matter. I described the problem, asked how I might approach it, and got back a design involving a queue, a worker, a retry policy and a small state table to track what had been compared.
It was a good design. I built it over about two weeks, it went into production, and it did the job for roughly a year. There was a dashboard. There was an on-call note about what to do when the worker fell behind, which it did occasionally, and a bit of tooling to replay a day.
What ended it was a conversation with a colleague working on something similar. She asked why we were not just running a nightly query across both sides and looking at the rows that differed. I started to explain why that would not work, and about a sentence into the explanation I realised it would work. It would have been a view, a scheduled job, and a report. No worker, no queue, no state to get wrong, no on-call note.
The design I had built was not wrong. Everything about it was defensible. It was simply the first workable thing in a space that had at least three other workable things in it, and I had never seen them, because by the time I started thinking properly I already had something to think about.
That is the part I keep turning over. Years ago, faced with that problem, I would have spent a morning with a sheet of paper being bad at it. I would have written down four approaches, two of them stupid, and the act of rejecting the stupid ones would have taught me where the real constraint was. That morning is where I used to learn what the problem actually was, as opposed to what I first assumed.
I skipped it, and I did not notice skipping it, because what replaced it was not nothing. It was a competent answer arriving before I had formed an opinion of my own.
A wrong answer would have been safer. A wrong answer gets argued with. A good answer ends the search, and the cost is invisible, because you never get to see the three designs you did not generate.
So I do something slightly laborious now. Before I ask anything, I write my own list of approaches, however thin and however embarrassing. Two lines each. Then I ask, and whatever comes back goes on the bottom of that list as another entry, not as the conclusion.
It is often the best entry. But it is on a list I made, and I can still see what it is being chosen over.
– Serguey Asael Shinder
Leave a Reply