by Serguey Shinder
On a Tuesday in March 2024 our website woke up showing Monday's catalogue. About fourteen hundred price changes that should have gone live overnight had not, and the job that builds the catalogue was still running at seven in the morning, four hours after it normally finishes, using one processor flat out and producing nothing.
It was walking in a circle. Every product category in our system records its parent, and to build the menus and the breadcrumbs the job starts at each category and follows the parents up until it reaches the top. The previous afternoon somebody in the catalogue team had tidied the fixings section. Anchors had been a subcategory of Fixings, and she moved Fixings underneath Anchors, meaning to move the two apart in a second step she never got to. For one evening each was the other's parent, and a walk up from either of them never arrived anywhere.
The edit screen had checked what it knew how to check. The new parent existed. It was active. It was not the category itself. Every one of those conditions was true, and each row in the table was perfectly valid. What was broken was the shape, and the shape was not a property of any single row. The column let each category point wherever it liked. Everything that read the column assumed the result was a tree, and nothing that wrote to it had ever been asked to keep it one.
That was the part I kept turning over. We had a rule that the whole system depended on, and it lived only in the heads of the people who wrote the readers. Readers cannot defend a rule like that. By the time they discover the data is wrong, all they can do is fail, and ours failed in the most expensive way available, by hanging quietly rather than stopping with a name.
We changed three things. The screen that moves a category now refuses any parent that sits anywhere beneath it, checked in the same transaction as the move, so two people rearranging the same branch cannot each make half of a loop. The job that walks the hierarchy carries a limit: we have never had more than six levels, so at twenty it stops and reports which category it started from. And a small query runs every night looking for any category it cannot trace to the top. The first time it ran, it found two more loops, both from 2019, deep in a discontinued range that nobody browsed and the job had long been told to skip.
A table will hold any arrangement you let it. If a whole collection of rows has to keep a shape, the code that changes one of them has to look at the rest, because nothing else will.
– Serguey Asael Shinder
Leave a Reply