by Serguey Shinder
We wrote the order history endpoint in 2019 and it did not change once in five years. In March it took the whole service down for four hours, and the commit that broke it was written by somebody who had left the company in 2021 and was correct on the day it was merged.
It returned every line a customer had ever ordered. No page size, no cap, no argument about it in review, because at the time the largest account in our database had nine hundred and six lines and the largest plausible account anybody could imagine had a few thousand. It was a wholesale food business we signed in February. By the middle of March they had three hundred and forty thousand lines, and every time one of their staff opened the history page we built the whole thing in memory, serialised it, and sent it. Four of them opened it within the same minute on a Tuesday morning. The service ran out of heap, restarted, came back, was hit by the same four people wondering why the page had not loaded, and restarted again. Every other customer on that service saw nothing at all for four hours, and our new and largest customer spent their first fortnight with us watching a page that never finished.
The fix took half an hour. Pages of two hundred, a hard cap in the query rather than in the code that calls it, and a separate export path for anyone who genuinely wants everything. That is not the part worth writing down.
The part worth writing down is what we changed about the fixtures. Our test data had one account with forty orders in it, and it had had one account with forty orders in it since the week the repository was created. Every test we had ever written, including the careful ones, asserted that our code was correct for a customer who does not exist any more. So we built the fixtures from the real shape of the business. The largest account we actually have, the widest row, the oldest record, the one with the strange characters in it. When sales sign somebody unusual, the fixture changes.
What I carry is that code does not have a scale. Behaviour does, and behaviour is the code plus the data, and we only ever own half of that. The other half is decided by whoever is selling, and they are not obliged to tell us, and nothing in the repository will ever show it changing. So when I read a function now I try to ask what shape of input it quietly assumes, and then I go and find out who in the building is in a position to change that shape without telling me. There is always somebody, and they are usually doing a good job.
– Serguey Asael Shinder
Leave a Reply