by Serguey Shinder
The complaint was that the catalogue sometimes could not find a product, and that trying again usually fixed it. Sometimes. Usually. Those two words are how a support ticket tells you that nobody has understood the problem yet, and we sat with that ticket for most of a fortnight.
We ran four application servers behind a load balancer. The failures came in at roughly one in four requests, which should have told us something immediately and did not, because the pattern only becomes obvious once you already suspect it. What made it worse was that the same product code would fail and then succeed thirty seconds later, so every reproduction attempt looked like a fluke rather than a rule.
Product codes in that system were normalised before comparison. Whatever the customer typed, or whatever arrived from a partner, was converted to upper case and matched against the catalogue. An entirely ordinary line, written years earlier, no arguments beyond the string itself.
One of the four servers had been rebuilt after a disk failure a month before, and it had come back up with a different system locale. On that machine, converting a lower case i to upper case did not produce the letter I. It produced a capital I with a dot above it, which is correct, in the sense that it is exactly what the rule for that language requires. Every product code containing an i failed to match on that one server and matched perfectly on the other three.
The line of code had never mentioned a language. That is the part I keep returning to. It looked like a function of its input, and it read like one, and for years it behaved like one because every machine we ran it on happened to agree. What it was actually doing was reading a setting from the operating system underneath it and using that as a second, invisible argument.
We fixed it by naming the locale explicitly at every conversion, which took an afternoon. The more useful change was a short block that runs at startup and writes the locale, the time zone and the default character encoding into the log, and refuses to boot if they are not what we expect. Two of our environments were wrong. Neither had caused a visible problem yet.
What I look for now is any call that reads the world rather than its parameters. Upper and lower casing. Date formatting. Sorting text. Reading a file with no encoding stated. Anything that quietly resolves against a machine, because a machine is a thing that gets rebuilt by somebody in a hurry on a Sunday.
A function with hidden inputs is not a function. It is a question, and the answer depends on where you ask it.
– Serguey Asael Shinder
Leave a Reply