by Serguey Shinder
On the last Sunday in March our support desk was unstaffed for fifty one minutes, and the reason was four lines of date arithmetic written by somebody careful.
We ran a twenty four hour desk on three shifts, and the rota was generated a fortnight ahead by a small service that took a shift's start time and added its length in hours. Everything it stored was local time, written down the way a person would write it, because a person read it. The paging system worked the same way: when an alert fired it asked the rota who was on duty at that wall clock time and rang them.
On the night the clocks went forward there was no such person. The overnight shift handed over at half past one, an instant that did not occur that year, and the row the generator had written matched nothing the paging system ever asked for. An alert fired at seven minutes past one because a queue had begun to back up behind a partner who was refusing connections, and it went to nobody at all. The incoming shift found it at six, by which point there were twenty two thousand messages waiting and a morning of reprocessing in front of us, and a call to make to a customer who had noticed before we had.
The fix is the one everybody already knows and almost nobody has actually applied everywhere. We store instants in a single absolute form and we convert them for display at the last possible moment. Anything that recurs is stored as a rule rather than a list of moments, because every Monday at nine in a named zone is a different set of instants each year and only the rule survives. And we do no arithmetic at all on local times. Adding an hour to a local time does not give you a local time an hour later, it gives you a string that resembles one, and twice a year the resemblance fails.
We also wrote the test we should have had from the beginning, which generates a rota across both changeover weekends and across a zone with a half hour offset, and it runs every build rather than every spring.
What I carry from that Sunday is smaller than a rule about time zones. A timestamp is two separate facts. There is the instant, which is physics, and there is what a particular government has decided that instant is called, which is politics and changes by legislation. Almost every date bug I have met since was somebody storing one of those and quietly assuming they could recover the other. It works for about three hundred and sixty three days a year, which is long enough to feel like a safe assumption and never long enough to be one.
– Serguey Asael Shinder
Leave a Reply