by Serguey Shinder
For most of the time I have been doing this there was a window. Ours was two until four on a Sunday morning, and we did everything difficult in it. Schema changes that locked a table for ninety seconds. Certificate replacements. The restarts that had to happen in a particular order. On a bad night somebody was up until five. On a good night it was forty minutes and a message at ten past two saying it was done.
It has gone, and I do not think it is coming back.
It went slowly and for entirely good reasons. We signed a reseller in Sydney, for whom our quiet window is the middle of the working afternoon. A customer in California runs their end of day process at what is nine in the evening for them. An integration partner polls us every sixty seconds forever, including on Christmas Day, and treats a failed poll as an incident on their side. And our own customers stopped being people at desks. The last time I measured it properly, the two quietest hours of our week carried about four per cent of the day's orders. When we chose that window, the same two hours carried a third of one per cent.
The last time anyone proposed announcing a maintenance window, three customers objected inside the first hour, and one was right to, because the hour we picked was the hour their finance team closes the month.
So we changed how we build rather than when we deploy. Schema changes go out in two or three steps, each of which is safe with both the old and the new code running at once. Restarts are rolling, and the service has to tolerate two versions of itself talking to each other for twenty minutes, which is a harder thing to write. Behaviour changes go out behind a flag, turned on in daylight when people are awake to watch. All of it takes longer. A change that used to be one migration is now three deployments across a fortnight.
What I expect for the rest of my career is that this only goes one way. The moment a business sells beyond one time zone, or connects to anything that polls, the window closes and never reopens, and everything built on the assumption of a quiet hour goes with it. The overnight batch. The release weekend. The idea that you can ever stop the world for ninety seconds to be careful.
So the question I ask about a change has changed shape. It used to be whether it was safe. It is now whether it is safe to apply while the system is running, under load, with half the fleet still on the previous version and nobody watching. That is a much harder question, and the only one that will be asked from here on.
– Serguey Asael Shinder
Leave a Reply