by Serguey Shinder
We lost storage at our provider on a Tuesday in November and were degraded for three hours and ten minutes. It was the best incident communication we have ever done. The status page updated itself from our alerting within a minute of the first failure and every eight minutes after that. Affected customers had an email inside two minutes. The product itself carried a banner naming the functions that would not work.
During those three hours our support line took sixty one calls and our shared mailbox took over two hundred emails, almost entirely from people who had already received all of it.
I read the call notes the following week expecting to find that the messages had not arrived, or had been unclear. They had arrived and they were clear. Nobody was asking what was happening. They were asking whether their Thursday cut off still stood. Whether we would waive the late fee on a consignment their customer was now going to receive on Friday. Whether this was going to happen again before the end of the quarter. And a surprising number, in one form or another, were asking whether there was actually somebody there.
Every one of those is a question about judgement or commitment. Not one of them can be generated, because the whole value of the answer is that a person with authority is attached to it and can be held to it afterwards.
That is the shape I now expect the next ten years to have. The cost of producing accurate, timely, well written information about what our systems are doing is going to keep falling until it is effectively nothing, and every organisation will have it. Statements of fact are about to stop being a differentiator or a service. What stays scarce is a person who can decide something on Tuesday afternoon without asking anybody, and whose name is on it when it turns out badly.
So we have started staffing for that rather than for writing updates. Every automated message we send during an incident now carries a named person and a direct way to reach them, and that person is rostered, as an actual shift with cover, and they hold a standing authority to waive charges up to a limit and to commit us to a deadline. It costs us about the salary of one engineer.
Calls during our last incident were down by more than half, and I do not think it was because customers needed less. It is because the thing they had been ringing for was the one thing the automation could never send, and this time it arrived without them having to come looking for it.
I expect most of our industry to spend the next decade making that information cheaper. I think the advantage will sit with whoever spends it on the people allowed to make promises.
– Serguey Asael Shinder
Leave a Reply