by Serguey Shinder
Customer onboarding stopped on a Tuesday, and it took us most of the morning to establish that nothing we owned had broken, because nothing we owned was involved.
What had broken was an automation built by somebody in operations in one of those tools where you drag boxes onto a canvas and join them with arrows. A form collected the details. A spreadsheet held them. Three calls went out to our own system and to two suppliers. A welcome email went out. A record appeared in the sales system. It had been running for fourteen months, it had onboarded something over nine hundred customers, and that morning one of the suppliers had tightened a rate limit and the whole thing had quietly stopped halfway.
It was in no diagram of ours. It was in no monitoring, no backup, no disaster plan and no register of anything. The credentials it used belonged to her personally. And here is the part I want to be fair about: she had built it because the request had been sitting in our queue for nine weeks, and she had been entirely right to build it. It worked. It saved a genuinely tedious two hours a day and it ran for fourteen months without anybody's help.
My first instinct was a policy, and I am glad I did not act on it. A rule against building things outside engineering does not stop anything being built. It stops us finding out, which converts a system we can see into a system we cannot, and the nine week queue that caused it would still be sitting there.
I think this is the shape of the next decade and I do not think it is close. The number of people who can assemble a working system is rising very fast, and most of them will not be engineers, and they will be solving real problems that we are too slow to reach. That is not a threat to the job. It is a description of where the job moves. Somebody has to provide the credentials that can be rotated, the connectors that log what they did, the environment where a failure raises an alarm rather than being noticed on a Tuesday, and the list of what exists.
We have a register now, and putting something on it costs the owner nothing and buys them monitoring and a service account in their own name. Eleven things went on it in the first month. Four of them we had not known existed, and one of them was, on inspection, the most important scheduled job in the company.
The systems your company runs in five years will not all be written by the people you hired to write systems. The only decision available is whether they are built where you can see them.
– Serguey Asael Shinder
Leave a Reply