by Serguey Shinder
We turned the new pricing engine on for a tenth of our customers in the spring of 2021, watched it for two weeks, took it to everybody, and got on with the next thing. The toggle that controlled it stayed. Removing it was nobody's job, it cost nothing to leave, and after about a month I stopped seeing it when I read the file.
Four years later we ran a planned failover to the standby region on a Thursday morning, which is an exercise you feel rather good about, and served live traffic from it for six hours. In that window we issued a little over four hundred quotes at rates we had retired in 2021.
The standby had been built from the repository. In the repository that toggle defaults to false, because false is what it was on the day somebody wrote the line, and nobody goes back and changes a default once a rollout has finished. The real value lived in a configuration table in the primary region, set once by hand, replicated nowhere, written down in no document and owned by no one. Production had been running on an override for four years, and an override is invisible to every fresh environment you will ever build.
There is a second cost that had been accruing the whole time. A toggle keeps two paths alive. Ours had a branch that had not executed since 2021, had never been touched by a test since the rollout ended, and had been edited three times by people refactoring around it who had no idea it could still run. So what we served that Thursday was not even the old pricing engine as we remembered it. It was four years of unexamined maintenance on code everybody believed was dead.
What we changed is mostly bookkeeping. Every toggle now carries an expiry date next to its definition, and the build fails when one goes past it, which is rude and effective. When a rollout completes, the losing branch is deleted in the same week and the toggle goes with it. And we reconciled the defaults in the repository against what production actually runs, which took one afternoon and turned up two more.
The part I would generalise is about defaults rather than toggles. Every setting in a system has a value somebody chose deliberately and a value it falls back to, and the fallback was picked at the moment of least knowledge, by a person solving a different problem, and then never looked at again. Production hides this from you, because production has been corrected by hand over years of small interventions nobody logged. A brand new environment hides nothing. It tells you precisely what your code believes when there is no one present to correct it, and that belief is usually older than you would like.
– Serguey Asael Shinder
Leave a Reply