by Serguey Shinder
We were changing how a part of the product worked and about three thousand eight hundred customers needed to be told. Our marketing platform could not select that particular group, and the change was going out in two days, so somebody wrote a script that afternoon. It read the addresses from a query, composed one message, and sent it.
It sent one message. That is the whole of it. The list went into the recipients field of a single email, so every one of those customers received a message with three thousand eight hundred addresses printed at the top of it, and the first replies began arriving within four minutes, several of them helpfully to everyone.
Our customer list is commercially meaningful. Knowing which companies use us tells you something about those companies, and two of the names in that header were direct competitors of each other. Beyond the disclosure itself, we spent the following fortnight on notifications, a regulator conversation and a large number of apologies, and I would rather not add up what it cost.
The mistake in the code is two characters wide and not worth discussing. What I have thought about since is where that script lived. Everything I would point to as our security practice attaches to the application. Code review, tests, a pipeline, a staged rollout, an approval before anything reaches production. None of it touched this, because this was not the application. It was a file on somebody's laptop, written in an afternoon, run once, and deleted.
That category is enormous and I had never once thought of it as a category. The backfill script. The one-off correction to fix a bad import. The query somebody runs against the replica to answer a question from finance. The little exporter that builds a file for a partner because the proper integration is three quarters away. Every one of them operates on the whole customer base at once, with production credentials, written under time pressure, reviewed by nobody, and each is capable of doing more damage in one execution than any endpoint we own.
The controls we added are dull and have held. Anything that sends to, or writes to, or deletes more than a handful of customer records runs through a small tool with a compulsory dry run that prints the count and the first and last recipient and requires somebody to type the number back. Our mail helper refuses more than one recipient unless explicitly told otherwise at the call site. And operational scripts of that kind now live in the repository, not on a laptop, which gets them a reviewer without anybody having to insist on one.
We had built a careful perimeter around the front door and left the tools that operate the building on a hook by the entrance.
– Serguey Asael Shinder
Leave a Reply