by Serguey Shinder
I needed to reproduce a bug that only appeared with a particular shape of customer record, and after two days of failing to invent data that triggered it, someone told me the easiest thing was to look at staging, because staging had a copy of production from a few months back. He said it the way you mention where the coffee is kept. It was not a secret and it was not a scandal. It was simply how things had always been.
I used it, found the bug, and fixed it. And then I sat with what I had just learned, which was that a full copy of real people's information was living in an environment that had none of the protections we applied to production. No access review, no audit trail worth the name, a shared login that predated most of the team, and a database whose password had been in a wiki page for as long as the wiki had existed. The data was identical. Everything around it was not.
What made it dangerous was exactly what made it convenient. Staging existed to be easy. That was the point of it. People needed to get in quickly, try things, break things, and get out, and every control we might have added would have been friction in a place that was deliberately frictionless. So the protections had never been added, and the data had been copied in anyway, and the two decisions had been made years apart by different people who never spoke to each other.
I had spent my whole career thinking about security as a property of systems, and this was the day I understood it is a property of the weakest place a thing exists. Production was well defended. That defence was worth precisely nothing while the same records sat one environment over behind a password in a wiki. An attacker does not attack your architecture diagram. They attack the softest copy.
The fix was not clever and it was not popular. We stopped copying and started generating, which meant someone had to write a tool that produced realistic data with the awkward shapes real data has, and that took weeks nobody had budgeted. For a while the bugs were harder to reproduce and people said so, loudly. It was genuinely worse before it was better.
But it changed how I ask questions now. When someone tells me a system is secure, I no longer ask about the system. I ask where else the data lives, who copied it there, when, and what protects it in that place. The answer is almost always somewhere nobody was thinking about, created for a good reason on an ordinary afternoon by somebody just trying to get their work done.
– Serguey Asael Shinder
Leave a Reply