by Serguey Shinder
A colleague in sales searched for a prospect's company name, the way anybody does before a call, and our demo site came back on the first page of results. Not our marketing pages. The demo. And the reason it matched was that the prospect's competitor was in it, by name, with their order history.
The demo environment existed for a good reason. Sales had complained for years that showing the product with invented data made it look like a toy, because invented data has no texture. Three customers, four orders each, names out of a word list. So a nightly job was built that restored a copy of the production database into the demo environment, and from that week the demos went noticeably better.
Everything else about that environment followed from the word demo. It sat outside our normal deployment path. It had one shared login, written down in a sales handbook, unchanged since it was created. It had no robots file, so it was crawled, and because the product renders customer names into page titles, a search engine had built an index of real companies and the amounts they had spent with us.
The failure was not the crawler and it was not the shared password. The failure was that we had drawn our security perimeter around a word. Production was the thing we protected. Everything else was scenery, and the controls we applied, the access reviews, the network restrictions, the logging, all attached themselves to the label rather than to the data. Meanwhile the most sensitive asset in the company was being copied nightly into the least protected room we had, by a job that ran perfectly, in full view, for two years.
Data does not become less sensitive by being copied somewhere with a different name. I can write that sentence now and it sounds obvious, and I still did not act on it for two years, because the copy was made by an automated process nobody thought of as a decision.
Getting the pages out of the search index was slower than any of us expected and involved asking politely and then waiting. The real work was different. That restore job now runs a masking step, so names, addresses, contact details and amounts are replaced with plausible substitutes before anything lands. The demo environment sits behind the same sign in as everything else. And once a quarter somebody produces a list of every place a copy of the customer database exists, which was the exercise that actually frightened people, because the first version of that list had nine entries on it and four of them were laptops.
I no longer ask which of our systems are production. I ask where the customer data is, and then I go and look at what is protecting it there.
– Serguey Asael Shinder
Leave a Reply