by Serguey Shinder
The log search was being demonstrated to a new starter on a Wednesday, as part of the ordinary tour, and somebody typed in a customer's email address to show how the filters worked. Among the results was a password reset link that had not yet expired and, a few entries below it, a complete card number that a customer had typed into a field meant for an order reference.
Nobody had decided to store any of that. Eighteen months earlier we had been debugging an awkward partner integration and switched on body logging for the requests and responses of one service. It helped, the problem was found, and the setting stayed on because turning it off again was not on anybody's list. Every request through that service since had been written out in full, indexed, retained for four hundred days and shipped to our logging vendor.
What made this worse than a badly chosen database column was who could reach it. Our database has roles, and the table that holds card data has a grant list four people long. The log platform had exactly one permission, which was whether you worked here, and a search box. Something like ninety people could have run that query, and nothing told us how many ever did.
Cleaning it up took a fortnight. Deleting from an append only index is not a delete, it is a ticket with a vendor and a waiting period, and until that finished the safe thing was to remove everyone's access, which took most of the platform's usefulness with it.
The change that actually mattered was going from a list of things to hide to a list of things to keep. Our logging helper had a denylist with password and token and card on it, and a denylist only ever contains the fields somebody has already been burned by. A request body is now not logged at all unless individual fields are named at the call site, and everything else is replaced inside the process before it goes anywhere. Bodies live for a week. Reading the raw index is a role you request.
The habit I would press on anyone is to stop picturing customer data as living in the database. It lives in the database, and in the logs, and in the labels attached to metrics, and in the error tracker with a stack frame's worth of arguments beside it, and in the analytics tool, and in the screenshots support staff paste into tickets. Those copies are kept longer than the original, guarded more loosely than the original, and several of them sit on somebody else's infrastructure. We had been careful about the one place the data was designed to live and careless about the six places it also happened to be, each created by a colleague solving a real problem in an afternoon.
– Serguey Asael Shinder
Leave a Reply