by Serguey Shinder
A partner integration was failing intermittently and we could not see why, so we turned on full request and response logging in production for the afternoon. It was the obvious move. We found the problem in about two hours, which was a malformed header from their side, shipped a fix that evening, and everybody went home pleased.
Nobody turned the logging off. The flag was an environment variable, set in a deployment configuration that lived in a different repository from the application, and there was no moment in the day that would naturally have prompted anyone to revisit it.
I found it three months later, and only by accident. I was building a dashboard, searched our log system for the name of a header, and got back several thousand matches containing complete authorisation headers. Underneath them were full request bodies. Session tokens. A customer's date of birth, address and bank details, printed in plain text, ninety-one days deep.
The obvious thought is that the fix was to turn the flag off, and turning it off took four minutes. That was perhaps a twentieth of the work. Our log system was searchable by anyone with a company account, which at the time was about eighty people including two agencies. Logs shipped nightly into the analytics warehouse, where retention was measured in years and access was granted more generously still, because it was thought of as a place for aggregates. There were backups of both. Every token that appeared in those lines had to be treated as compromised and rotated, and every copy had to be found and purged, which took the better part of two weeks and involved a team that had nothing to do with the original incident.
What I had got wrong was to think of a log line as a description of an event. It is not. It is a copy of data into a second system with its own access rules, its own retention, its own backups and its own audience, and that copy is made by the least examined line of code in the application.
Two things changed. Redaction moved into the logging layer itself, by field name, denying request and response bodies by default, so that no individual developer's judgement on a bad afternoon is the last line of defence. And temporary settings became genuinely temporary: any flag of that kind now carries an expiry timestamp and the application refuses to honour it after twenty-four hours, which means somebody has to consciously renew it rather than consciously remember it.
Temporary is a description of an intention, not a property of a system. If a thing is meant to end, something other than my memory has to be the thing that ends it.
– Serguey Asael Shinder
Leave a Reply