by Serguey Shinder
It was eleven characters, it had a company joke in it, and five of us could have typed it in our sleep. The account was called svc-ops. It had been created by somebody who left before I joined, it could do essentially anything to the production database, and it was how we did anything urgent at two in the morning.
Nobody was being reckless. Everyone who used it was senior and careful. We knew the password was shared and told ourselves it was fine because we trusted each other, which was true, and because the audit log recorded everything, which was also true.
The problem arrived on an ordinary Thursday. A table had been altered in a way that broke a downstream report, at some point in the previous fortnight, and we needed to know what change had been intended so we could decide whether to keep it or reverse it. The audit log was complete and detailed and entirely useless. Every line said svc-ops. Every line, for two weeks, for five people.
So we did the only thing available, which was to ask each other. Three of us were fairly sure it was not them. One was on holiday. One thought it might have been him but could not remember why, and that uncertainty, from a good engineer about his own work a fortnight earlier, is not unusual at all. We reconstructed the intent from the shape of the change and guessed.
What struck me afterwards was that we had been thinking about the wrong risk the entire time. When anyone raised the shared account, the conversation was always about whether an outsider might get the password, and our answer was that it was in the password manager, the manager was well secured, and access to it was tightly held. That answer was correct and it addressed a threat that never materialised.
The risk that actually cost us was not intrusion. It was that we had destroyed our own ability to answer a question about ourselves. Attribution is not a control you put in place to catch a wrongdoer. It is the thing that lets a team reconstruct its own history when something needs explaining, and we had traded it away for the convenience of not having to provision five accounts.
We split it up over the following month. Five named accounts, each with the same powers, each logging under a human name. The two in the morning workflow got no slower.
I have stopped asking whether a shared credential is safe. I ask a duller question now. When something unexpected turns up in three weeks, will this system be able to tell us who did it and what they were trying to achieve. If the answer is no, the account is a problem regardless of how well the password is protected.
– Serguey Asael Shinder
Leave a Reply