by Serguey Shinder
For years I had backups, and I was proud of them. They ran on a schedule, they reported success, they filled up storage steadily and reassuringly. Every night the system dutifully copied everything that mattered, and every morning I saw the green checkmark that told me it had worked. I slept well on the strength of that checkmark. What I never once did, in all those years, was actually try to restore from one.
The backups were a story I told myself, and the story was comforting precisely because I never tested it. A backup you have never restored is not a backup. It is a hopeful copy, an act of faith, a file that claims to contain your safety without ever having proven it. I had confused the existence of the backup with the ability to recover, and those are separated by a gap I had simply never walked across.
I found out the hard way, as one does. When I finally needed a restore, urgently, with real data lost and real pressure on, I discovered that the backups had been faithfully saving something subtly broken for months. A configuration change had quietly corrupted what was being captured, and because nothing ever read the backups back, nothing ever noticed. The green checkmark had been reporting the success of the copy, not the usefulness of the result. It had been telling the truth about the wrong thing.
The failure was not the corruption. Corruption happens. The failure was that I had built an entire sense of safety on a process whose output I had never verified, and verification was the only thing that could have caught it, and verification was the one step I had skipped for years because the checkmark felt like enough. I had automated the reassuring part and neglected the part that actually mattered.
What I learned is that untested safety is not safety. It is the feeling of safety, which is worse than no feeling at all, because it stops you from looking. A person with no backups is careful. A person with backups he has never tested is confident, and confidence built on an unverified foundation is exactly the thing that gets you hurt, because it removes the vigilance that might otherwise have saved you.
Now I restore from backups on a schedule, deliberately, into a scratch environment, not because I expect them to fail but because the only way to know they work is to make them work. The drill is tedious and feels like a waste every single time it succeeds. But the one time it does not succeed, it will not be during a real emergency, and that is the entire point. Safety you have not tested is just a hope wearing a checkmark.
– Serguey Asael Shinder
Leave a Reply