by Serguey Shinder
We had monitoring, and I was proud of it. Failed logins from unusual locations produced an alert into a channel the whole team could see. It was exactly the kind of thing you point at in a review when someone asks how you would know if something were wrong, and the answer was clean: we would know, because the alert would fire.
It fired about forty times a day. Most of them were a scanner sweeping the internet, some were a colleague travelling, a few were a service account with a stale credential retrying forever. All of it was noise, none of it was actionable, and within about three weeks of turning it on the whole team had developed the same reflex, which was to glance at the channel and move on without reading.
I do not remember deciding to stop reading. That is the part worth dwelling on. There was no meeting where we agreed the alert had become useless, no moment of choice I could point to and regret. We simply adapted, the way you stop hearing a road outside a window, and the adaptation was complete and entirely invisible to us.
What broke the spell was a routine review months later, when someone scrolled back and found a sequence in the middle of an ordinary Tuesday afternoon that did not look like the scanner at all. Same account, unusual pattern, several failures, then a success. It had been sitting in that channel, visible to eight people, for weeks. The system had done its job perfectly. We were the part that failed.
The lesson I took was not about that particular alert. It was that a detection system includes the humans reading it, and any measure of the system that stops at the words "the alert fired" is measuring roughly half of it. An alert nobody reads is not weak protection. It is worse than having no alert at all, because it consumes the attention a real one would have had, and it lets you believe you are covered while you are not.
So we did something that felt at the time like a downgrade. We deleted most of the alerts. Whatever survived had to pass a single test: if this fires, what does a person do in the next ten minutes? If the honest answer was "look at it and probably nothing", it went. We ended up with four alerts where we had twenty, and for the first time the channel was quiet enough that a message appearing in it meant something.
Noise is not neutral. It trains people, reliably and quickly, to ignore the exact channel you have chosen to shout down. Every alert you add that does not require a response is teaching your team, patiently, to miss the one that does.
– Serguey Asael Shinder
Leave a Reply