Your monitoring is not broken: it is shouting

There is one figure that turns up in every industry report and that almost nobody reads to the end. Vectra AI puts the average organisation at 2,992 security alerts a day in 2026, and says 63% of them go unaddressed. The striking part comes next: the volume has been falling for three years (4,484 in 2023, 3,832 in 2025, with no public figure for 2024) and the unaddressed share has not moved. Filtering harder has fixed nothing, because the problem was never how many alerts arrive.

It is worth saying where that number comes from before using it: these are security alerts, from operations centres watching for threats. Our patch is wider and duller (the Zabbix that watches the nodes, the probe asking about latency, the backup job that took twice as long last night), and there we have no industry statistic to wave around. What we have is inherited platforms. When we take over one that has been running on its own for years, the pattern we usually find is this: a pile of sensors, a pile of emails, and an Outlook folder with a rule sending them straight to read.

The number that falls and the problem that does not

The series is the useful part: 4,484 daily alerts in 2023, 3,832 in 2025 and 2,992 in 2026. A third less noise in three years. That is exactly what every product bought to "reduce noise" promises, and it worked. And still, most of them go unread. The same count cites Microsoft and Omdia's State of the SOC 2026, with two figures pointing the same way: 46% of alerts turn out to be false positives and 42% are never investigated.

An honest aside: 63% from one source and 42% from the other do not add up, and we are not going to pretend they do. They measure different things with different samples, and that in itself says something about how hard this is to measure. What survives the discrepancy is the order of magnitude: between four and a little over six out of every ten warnings end up untouched. If that figure were 10% in your shop, this would be a different conversation. It rarely is, in any shop we have seen.

An alert with no owner is a log with pretensions

Monitoring usually gets designed backwards. You start from "what can I measure", which today is essentially everything, and end up with a beautiful dashboard nobody looks at because something is always red. We make each alert earn the right to exist, and for that we want four answers before it is created:

The rule is deliberately hard: if a condition does not have all four answers, the alert is not created. It stays as a metric, gets drawn on a panel and is reviewed when it is time. Measuring a lot does no harm; turning every measurement into an interruption somebody will have to learn to ignore does.

"CPU at 90%" is not a problem, and that is the problem

The fixed threshold is the biggest noise factory we know. A virtualisation node at 90% CPU during the nightly backup window is doing precisely what it was bought to do. A disk at 85% says little either; what says something is how fast it got there. And the alert fires anyway, every night, until somebody silences it "for a couple of days".

The alternative is not exotic and has been in the tools you already own for years. Zabbix documents two prediction functions, timeleft() and forecast(): the first works out how long until a metric reaches a threshold, the second estimates what value it will reach later on, both by fitting a model over history. With that, the alert stops being "the disk is at 85%" and becomes "at this rate, this disk fills up on Thursday". The first interrupts you; the second gives you three days to ask for a quote. Same metric, two entirely different products.

The other piece almost nobody configures is trigger dependencies. When a node goes down, the box should send one email saying the node is down, not forty from the forty services that lived on it. You configure it once, and it separates a fault from an avalanche. Along those lines, the official Zabbix integration in Proxmox saved us a fair bit of handiwork, but it does not save you judgement: the template gives you the data, the decisions are still yours.

What wakes us at night (the list is short on purpose)

This is our list, not a universal truth, and it shifts with what the client has at stake. But the criterion travels: wake someone for what already hurts, or for what will hurt before anyone reaches the office.

• A user-facing service down, not a component down. The difference matters: one node fewer in a cluster with headroom is not urgent; the application that does not answer is.
• Losing the second leg, not the first. A power supply, a link or a replica going down is the system working as designed. Losing the one that was left is another matter, and almost nobody has that alert configured.
• Cluster quorum at risk. Here the margin between "fine" and "everything stops" is one node wide.
• A backup that cannot be restored, which is not the same as a failed backup job. A job finishing green while verification says otherwise is the worst kind of silence.
• Network degradation the user already notices. Not the ping, which nearly always answers: packet loss and sustained latency.
• A high-confidence security signal from EDR/MDR: blocked execution, a new privileged account, mass deletion. High confidence, not generic "suspicious activity".

Everything else (temperatures, spikes, planned reboots, certificates sixty days out, pending updates) goes into the morning report. It is a deliberate decision, and a hard one to defend in a meeting: everything promoted to the list above takes attention away from the rest. An on-call rota with six reasons to ring works. With sixty, it stops ringing altogether.

The fault that trips no threshold

There is one recurring phone call: "it is slow". Everything is green, the ping answers, CPU is calm, no service is down. And the user is right. It is nearly always 2% packet loss with jitter on the link, or a route that changed and now goes the long way round: nothing a binary "responds / does not respond" threshold will ever see. That is why we measure latency continuously and read it as a series, not as a yes or a no; it is why we maintain our own SmokePing Docker image. As a network operator with our own network, we see this from both sides: the customer calling, and the BGP and NetFlow telling us where the traffic actually goes.

Silencing well is not looking away

There is a half-moral idea going around that silencing an alert is cheating. We hold the opposite view: silencing is part of the design. Declared maintenance windows so a planned reboot wakes nobody. Dependencies so the cause speaks and the symptoms shut up. Grouping so one incident is one warning and not two hundred. Hysteresis so a metric dancing around the threshold does not produce twenty "problem" and "resolved" pairs in an hour.

What is cheating is the silence with no end date. The alert somebody muted "until Monday" eight months ago is the most dangerous debt in a platform, because from the outside it looks a great deal like coverage: the sensor is there, the panel says green and nobody will find out. So every silence gets an expiry and a written reason. When it expires, either the cause gets fixed or somebody consciously decides that thing is no longer watched. Both are legitimate answers. "I forgot" is not.

How you actually prune this on a Tuesday

When we inherit a platform that shouts, we redesign nothing on day one. We pull the last thirty or ninety days of history, group it by trigger and sort by how many times each one fired. The list always has the same shape: a handful of triggers accounts for most of the volume, and the long tail barely registers. That is where the leverage is.

For each of those, one of four decisions and nothing else: fix the cause (the disk that fills every week does not need a better alert, it needs log rotation); change the condition (minimum duration, hysteresis, dependency, prediction instead of threshold); demote it to a report; or delete it. Yes, delete it. Monitoring accumulates by default because removing things is frightening, and that fear is exactly what produces platforms with four thousand alerts a day.

And a success metric that is not the obvious one. Chasing "fewer alerts" is achieved by switching things off and means nothing. What we watch is the share of alerts that led to an action. If that goes up, monitoring is improving even if the total stays put. We applied the same criterion a couple of weeks ago to prioritise patches, when July's Patch Tuesday brought 622 vulnerabilities at once: sorting the list was the whole job.

And we review this quarterly, because monitoring left unpruned grows back on its own: every new project brings its sensors, every scare brings its souvenir alert, and in a year you are back where you started.

The 24/7 that is not 24/7

Here comes the uncomfortable part, and we say it even though it is our own business being questioned. Receiving an alert at three in the morning is not 24/7 support. It is an email at three in the morning. The service starts where somebody reads it, decides and touches something. If monitoring sends three hundred emails a day to a generic address, what you have built is a folder.

The five questions we would put to any on-call contract, ours included: does the alert reach a specific person on rota or a shared mailbox? how quickly does somebody commit to responding, and is it written down? what can that person touch without calling anyone — restart a service, migrate a machine, shut a port — and what not? is there a written procedure per alert type, or is it improvised? And the one that stings: when did somebody last test the whole chain, at night, with a fake alert?

A closing line without a moral. Take last week's alerts and count how many made somebody touch something. That number (not the sensor count, not the dashboard count, not the vendor logo) is your monitoring. The rest is decoration with notifications.

Sources (verified): the average of 2,992 daily security alerts in 2026, 63% unaddressed, and the series 4,484 (2023) and 3,832 (2025): Vectra AI, "Alert fatigue: causes, real cost, and how to fix it". The 46% false-positive and 42% uninvestigated figures are attributed to Microsoft and Omdia's State of the SOC 2026 and we took them from that same Vectra page: we have not verified the original report first-hand, so we present them as a second-hand citation. The prediction functions timeleft() and forecast(), with their fitting models over history: official Zabbix documentation, "Predictive trigger functions". Our use of Zabbix and SmokePing, our own network with BGP and NetFlow, and the philosophy of alerts that matter are ours. The four-question test, the list of what wakes us and the five on-call contract questions are ours too, not anybody's doctrine.