Every monitoring system that has been running for more than a year contains rules nobody remembers writing, firing at thresholds nobody chose, into a channel nobody opens. Nobody decided to stop reading it. The channel simply reached the point where thirty messages arrive on a quiet day and none require anything, so ignoring it became the rational response. The one message that mattered looked identical to the rest.
This is not a tooling problem. The good tools are mature and mostly free. It is a problem of what gets turned into an alert in the first place, and the fix starts by deleting more than you add.
Four ways alert fatigue gets manufactured
Alerting on causes instead of symptoms. A system can fail in hundreds of ways, but users experience only a handful: it returns errors, it is slow, or it is unreachable. Writing a rule per imaginable cause produces a set that grows without limit and still misses the failure nobody predicted, because the prediction was the input.
Thresholds copied from defaults. A vendor’s default is chosen to look sensible in every environment, which means it is calibrated for none. A database server that sits at eighty percent memory is doing its job. The rule that fires at seventy is describing a product decision, not your system.
Alerts nobody owns. A rule added after an incident three years ago, by an engineer who has since left, stays enabled because nobody feels entitled to remove it. Every rule needs a named owner, and if no one will claim it, that is already the answer.
Notifications routed to a channel rather than a person. A message addressed to everyone is addressed to nobody. Post an alert into a shared channel of six and you get six people each assuming one of the other five picked it up.
The test an alert has to pass
Three questions, asked before the rule is written rather than after it has been ignored for a year.
Is it urgent? If it can wait until Monday with no consequence, it is a backlog item, not a page.
Is it actionable? If the person receiving it has neither the information nor the permissions to change the outcome, the notification produces anxiety and nothing else.
Does it need a human? If the correct response is always the same sequence of steps, automate the sequence. Keep an alert only for the case where the automation fails.
Anything that fails any of the three should not interrupt a person. The signal is not discarded, it is routed somewhere else.
Disk at eighty percent, checkout returning errors
Disk utilisation at eighty percent is an observation. What matters is the gradient. Losing one percent of free space per day gives you twenty days, which is a ticket for next sprint. Losing ten percent per hour gives you two hours, which is a page. Alerting on predicted time to exhaustion rather than a fixed percentage removes an entire category of notification.
A rising error rate on checkout is an incident from the first minute, regardless of what any resource graph says. Customers do not perceive CPU utilisation. They perceive that their payment failed.
The contract you signed and the target you should hold
Three terms get used interchangeably and mean different things.
An SLI is a measurement: the proportion of requests served successfully, or served under a latency threshold, or the proportion of probes that succeed. It is collected automatically.
An SLO is the internal target for that measurement, agreed with the business rather than picked by engineering alone.
An SLA is the contractual commitment, with service credits attached when it is missed.
The important part is the gap between the last two. If the contract promises 99.9 percent monthly availability, that permits 43.2 minutes of downtime in a thirty-day month. Setting the internal SLO at the same number means the first signal that you have a problem is a customer invoking a credit clause. Set the internal target at 99.95 percent, which allows 21.6 minutes, and the team is alerted at half the contractual allowance, leaving the rest as margin. The SLA is a floor. The SLO is where you actually operate.
What the average is hiding
Mean response time is the most reassuring number on any dashboard and the least useful. If ninety-five percent of requests complete in seventy milliseconds and five percent take six seconds, the average stays under four hundred milliseconds while one interaction in twenty is unacceptable. Percentiles expose exactly the users the mean dilutes, and they are disproportionately your largest accounts, because they carry the most data and trigger the slowest queries. Set the SLI on p95 or p99, never on the average.
Error budgets make the argument arithmetic
Availability targets become useful the moment you invert them. A 99.9 percent success target against two million requests per month allows two thousand failed requests. That is the error budget, and it converts a discussion about whether the system feels reliable into a number both engineering and product can read.
With eight percent of the budget spent by mid-month, a risky migration is affordable. With ninety percent spent, priority shifts from features to stability, and that is a rule rather than an argument. A budget exhausted repeatedly is evidence for investment. A budget never touched over two quarters is evidence that the target was set too low and that money is being spent on reliability nobody asked for.
Burn rate, and why two windows are enough
Alerting on an SLO breach is alerting after the fact. Alert on the rate at which the budget is being consumed instead.
A burn rate of 1 spends the whole monthly budget exactly at month end. A burn rate of 14.4 sustained for one hour consumes two percent of the budget in that hour, because one hour is one seven-hundred-and-twentieth of a thirty-day month. Against two million monthly requests, that is roughly 2,778 requests in the hour, an error rate of 1.44 percent, and forty failures. That is a fast window, and it justifies waking someone.
A burn rate of 6 over six hours consumes five percent. That is a slow window. It does not warrant a phone call at three in the morning, but it will exhaust the budget before month end if it continues, so it becomes a ticket with a deadline.
Two rules of this shape replace several dozen threshold alerts, and unlike thresholds they stay correct when traffic doubles.
Everything that should not page
Cutting alerts does not mean discarding information. It means routing it deliberately.
Signals useful during diagnosis but not on their own go to a dashboard, opened during an incident rather than watched continuously. Work that must happen but not tonight becomes a ticket with an owner and a date. Trends go to a thirty-minute weekly review covering three things: budget consumption per service, every alert that fired falsely, and every rule that has not fired in ninety days. The first group gets fixed, the second gets deleted. Without that review the alert set only grows.
Rotations, escalation and the handover problem
Follow-the-sun coverage is genuinely better than a night rotation, since nobody debugs well at four in the morning. It also introduces a cost that is regularly underestimated: three regions on eight-hour shifts means two handovers a day, over seven hundred a year, and each one is an opportunity to drop context.
Handover quality is therefore an engineering concern, not an administrative one. A written handover, even between people sitting together, covering what fired, what is unresolved, and what is scheduled next, is worth more than any escalation policy.
The rotation itself needs a single named primary who receives the page on a device rather than in a channel, a secondary who is called automatically when the primary does not acknowledge inside fifteen minutes, and an explicit path to whoever can authorise a decision the on-call engineer cannot make alone, such as a rollback that loses data.
Every alert payload carries a link to its runbook. Not a wiki homepage, the specific document, answering what the alert means, how to confirm in two minutes whether it is real, the first diagnostic step, and who to escalate to. An alert without a runbook is an alert that gets silenced on its third firing.
Review without blame
An incident review that identifies a person produces engineers who report less. A useful one produces a timeline, the time to detect, the time to diagnose, and the property of the system that allowed a normal human mistake to have that effect. The output is one or two specific changes with owners and dates, not a commitment to be more careful.
Cardinality and retention are the real limits
Two constraints govern what you can measure at all.
Cardinality is the first. A counter labelled with 200 endpoints, 20 status codes and 15 instances produces 60,000 distinct time series. Add a customer identifier across 5,000 customers and it becomes 300 million, which will take down the monitoring system before it tells you anything. Unbounded identifiers belong in logs or traces, never in metric labels.
Retention is the second. Fifty gigabytes of logs per day is 1.5 terabytes a month and over 18 terabytes a year, and hot searchable storage is priced accordingly. Tiered retention, with a short hot window and cheaper archival behind it, covers the realistic case: almost every search happens within days of the event.
Retention also has a legal dimension that engineering teams tend not to claim. Logs contain personal data almost by default: IP addresses, user identifiers, email addresses in query strings, sometimes entire request bodies. Under GDPR that is processing, whether or not anyone decided to do it. The retention period needs to be set and documented, access to the log platform needs to be restricted to people with a reason to have it, and sensitive fields are better stripped at ingestion than discovered later during a subject access request.
Where to start
Export every alert that fired in the last thirty days and mark whether a human did anything in response. Delete the rules where the answer is no, this week. Then pick one service that customers actually notice, define one SLI and one SLO for it, and replace its threshold alerts with a fast and a slow burn rate rule.
We design and calibrate alert sets, SLOs and runbooks as part of monitoring and observability. If the requirement is that somebody is genuinely awake and accountable at three in the morning, that is managed services, and the distinction is worth settling before an SLA is signed rather than after.