An alert that doesn't require action shouldn't page anyone
Every informational alert routed to the same channel as a real page trains the on-call engineer to skim, which is exactly the habit you do not want during a real incident.
There is a common anti-pattern in observability where engineering teams try to monitor everything, and as a result, they end up alerting on everything. A database CPU spikes to eighty percent during a routine nightly backup. An autoscaling group spins up three new containers to handle a traffic surge. A data ingestion pipeline completes a large job five minutes slower than usual.
These are system events. They are useful data points for a telemetry dashboard. They are not emergencies.
Yet, in many organizations, these events trigger the exact same PagerDuty notification or drop into the same high priority Slack channel as a catastrophic database failover.
The Cost of Alert Fatigue When an engineer gets woken up at three in the morning for a temporary memory spike that the orchestrator automatically resolved two minutes later, they learn to stop trusting the monitoring system. The psychological response is immediate. They start swiping away notifications on their phone without fully reading them. They mute channels. They build mental filters.
When an actual severity one outage occurs a week later, it gets buried in a sea of meaningless noise. The incident response is delayed not because the monitoring failed to catch the issue, but because the human operator was conditioned to ignore the alarm.
The Rule of Action The criteria for configuring an alert should be brutally simple. If this alarm fires, does a human being need to drop what they are doing and immediately execute a remediation step?
If the answer is no, the alert should not exist.
If a system automatically recovers via autoscaling, do not page the on-call rotation. If a disk is at seventy percent capacity and current usage patterns show it will not fill up for another three weeks, create a low priority ticket in the backlog. Do not trigger an active incident.
Segregating the Noise Building a mature observability posture requires strict segregation of information. Dashboards are for visualizing system health and investigating trends. Logs are for debugging. Dedicated, muted channels are for auditing automated system actions.
Paging channels must be kept sacred. They should be reserved exclusively for binary states where the system is broken and requires immediate human intervention to prevent business impact.
Modern observability tools are incredibly powerful, but configuring them to scream about every minor metric creates a culture of apathy. Protect the attention of your engineering team. When the system finally does page them, they should know without a doubt that the house is actually on fire.