A runbook nobody has tested is a guess with formatting
Runbooks rot the exact same way documentation does. If a mitigation guide has not been used in an actual production incident or a scheduled drill in the last six months, you have to assume at least one critical step is fundamentally wrong.
When a platform team writes a runbook for a database failover or a cache restoration, it is perfectly accurate on the day it is published. The CLI commands work, the internal dashboard links resolve, and the required permission groups are correct. But infrastructure is not static. Over the next two quarters, the database engine gets upgraded, the authentication provider rotates its tokens, and the monitoring team migrates the dashboards to a new observability tool. The underlying system drifted, but the static markdown file sitting in a wiki stayed exactly the same.
The Incident Room Discovery Discovering this drift during a critical outage is an operational nightmare. An on-call engineer gets paged at two in the morning for a severe storage latency spike. They open the runbook, copy the first mitigation command, and paste it into their terminal. The command fails with a syntax error because the internal CLI tool deprecated that specific flag three months ago.
The engineer is now forced to debug the runbook itself while the actual production environment continues to degrade. A document designed to reduce cognitive load during a crisis suddenly becomes an active liability. The team loses valuable mitigation time trying to figure out which parts of the guide are still relevant and which parts are historical fiction.
Proving the Mitigation To prevent this decay, operational mitigation guides must be treated with the same rigor as application code. You cannot just write them and forget them.
Engineering organizations must mandate regular game days or chaos engineering drills. You intentionally simulate a failure in a staging environment and force the on-call rotation to blindly follow the existing runbook without relying on tribal knowledge. If a step fails, the drill stops, and the documentation is immediately patched.
Furthermore, any runbook step that can be automated into an executable script should be. Human processes decay silently. A static document will never tell you it is broken. A mitigation plan is only valid if you have recently proven that it works. If you have not tested it, you do not have a runbook. You just have a formatted guess.