SLO-Driven Incident Response: From Alert Fatigue to Actual Reliability

Pick a user journey that matters ('checkout submission'). Define an SLI — a measurable quantity like 'p95 latency < 500ms AND status < 500'. Set an SLO — the target success rate over a window, e.g. 99.5% over 30 days. The gap between 100% and your SLO is your error budget. Every failure spends from it. When you're on track to burn the whole budget before the window ends, that's when to page someone.

This one paragraph replaces every 'server is at 90% CPU' alert you have.

The classic 'error rate > 1%' alert has two failure modes: it pages for 30-second blips, and it stays quiet for hours-long slow degradations. Multi-window burn-rate alerts fix both:

Fast burn — 2% of the 30-day budget in 1 hour:
  short_window(5m) AND long_window(1h) both above 14.4x baseline error rate
  → page immediately (Sev1/Sev2)

Slow burn — 10% of the budget in 6 hours:
  short_window(30m) AND long_window(6h) both above 6x baseline
  → ticket, address in business hours (Sev3)

The two windows together mean brief spikes don't page (short window OK) and sustained degradations don't hide (long window catches them). Google's SRE workbook has the full derivation; every mature on-call rotation I've worked with runs some version of it.

Write down the criteria for each severity so on-call doesn't have to invent them at 3am:

• Sev1 — major user-visible outage, revenue impact, or data loss risk. Page immediately, all-hands bridge, exec notification within 15 min.
• Sev2 — significant degradation for a segment of users, or a Sev1 waiting to happen. Page primary on-call, updates every 30 min.
• Sev3 — minor degradation, workaround exists, or internal only. Ticket, handled next business day.

The specific labels matter less than agreeing on them and using them consistently. Post the criteria in your incident channel topic.

For every Sev1/Sev2, spin up a dedicated Slack channel and follow the same pattern:

1. First message: one-liner summary and current impact.
2. Assign an Incident Commander (coordinates, does not fix), a Comms lead (updates status page and stakeholders), and a Scribe (keeps timeline).
3. Every status change (mitigation started, mitigation working, back to normal) is posted with a timestamp.
4. No debugging tangents in the main channel — spin off threads and report back.
5. Close with 'incident resolved at HH:MM UTC, review scheduled for [date]'.

This looks like ceremony until you've been in a 6-hour incident where nobody wrote anything down. Then it looks like sanity.

Within a week of every Sev1/Sev2, hold a blameless review. Publish a written doc that has:

• Summary: 3-5 sentences, what happened and what the user saw.
• Timeline: minute-by-minute from detection to resolution, with actor names.
• Contributing factors: not 'root cause' — real incidents have several. What conditions had to be true for this to happen?
• What went well: seriously. Positive reinforcement matters.
• What went poorly: without blaming individuals.
• Action items: specific, assigned, dated. Not 'be more careful' — 'add pre-deploy check X, owned by Y, due Z'.

Blameless means you can name what a person did without shaming them for it. The system enabled the mistake; the fix is in the system.

Track and review each quarter:

• Number of Sev1/Sev2 incidents.
• MTTR (mean time to resolution) by severity.
• Percentage of action items completed by due date.
• SLO attainment across each service.
• On-call load (pages per person per week) and its trend.

If action items aren't closing, incidents will repeat. If on-call load is climbing, engineers are burning out. These are engineering metrics as important as feature velocity.