Incident Response and Postmortems

1. Problem and provable outcome

During an incident, the fastest story is not necessarily the most accurate one. Alerts, a recent change, red dashboards, and pressure to restore service can turn temporal correlation into an assumed cause. This lesson produces Incident Response and Learning Record v1: a traceable record that keeps impact, direct evidence, hypotheses, actions, current service state, causal uncertainty, and follow-up work separate.

The bounded outcome is to run a fully synthetic local incident from detection through stabilization and learning without treating service restoration as proof of cause or replacing evidence with a retrospective narrative.

2. Lesson boundary and prerequisite

This lesson owns incident response and postmortem learning. SLOs/error budgets/RPO/RTO belong to OPS-009; chaos/failure testing belongs to OPS-011. We use adjacent concepts only to interpret impact and recovery, without absorbing their outcomes.

No production system, live endpoint, real credential, customer data, or external action is used. Every lab input is a local synthetic fixture.

3. Mental model: response is not the same as learning

Keep three outputs separate: impact reduction, recovery verification, and causal learning. A mitigation can be operationally justified while the cause remains open. A process can be alive while user recovery is unproven. A useful postmortem can identify several contributing conditions rather than one simplistic root cause.

A defensible record answers: what was observed, what was inferred, what action was taken with what risk, what actually proves recovery, and which causal claims remain open?

4. Bound impact before claiming cause

Impact is an operational claim: which users or capability, what time window, and what severity were affected. Early in the incident, cause may be unknown, but impact should be bounded by direct signals such as error rate, failed transaction count, or a latency window.

“Everything is broken” is not sufficient evidence. The record names the signal that supports impact and preserves unknowns.

5. Timeline: observation, inference, and ordering uncertainty

A timeline is more than timestamps. Each row should identify its evidence type: alert, log, trace, change event, operator action, or stakeholder update. If systems do not share a trustworthy clock, do not manufacture precise ordering.

A config change at 10:02 and an error spike around 10:02–10:03 may be correlated. “The change caused the incident” remains inference until stronger evidence supports it.

6. Keep triage hypotheses falsifiable

A triage hypothesis guides probes; it does not lock the story. State what observation would strengthen it and what observation would weaken or refute it. When contrary evidence appears, the hypothesis changes; the timeline is not rewritten to preserve the old story.

Example: “If routing change R is causal, a bounded rollback on route A should reduce route-A errors in the next window without changing route B.”

7. Mitigation is not root-cause closure

Incident response may prioritize reducing impact. A bounded rollback, traffic shift, or local feature disable can be justified before causal analysis is complete. The mitigation still needs scope, operational risk, rollback path, and expected evidence.

When mitigation succeeds, record “impact decreased.” Do not write “the cause was fixed” unless causal evidence supports that claim.

8. Verify recovery independently

Recovery must be supported by a service-relevant signal, not process liveness alone. In this lesson, valid evidence can combine error rate returning to baseline, synthetic transaction success, and stability across an observation window.

If telemetry is incomplete, the correct state is RECOVERY_UNPROVEN. Missing evidence is not permission to invent confidence.

9. Communication: Known / Suspected / Next update

Stakeholder communication should distinguish Known, Suspected, and Next action / next update. This prevents early hypotheses from becoming public causal claims. Impact and current service state should be clearer than speculative technical detail.

Example: “Known: checkout failure rate elevated. Suspected: recent routing config is correlated but unproven. Action: bounded rollback on route A. Next update after the recovery-verification window.”

10. Blameless postmortems

Blameless does not mean accountability disappears. It means the analysis focuses on system conditions, available information, guardrails, interfaces, runbooks, and reviewable decisions rather than indicting a person.

Rewrite “Engineer X pushed the bad config” into a system/evidence question: “What conditions allowed the change to reach this path with insufficient validation, and what signal or review could have detected it earlier?”

11. Contributing causes and causal uncertainty

Incidents can have multiple contributing conditions: change risk, observability gaps, timeout interactions, or runbook ambiguity. Forcing one root cause often discards useful information.

Record each contributing factor with evidence and bounded confidence. Plausible-but-unproven explanations remain OPEN. Service restoration is an observation; causal closure is a separate analytical claim.

12. Incident Response and Learning Record v1

The artifact contains incident_id, impact, detection signal, timeline rows, hypotheses, mitigation decision and risk, stakeholder updates, recovery evidence, contributing factors, causal unknowns, postmortem trigger, follow-up actions, owner, evidence-of-done, and recurrence sensor.

This artifact is a MAKNOW instructional synthesis; NIST and Google do not mandate this exact form. Its value is traceability: every conclusion should point back to evidence.

13. Failure probe and transfer practice

The first case places a change event deliberately close to the error spike to tempt the shortcut “latest change = cause.” Contrary evidence must refute or bound that shortcut. The transfer case uses a different initiating change, incomplete telemetry, and a communication constraint; the first causal story cannot be copied.

Transfer passes only when the new record is rebuilt from raw fixture evidence and new unknowns remain explicit.

14. Assessment and acceptance criteria

Four critical dimensions are incident-evidence timeline, mitigation/recovery distinction, blameless causal learning, and follow-up closure. Blocking answers must show a bounded mitigation, recovery independently verified or explicitly unproven, and action items with owner, evidence-of-done, and recurrence sensor.

“The incident is fixed” is insufficient because it proves neither current state, cause, nor recurrence-risk reduction.

15. Sources, freshness, and claim limits

NIST SP 800-61 Rev.3 was finalized in April 2025 and integrates incident-response recommendations into broader cybersecurity risk management aligned with CSF 2.0. Google SRE treats postmortems as learning artifacts and emphasizes blamelessness, contributing causes, incident impact, response actions, and preventive follow-up.

This lesson uses those sources for principles and boundaries, not as a claim that one universal incident runbook exists. Sources were checked on 2026-09-19; a material authority or fixture change requires a new qualification cycle and package version.