Notification Routing, Deduplication and Escalation

1. The real problem: every signal must not fan out to every channel

If each event is forwarded to email, chat and a pager, one underlying condition can create dozens of notifications, trigger races between responders and leave ownership unclear. This lesson is not about sending more messages. It builds Notification Routing, Dedupe & Escalation Matrix v1, where delivery decisions remain separate from the actual condition state.

2. Lesson boundary and owned artifact

Each matrix row records signal_id, notification_key, group_key, severity/urgency/actionability, owner, route, mute/inhibit state, delivery result, acknowledgment, resolution, escalation stage, deadline, retry budget, suppression reason and terminal state. Obligation Lifecycle belongs to PLN-005; Executive Docket belongs to PLN-007.

3. Condition, notification and delivery are different objects

A condition is the underlying state. A notification is a decision to attract attention. A delivery_attempt is only a channel transport event. SMTP or chat success does not prove that the owner saw the message, acknowledged it, or resolved the work. This separation is the foundation of auditable escalation.

4. Derive a stable notification key from the condition

Three adapters can emit different signal_id values for the same condition. If the notification key is built from channel or message IDs, deduplication fails. The key should come from the owned condition identity and stable dimensions. Channel selection belongs to delivery policy, not condition identity.

5. Group and deduplicate before fan-out

Prometheus Alertmanager officially supports grouping, deduplication and routing to reduce duplicate notifications. This lesson uses that as an operational reference, not as the definition of MAKNOW semantics. Three signals for one condition should stay in one group while the matrix records how many duplicates were suppressed.

6. Route by actionability, urgency and owner

High severity alone does not justify paging. Ask whether a concrete human action exists, whether an owner is known and how short the response window is. Google SRE guidance says good alerting should be timely and actionable, and non-actionable alerts create noise. Route selection therefore needs operational evidence, not merely a critical label.

7. Learn from group wait, group interval and repeat interval

Alertmanager exposes group_wait, group_interval and repeat_interval for grouped notifications. The lesson does not copy those defaults into MAKNOW. The useful idea is that first delivery, new changes inside an existing group, and reminder repetition are distinct timers and should not be collapsed into one escalation clock.

8. Mute and inhibition are not the same

A mute window suppresses delivery under a time/policy rule. Inhibition suppresses a target notification while a related source condition is active; for example, a parent outage can prevent child symptom alerts from paging separately. Suppression must keep a visible reason and must never masquerade as condition resolution.

9. Delivery success is never acknowledgment

Email can be DELIVERED while the owner has not acknowledged anything. Keep delivery_result and acknowledged_at independent. Until acknowledgment or resolution exists, the workflow is still open. This prevents transport success from being promoted into business truth.

10. Escalate only with a finite deadline and retry budget

If acknowledgment remains absent and the deadline expires, the policy may move to a backup owner or another channel. Retries must still be finite. retry_budget, escalation_stage and a terminal state prevent endless loops. Exhausted policy should become an explicit blocked/review state, not infinite repetition.

11. Resolution closes the escalation path

If the underlying condition is genuinely resolved before the next escalation deadline, another notification should not be sent merely because a timer is pending. Resolution must come from condition evidence, not delivery evidence. The matrix records RESOLVED and cancels pending escalation.

12. Worked example: three signals, two channels, one condition

An overdue obligation emits three signals in ten minutes. They share one notification key and group key, so they become one notification group. Email delivers, chat fails and acknowledgment is still missing. If a valid parent incident exists, inhibition may suppress the child page. Without inhibition, remain in WAIT_ACK until the deadline and escalate only after that trigger.

13. Keep output states distinct

DELIVER means policy permits sending. WAIT_ACK means delivery may have happened but acknowledgment has not. ESCALATE requires a finite valid trigger. SUPPRESS covers mute/inhibit/dedupe and is not resolution. DEFER_TO_DOCKET is for low-urgency actionable work that should not page. REVIEW_REQUIRED preserves missing decisive evidence, and RESOLVED requires actual condition closure.

14. Deterministic failure probes

Four shortcuts must fail: (1) the same condition from three channel adapters becomes three pages; (2) successful delivery with no acknowledgment is marked RESOLVED; (3) a valid parent incident exists but inhibition is ignored; (4) the condition resolves before the next deadline but the timer still pages. Each failure needs an auditable state and reason.

15. Transfer: a low-urgency reminder should create a docket item, not a page

Preserve the same separation in a new context: a low-urgency but actionable opportunity reminder. The notification key still comes from the condition and delivery/ack/resolution stay separate, but the route should be DEFER_TO_DOCKET. Recompute urgency and route from the new evidence rather than copying the baseline result.