Repair Patch Classes, Canary and Rollback

1. Problem: a green test is not a safe rollout

A prior incident has been reproduced and a candidate patch exists. The common shortcut is to treat a few passing tests as permission for a broad rollout. This lesson instead produces an auditable release-safety artifact: Patch Classification, Canary & Rollback Plan v1.

2. Lesson boundary

PLN-028 does not own initial diagnostics; it consumes reproduction_ref from PLN-027. It also does not own ARC-001 venture architecture. Its owned outcome is patch classification, canary/control, observation, promote/abort gates, and rollback.

3. External FACT versus local DESIGN

Google SRE defines canarying as a partial, time-limited deployment that is evaluated to decide whether rollout should proceed. Kubernetes documents rollout history, rollback, and a multiple-Deployment canary pattern. Patch classes, thresholds, and typed states in this lesson remain MAKNOW DESIGN.

4. Classify the patch before rollout

Use at least CODE, CONFIG, DEPENDENCY, DATA_SCHEMA, and POLICY. A reproduced cause may justify COMPOSITE_BOUNDED when multiple surfaces must change atomically. Do not combine unrelated repairs and attribute one canary outcome to both.

5. Identity: patch, incident, and change set

Keep incident_id, patch_id, and change_set_hash distinct and linked. If the change set changes after the canary plan is defined, prior canary evidence is no longer an oracle for the new bytes and the plan must be reevaluated.

6. Reproduction reference is an entry gate

reproduction_ref must point to repeatable evidence for the same failure. A new unit test is useful, but without the reproduced failure link it does not prove that the patch addresses the observed cause. Missing reference → HOLD.

7. Verify rollback before canary

The rollback target must identify a known-good state. Feasibility is more than “a command exists”: record what is reverted, what is not, and how post-rollback health will be verified. Unknown target or infeasible rollback → HOLD/BLOCKED.

8. Keep canary and control concurrent

The canary is a limited subset while the stable control remains observable during the same period so signals are attributable. A before/after comparison alone is insufficient for this MAKNOW contract. Cohorts need explicit identity and scope.

9. Observation window comes from the failure mode

There is no universal window. Fast failures such as crashes may surface quickly; delayed failures such as cache drift or queue buildup need longer observation. A window shorter than the failure latency yields INCONCLUSIVE/HOLD.

10. Predeclare success and abort signals

Success signals state what must improve or remain healthy. Abort signals state what immediately blocks promotion. Healthy aggregate averages do not cancel an active abort signal.

11. Error-budget guard and promotion gate

Promotion is allowed only after the observation window completes, success signals pass, abort signals remain clear, the guard is not violated, and rollback is still verified. Every decisive gate matters; there is no majority vote.

12. Baseline — bounded composite patch

Incident INC-482 reproduces a dependency regression coupled to one routing condition. Both changes share one cause and one rollback target, so COMPOSITE_BOUNDED is defensible. Canary=5%, Control=stable 95%, Window=30m; Success=error-rate parity + no queue regression; Abort=canary error delta >2pp or crash. All gates clear → PROMOTE.

13. Three dangerous shortcuts

(1) Combine two unrelated repairs in one canary → attribution is lost and state is HOLD. (2) Start without a verified rollback target → HOLD. (3) Promote while an abort signal is active because global averages look healthy → ROLLBACK.

14. Transfer — config patch with delayed failure

The transfer patch is CONFIG and its rollback action is immediate, but the failure signal appears after about 120 minutes during cache refresh. The 30-minute baseline window is invalid. Recompute to Window=180m, post-rollback verification=180m, and a delayed cache-mismatch abort signal. If the full window completes cleanly, PROMOTE can be supported.

15. Lab, assessment, and freshness

Run python run_checks.py under lab/PLN-028-LAB1/learner/. Eight fixtures cover Baseline, Missing Reproduction, Missing Rollback, Abort Active, Mixed Unrelated Repairs, Stale Evidence, Short Delayed Window, and Transfer. External FACT claims were rechecked at the recorded date; canonical package changes require a new version.