Fleet Rollout: Test to Canary to Batch to Fleet

1. Install success is not fleet-rollout authority

A release can be healthy on one test node and fail at scale because hardware, operating-system versions, workloads, and network conditions differ. Command success or a timer is therefore execution evidence, not sufficient promotion evidence. Every rollout decision must be bound to an exact release digest, a declared cohort, a completed health window, and explicit budgets.

2. LOC-017 ownership boundary

This lesson owns only the Progressive Fleet Rollout & Halt Ledger: TEST → CANARY → BATCH → FLEET. Per-node drift classification and reconciliation belong to LOC-016; restore and watchdog proof belong to LOC-018. LOC-017 may consume those observations as gate evidence without absorbing their implementation logic.

3. Lock the release before every cohort

rollout_id alone is not enough. Every stage must carry the same approved release_digest and release_generation. If the digest changes between canary and batch, the canary evidence no longer proves the new artifact. The stage becomes BLOCKED until evidence is rebuilt for the new binding.

4. TEST is only the smallest blast radius

TEST is useful for obvious failures and basic health checks, but it is not representative of the whole fleet. A healthy TEST may permit entry into CANARY only when baseline and post-update probes are recorded and the health window completes. A successful TEST never authorizes FLEET directly.

5. Define CANARY by cohort and health window

The canary needs an explicit selector, planned nodes, reachable nodes, and observation window. Google Cloud Deploy models canary as progressive phases before stable and can attach analysis before advancement. MAKNOW uses this only as a reference pattern: promotion must come from your own fleet evidence, not from a product label.

6. Disruption budget bounds concurrency

VM Manager illustrates zone-by-zone or concurrent-zone rollout with a disruption budget. In the MAKNOW contract, disruption_budget bounds tolerated simultaneous disruption in a cohort. If disrupted or failed nodes exceed that budget, the rollout must not continue merely because other nodes are healthy; emit HALT or, when rollback evidence is ready, ROLLBACK.

7. BATCH must be bounded and repeatable

After CANARY, divide the fleet into bounded batches. Reapply the same digest, selector discipline, health probes, failure budget, and pause point to every batch. Azure VM Scale Sets also models rolling updates as batches constrained by health limits; this is a control reference, not a ready-made MAKNOW policy.

8. Reachable is not the same as planned

Record planned_nodes, reachable_nodes, and updated_nodes separately. Do not silently remove unreachable nodes from the denominator to make success rates look better. If an exclusion is authorized, record the node, reason, owner, and recheck point; otherwise promotion is BLOCKED.

9. Complete the health window

A green probe immediately after installation can be transient. health_window needs an explicit duration and close condition, with baseline compared to post-update health. If a canary becomes unhealthy just before the window closes, earlier positive signals do not override the failure.

10. Failure budget differs from disruption budget

Disruption budget limits simultaneous impact; failure budget limits tolerated failed outcomes for promotion. Concurrency can remain compliant while the second batch exceeds the failure budget. In that case the rollout must halt or roll back; a successful first batch is not permission to continue.

11. Promotion is evidence-driven

Emit PROMOTE only when artifact binding is stable, cohort evidence is complete, the health window is closed, updated/reachable counts reconcile, unhealthy nodes remain within policy, and no decisive contradiction is open. At FLEET stage, emit CONVERGED only after the required evidence closes; do not claim fleet success earlier.

12. Define HALT and ROLLBACK in advance

Record halt_condition and rollback_target before rollout. HALT stops advancement pending recheck or repair. ROLLBACK means changed nodes must return to the prior target and rollback evidence must itself be recorded. A rollback command without a target and verification is only intent.

13. Worked example: late canary failure

Release R42 is healthy on TEST. A two-node CANARY initially looks healthy, but one node shows abnormal restarts minutes before the health window closes. The installer returned success on both nodes, yet the post-update health gate failed and the failure budget is zero. Promotion is wrong; the bounded decision is ROLLBACK or HALT, leaving the fleet untouched.

14. Deterministic failure probes

Run four mutations: make one canary unhealthy just before the health window closes; leave 20% of a batch unreachable; change the digest between canary and batch; and exceed the failure budget in the second batch after the first passed. Every mutation must produce a typed state, an audit_event, and a clear next action. Installer success cannot erase any of them.

15. Transfer: two sites with intermittent connectivity

In transfer, split the fleet across two sites while Site B has intermittent connectivity. Recompute cohort size, disruption budget, and pause/recheck points, and keep unreachable nodes explicit. If Site B evidence is incomplete, do not copy the baseline decision; remain BLOCKED until recheck or an authorized, documented exclusion changes the evidence. The contract stays stable while environment-specific values change.