Desired State and Drift Reconciliation

1. Process alive is not desired-state proof

Node N-24 can report a healthy heartbeat while its worker is still on build 7.3 even though the desired generation requires build 7.4 with config digest D7. “Process alive” is therefore only one observation. Reconciliation needs exact desired binding, a fresh observed snapshot, and post-action proof. Command success or a green service status alone does not prove convergence.

2. LOC-016 ownership boundary

This lesson owns only the Desired-State Drift Reconciliation Ledger v1: desired state → fresh observation → ownership classification → authorized/idempotent action → fresh convergence probe. Staged rollout belongs to LOC-017 and backup/watchdog proof belongs to LOC-018. Their evidence may feed a gate without moving their implementation logic into this lesson.

3. Bind desired state by generation and digest

desired_generation identifies the logical version of intent and desired_digest binds the content of that intent. A generation label without its digest can silently point at changed content. Every decision should trace to both values and their provenance before a repair is even considered.

4. Observation must be fresh

A recent heartbeat does not make the whole state snapshot fresh. observed_at plus observation_ttl determine whether package version, configuration, service settings, and policy inputs are still valid for a decision. If the snapshot is older than the TTL, repair is BLOCKED until a fresh observation exists.

5. Classify drift before acting

Not every difference is a repair target. Classify ownership for every resource_key: MANAGED for fields MAKNOW may reconcile, OPERATOR_OWNED for local overrides owned by operations, and UNKNOWN when ownership is not proven. Blindly overwriting every difference destroys the authorization boundary.

6. Preserve operator-owned exceptions

Suppose every managed field matches desired state but a locally approved maintenance flag differs. The reconcile loop must not revert it just to eliminate a diff. Record the exception and its provenance, then evaluate managed convergence separately from operator-owned state.

7. Keep validation separate from enforcement

Google VM Manager separates OS Policy Validation from Enforcement, while DSC separates Get/Test/Set operations. These are reference patterns: detect first, then evaluate authorization and preconditions, then act only inside the owned boundary. Detection itself is not authorization.

8. Actions must be idempotent and content-bound

authorized_action should bind to the resource and desired digest, while an idempotency_key makes replay of the same intent safe. If the same desired generation is replayed after convergence, the loop should not create another change; it should report that the state is already converged.

9. Retry needs a budget and backoff

Infinite retry is not reconciliation; it is hidden oscillation. Record attempt_count, retry_budget, and backoff. If a fresh post-action observation still differs and budget remains, the decision may be RETRY. When the budget is exhausted or another writer keeps changing the field, stop with BLOCKED or CONFLICT.

10. Treat a conflicting writer differently from ordinary drift

If another controller changes a managed field immediately after repair, the problem is not simply “remaining drift.” Record conflicting-writer evidence and stop the repair loop. Continuing to overwrite can create controller oscillation. CONFLICT is a real operational state, not something to hide with more retries.

11. Command success differs from convergence

A package installer may return exit code zero while the next snapshot still reports the old digest. That command is execution evidence only. post_action_observation must be fresh and convergence_status must come from observed state. CONVERGED is forbidden until owned fields match desired state.

12. Every decision needs a reason and recheck trigger

Each bounded decision needs a reason and recheck_trigger. A BLOCKED state that never says what evidence must change is not operationally useful. Examples include refreshing a stale snapshot or resolving controller ownership before re-evaluating a conflict.

13. Worked example: build 7.3 and config D7

N-24 should run build 7.4 with digest D7. A fresh snapshot shows build 7.3 plus an approved maintenance override. The build is managed drift; the maintenance override is operator-owned. Repair only the build, obtain a fresh snapshot proving 7.4 + D7, and leave the override untouched. Managed state can now be CONVERGED.

14. Deterministic failure probes

Run four mutations: make the snapshot older than its TTL while heartbeat stays healthy; change only an operator-owned maintenance field; replay the same desired generation after convergence; and report command success while the post-action digest still differs. The evaluator must independently test freshness, ownership, idempotency, and observed convergence. No shortcut may manufacture success.

15. Transfer: node returns after two days offline

The node was offline for two days and returns with an old desired generation plus a locally approved maintenance override. Refresh desired authority first, take a fresh snapshot, and reclassify ownership. Do not copy the baseline decision. If the local desired generation is stale, remain BLOCKED until intent is refreshed, and never overwrite the valid operator override.