Lease Protocol: Ack, Renew, Cancel, and Progress

1. The real problem: delivery is not ownership forever

A coordinator can deliver a job and still have no durable proof that a worker remains authorized to execute it. Network loss, retries, stale messages, reassignment, and cancellation can create two apparent owners or revive work that already expired. This lesson produces Lease Lifecycle & Progress State Machine v1: an auditable contract that separates delivery, ACK, active authority, renewal, progress, cancellation, expiry, and terminal result.

2. LOC-011 owns the lease lifecycle boundary

LOC-011 owns lease identity, generation, holder binding, ACK deadline, expiry, renew sequence, progress sequence, cancellation state, and terminal reconciliation. Worker enrollment and credential generation belong to LOC-009; heartbeat/inventory health belongs to LOC-010; durable offline spool/replay belongs to LOC-012. A Kubernetes Lease is a useful coordination reference, but it does not define MAKNOW job semantics.

3. A lease needs identity, holder, and bounded time

At minimum record job_id, lease_id, lease_generation, holder_identity, issued_at, lease_duration, and expires_at. The holder is authorized only for that lease generation and bounded interval. Reusing a lease id without generation, or treating a worker name as permanent ownership, makes stale traffic indistinguishable from current authority.

4. Kubernetes provides a reference model, not the whole protocol

Kubernetes LeaseSpec exposes fields such as holderIdentity, acquireTime, renewTime, leaseDurationSeconds, and leaseTransitions. Its duration is measured against the last observed renewTime. Those semantics illustrate explicit holder and renewal timing. MAKNOW adds its own ACK, progress, cancellation, generation, and terminal-result rules; do not claim Kubernetes proves those design choices.

5. Delivery and ACK are different states

A delivered job is merely offered. ACK means the worker received and accepted the offer before ack_deadline; it does not mean the job succeeded. A useful transition is OFFERED → ACKED → ACTIVE. If ACK never arrives before the deadline, the coordinator can expire or reassign according to policy. Never promote ACK directly to COMPLETED.

6. Renewal extends only the current generation

Every renewal binds to lease_id + lease_generation + holder_identity + renew_sequence. A valid renewal arrives before expiry, belongs to the current holder/generation, and advances the accepted renew sequence. It updates renew_time and recomputes expiry from the lease contract. A renewal from generation 7 must not extend generation 7 after generation 8 is already active.

7. Expiry is an authority boundary

When the lease expires, the previous holder no longer has current execution authority under this contract. If the coordinator assigns a newer generation, old traffic can still be retained for audit, but it is not allowed to revive ownership. Clock source and timestamp provenance matter: expiry decisions must use the coordinator's declared time policy, not an untrusted worker clock alone.

8. Progress is monotonic evidence, not authority

A progress message binds to the active lease generation and carries a progress_sequence. Accept a progress update only when its generation is current and its sequence advances the accepted progress sequence. If sequence 12 arrives after sequence 14, keep the newer state and record the late/out-of-order message. Progress never renews a lease unless the protocol explicitly couples those actions—and this lesson keeps them separate.

9. Cancellation needs request, acknowledgement, and reconciliation

cancel_requested records coordinator intent; cancel_ack records worker acknowledgement. Neither automatically proves the process stopped. While disconnected, the worker may produce a late progress event. Preserve that event as late/stale evidence, keep cancellation/expiry policy authoritative, and require an explicit terminal reconciliation such as CANCELLED, FAILED, or a bounded unresolved state.

10. Terminal result closes work; ACK does not

A terminal result must bind to the active lease identity/generation and be accepted only if it satisfies the terminal-result contract. SUCCESS, FAILED, or CANCELLED are different from delivery ACK. If a lease was ACKed but expired before any valid terminal result, the correct record is still non-success: ownership expired and completion remains unproven.

11. Stale and duplicate messages are first-class outcomes

Do not silently discard evidence or reinterpret it as current. Record stale_generation, late_progress, duplicate renews, and duplicate terminal attempts with explicit audit events. Idempotent duplicate traffic for the same accepted sequence may be ignored safely, but a lower generation or contradictory holder must never mutate the active lease.

12. Ownership conflict must block optimistic execution

If two different holders claim the same active generation, or coordinator state contains contradictory active holders, preserve ownership_conflict and stop automatic promotion. The coordinator must resolve which lease record is authoritative before more work is treated as valid. A high-confidence heartbeat from one worker cannot erase contradictory lease evidence from another.

13. The evidence matrix makes the decision reproducible

For every case record: Job ID, Lease ID, Generation, Holder Identity, Issued At, Lease Duration, Expires At, ACK Deadline, ACK State, Renew Sequence/Time, Progress Sequence/Value, Cancel Requested/ACK, Terminal Result, Stale Generation, Late Progress, Lease Expired, Ownership Conflict, Decision, and Reason. A reviewer should be able to reproduce the state transition without hidden reasoning.

14. Decisive failure probes

Reject these shortcuts: (1) generation 7 expires and sends a late renew after generation 8 is active → keep generation 8 and record stale rejection; (2) worker ACKs but never returns a terminal result → ACK is not completion and expiry remains authoritative; (3) progress sequence 12 arrives after 14 → never move progress backward; (4) cancellation is requested during disconnect and late progress arrives → preserve cancellation/expiry policy and require terminal reconciliation. The learner lab hides final labels while testing these invariants.

15. Transfer to a reordered streaming transport

Replace request/response polling with a streaming channel that can reorder progress and renewal messages. Keep lease identity/generation, holder binding, expiry, monotonic progress, cancellation, and terminal semantics unchanged. Recompute each message from its own identity, generation, sequence, and timestamp; do not trust transport arrival order as authority. A reordered channel may change audit timing, but it must not change who owns the lease.