Background Scheduling for Updater

When “the timer fired” is not evidence that the updater ran once

Three updater workers restart after a six-hour outage. A daily calendar occurrence is overdue. Two workers notice the same due job, while the durable record for the previous occurrence is initially unclear. If both workers interpret “the scheduler fired” as authorization to execute, the updater can duplicate semantic-diff work, overload an external source, or create conflicting run history.

This lesson builds Updater Scheduling & Lease Contract v1. The trigger is only one input. A safe decision also needs missed-run policy, bounded jitter, a single valid lease owner, an idempotency key, bounded retry/backoff, durable last/next-run evidence, result state, and a named reviewer. When those fields disagree or remain unknown, the correct answer may be HOLD rather than “run now.”

Route orientation and ownership boundary

MKN-009 owns background scheduling for the updater. MKN-007 is the hard conceptual prerequisite and MKN-008 is the soft prerequisite. The lesson does not redo Documentation Monitoring & Semantic Diff from MKN-008 and does not absorb speech timing from MKN-010. Its bounded question is: given a due updater job, what durable scheduling, lease, idempotency and retry evidence is required before one bounded execution attempt can be reviewed?

Foundation learners receive a complete field map and one guided outage scenario. Repair learners isolate one broken field/state and rerun only that gate. Experienced learners must defend catch-up, jitter, lease TTL and retry trade-offs without treating a named scheduler feature as sufficient proof.

Mental model: trigger → durable occurrence → lease → idempotency/history → execution → result → retry or next run

Use this chain:

trigger due → identify scheduled occurrence → apply missed-run policy → acquire one lease → verify idempotency/run history → execute bounded updater → record result → retry/backoff or compute next run → release/expire lease

The trigger plane answers “when should work be considered due?” The execution plane answers “which worker, for which occurrence, may attempt it?” The evidence plane records what actually happened. Keeping these planes separate prevents a scheduler callback from becoming false exactly-once evidence.

Foundation contract: eighteen auditable dimensions

job_id, schedule_type, schedule_expression, and timezone identify the schedule. last_run_at and next_run_at preserve durable run chronology. missed_run_policy states whether missed occurrences are skipped, coalesced, or represented by one bounded catch-up. jitter_window records how herd behavior is dispersed.

lease_id, lease_owner, and lease_expires_at bind one current owner to the occurrence. idempotency_key binds semantic work to an occurrence so retries and duplicate deliveries can be recognized. attempt, retry_budget, and backoff_state bound failure recovery. run_state and result_ref record what the job believes happened, and review_owner preserves accountability.

These fields are evidence. They do not magically make a scheduler exactly-once; they make ambiguity visible and auditable.

Deep explanation: FACT, INFERENCE, and DESIGN must not collapse into one claim

systemd timer semantics — FACT. Timer units support monotonic trigger forms such as OnBootSec= / OnUnitActiveSec= and real-time calendar triggers with OnCalendar=. For calendar timers, Persistent=true can cause an immediate activation when an occurrence was missed while the timer was inactive. RandomizedDelaySec= can spread activations over a window to reduce synchronized load spikes.

Operational interpretation — INFERENCE. None of those trigger features proves that business work committed exactly once. A scheduler can deliver again after restart, multiple instances can observe the same due occurrence, and an execution can fail after an external side effect but before durable local result recording. Therefore last/next-run, lease, idempotency, attempt and result evidence should be persisted independently from the schedule expression.

MAKNOW policy — DESIGN. Catch-up policy, lease TTL, retry budget, backoff and review states are local authority. We may use systemd as an implementation reference, but we do not import its behavior as our concurrency or correctness policy.

Worked examples: outage recovery, duplicate acquisition, and short-interval transfer

Baseline outage recovery. A daily calendar updater was offline for six hours. The raw state first shows an overdue occurrence and unclear prior outcome, so execution is held. The run-history/idempotency record is then reconciled to NO_COMMITTED_RESULT. Exactly one worker acquires lease lease-302-01, the missed policy is SINGLE_BOUNDED_CATCH_UP, and the occurrence uses one durable idempotency key. The resulting state is BOUNDED_CATCH_UP_REVIEW_REQUIRED—not “exactly once guaranteed.”

Invalid shortcut. Two workers both claim an unexpired lease for the same occurrence. Even if they have the same cron expression, the contract returns HOLD_CONCURRENT_LEASE_CLAIM. Scheduler correctness does not repair ownership ambiguity.

Transfer. Change the job from a daily calendar trigger to a five-minute interval in a multi-instance deployment. The old daily catch-up rule is not copied. The new fixture coalesces stale interval ticks, narrows jitter, computes a slot-specific idempotency key, and re-evaluates lease and retry evidence. The result is INTERVAL_RUN_REVIEW_REQUIRED only after those changed fields are explicit.

Failure modes that must fail closed

Hold execution when two workers claim the same active lease, the previous outcome is unknown, the lease expired while the old outcome is unknown, the idempotency key is missing, or retry budget is exhausted. Hold when next_run_at is inconsistent with last_run_at, when jitter exceeds the declared window, when a required backoff state is absent on a retry, or when review_owner is missing.

A restart after a missed calendar occurrence also requires an explicit missed-run policy. Persistent=true may deliver a catch-up activation in systemd, but it does not decide whether MAKNOW should backfill every missed logical occurrence, coalesce them, or skip to the next schedule.

Alternatives and trade-offs: calendar, interval, catch-up, jitter, and lease duration

Calendar scheduling is useful when the business meaning is tied to wall-clock time, but it introduces timezone/DST/missed-run questions. Interval scheduling is useful for regular polling, but the policy must define whether the next interval is anchored to start time, completion time, or a stable slot.

Catch-up improves freshness after downtime but can create load spikes or repeated work. Skip/coalesce limits backlog but may sacrifice intermediate observations. Jitter reduces herd behavior, but too much jitter can violate freshness expectations. Long leases reduce overlap risk but delay recovery from dead workers; short leases recover faster but require heartbeat/renewal or stronger idempotency. There is no universally correct number—the contract must state the chosen boundary and what evidence justifies it.

Experienced bridge: scheduling is a distributed state machine

Treat every scheduled occurrence as an identity-bearing state transition, not an anonymous callback. A robust implementation normally needs an atomic lease or comparable mutual-exclusion primitive, a durable occurrence/idempotency identity, and result recording that tolerates retries. “Exactly once” is usually the wrong promise; a better design is at-least-once delivery plus idempotent/transactionally guarded effects, or an explicitly held state when the effect cannot be safely repeated.

The lab stays deliberately offline and synthetic. It verifies the decision contract, not a particular database lock or queue product.

Guided and independent practice

1. Open the outage baseline and identify schedule identity, missed-run policy, current lease, idempotency key and retry budget. 2. Mutate the fixture so two workers acquire the same due occurrence and explain which field/state becomes blocking. 3. Restart after a missed calendar run with missed_run_policy=UNKNOWN; do not invent catch-up behavior. 4. Expire a lease while the previous outcome is unknown and determine why a new attempt cannot safely proceed. 5. Exhaust retry budget while another schedule becomes due and keep the new due event from bypassing the exhausted budget.

For independent practice, remove the fixture’s expected decision. Recompute it from visible evidence only.

Verification evidence: what must another reviewer reproduce?

A reviewer must be able to reproduce: which occurrence is due; whether calendar/interval semantics are explicit; what happens to missed occurrences; whether jitter is inside its declared bound; whether exactly one non-expired lease owner exists; whether the idempotency key identifies this semantic occurrence; whether prior outcome is known enough to retry; whether attempt is within retry budget; whether backoff is valid; what the current run/result state says; and who owns review.

The deterministic lab probes duplicate lease acquisition, unknown missed-run policy, unknown prior outcome, lease expiry, exhausted retries, missing idempotency, invalid chronology, jitter overflow, missing backoff/owner, and forbidden baseline-oracle reuse in transfer.

Assessment and transfer: daily calendar → five-minute multi-instance interval

Assessment has four parts. First, state the owned boundary and one non-goal. Second, build the baseline Updater Scheduling & Lease Contract v1. Third, diagnose the two-worker duplicate-acquisition mutation. Fourth, change to a short interval job in a multi-instance deployment.

The transfer must recompute missed-run behavior, jitter, occurrence identity, lease semantics and retry/backoff. Reusing SINGLE_BOUNDED_CATCH_UP merely because it worked for the daily calendar fixture is a failure. The baseline result is never an oracle for a changed schedule model.

References, freshness, and authority boundaries

The systemd timer semantics used for K1/K2 were rechecked on 13 September 2026. Current documentation still distinguishes monotonic and OnCalendar timers; Persistent=true remains calendar-only missed-run catch-up behavior; and RandomizedDelaySec= remains a load-spreading mechanism. Recheck by 2027-01-01 or earlier on material timer-semantic change.

All exact MAKNOW choices—missed-run policy names, lease requirements, idempotency policy, retry budgets, backoff and bounded decision states—remain local DESIGN authority, not systemd requirements.