Backup/Restore Smoke and Independent Watchdog

1. A backup file is not recovery readiness

A backup artifact proves only that something was stored. Recovery readiness requires a backup bound to a known source generation and digest, an isolated restore, structural and application-level checks, and auditable outcomes. Liveness also needs supervision outside the workload being supervised.

2. LOC-018 ownership boundary

This lesson owns only the Backup/Restore Smoke and Independent Watchdog evidence contract. Routine drift reconciliation belongs to LOC-016 and progressive rollout belongs to LOC-017. A valid restore does not prove watchdog detection, and a restart does not prove restored-data correctness.

3. Bind the backup to source content

Record at least backup_id, source_generation, source_digest, backup_created_at, and backup_digest. A filename or file-existence check is not enough. Without source generation or digest, the backup cannot be tied confidently to a particular source state.

4. What SQLite Online Backup proves

The SQLite Online Backup API copies database content into another database and can operate incrementally so the source need not remain locked for the entire copy. That mechanism can create a useful snapshot, but API completion alone does not prove application-level restore correctness.

5. Restore smoke must be isolated

Restore into a separate target so the recovery exercise does not overwrite live state. Record restore_target, start/completion timestamps, and isolation_confirmed. Restoring into the live path turns a smoke test into a destructive operation and weakens the evidence boundary.

6. Integrity check is one proof layer

PRAGMA integrity_check performs low-level SQLite consistency checks and returns ok when it finds no errors. Foreign-key errors require the separate foreign_key_check. Even an ok result does not prove business invariants or application behavior.

7. Run an application smoke after structural checks

After restore, evaluate at least one observable invariant such as required schema, a bounded record count, or a known state marker. Record application_smoke_result and expected_invariant. If the invariant fails, the outcome is RESTORE_FAILED even when integrity_check returned ok.

8. Re-bind restored content with digests

Connect source, backup, and restored artifacts with content evidence. source_digest, backup_digest, and optionally restored_digest expose corruption or artifact mix-ups. A file can keep the same filename after corruption; the name is not proof.

9. The watchdog decision must live outside the workload

A heartbeat generated and interpreted only by the workload is not independent supervision. watchdog_owner must represent an external supervisor, such as a service manager. The workload must not both fail and control the evidence that declares itself healthy.

10. WatchdogSec and the heartbeat deadline

With systemd, WatchdogSec defines the watchdog deadline and the service sends WATCHDOG=1 keep-alives. Missing the deadline must produce an auditable event. Process existence or a responsive UI is insufficient because an event loop can be wedged while the process still exists.

11. Restart budget must be finite

Automatic restart without a bound can create an endless failure loop. Record restart_count and restart_budget. A restart may be allowed while budget remains; after exhaustion, the state must become WATCHDOG_EXHAUSTED or escalate rather than retry forever.

12. Post-restart readiness is an independent gate

Process restart is not recovery success. Run readiness or an application probe again. If post_restart_readiness=false, do not emit RECOVERED; escalate or halt explicitly.

13. Worked example: valid backup, missed heartbeat

A node has a content-bound backup and the isolated restore passes every check. Later, the worker misses its heartbeat; an independent supervisor restarts it while budget remains, and post-restart readiness passes. The evidence pack now has two separate claims: RESTORE_PROVEN for data and RECOVERED for the watchdog path.

14. Deterministic failure probes

Test four mutations: corrupt one restored page while keeping the same filename; return integrity_check ok while the application invariant fails; keep the process alive while heartbeat exceeds the deadline; exhaust restart budget while readiness remains false. Every mutation must produce a typed state and auditable reason.

15. Transfer: Windows instead of systemd

In transfer, change the data directory and replace systemd with Windows service failure actions. Preserve the contract: content-bound backup, isolated restore, structural and application checks, independent supervisor, deadline, restart budget, post-restart readiness, and escalation. Platform mechanism changes; evidence requirements do not.