Heartbeat, Inventory, and Health Telemetry

1. The real problem: a fresh heartbeat is not a healthy node

A fresh heartbeat only proves that a sender emitted a signal within a recent time window. The same node may have stale inventory, a NOT_SERVING worker service, low disk headroom, memory pressure, version drift, or telemetry from an obsolete identity generation. This lesson produces Node Heartbeat, Inventory & Health Evidence Matrix v1: a contract that keeps liveness, inventory, service health, and resource observations separate and makes the health decision no stronger than the evidence.

2. LOC-010 owns the telemetry boundary

LOC-010 owns heartbeat evidence, observation/receipt time, inventory snapshot and generation, service instance/version/health, resource observations, freshness, and health synthesis. Enrollment and credential generation belong to LOC-009; leases and work authority belong to LOC-011; durable offline spooling belongs to LOC-012. Telemetry describes observed state and never authorizes a privileged action by itself.

3. Telemetry must bind to the correct identity generation

At minimum record worker_id, identity_generation, heartbeat_sequence, observed_at, and received_at. A recent timestamp does not make an old identity current. If re-enrollment advanced the worker to generation 8, a generation-7 heartbeat must remain stale or quarantined even when its timestamp is fresh.

4. What OpenTelemetry Resource and service identity contribute

OpenTelemetry Resources describe the entity that produced telemetry. Service semantic conventions distinguish the logical service from service.instance.id and record service.version separately. These semantics help correlation and provenance, but they do not define the MAKNOW node-health policy; health still comes from the bounded contract in this lesson.

5. Heartbeat is liveness evidence only

Define a liveness_window and compute heartbeat_age under a declared clock policy. A heartbeat inside the window may yield LIVE; an older or unavailable one yields STALE or UNKNOWN. LIVE never means inventory is current, the service is serving, or resources are safe. Transport keepalive is not service health either.

6. Inventory changes more slowly but has its own freshness

Inventory can include host_id, OS/architecture, CPU and memory capacity, service version, and an inventory generation. Its cadence may be slower than heartbeat cadence, but inventory_age and inventory_generation remain explicit. If local policy says 30 minutes and the snapshot is six hours old, preserve INVENTORY_STALE instead of presenting the old values as current.

7. gRPC Health is a service signal, not a whole-node verdict

gRPC Health defines Check and Watch, while the application updates SERVING or NOT_SERVING. That signal answers whether a service is ready to accept requests. A fresh heartbeat alongside NOT_SERVING must lower the bounded health state; liveness is not allowed to override service health.

8. Resource observations need provenance and missing-signal semantics

CPU, memory, disk free, and process uptime are example observations. Record when each value was observed, from which source, and under which local policy. If disk_free is decisive but absent, do not invent a value. Preserve UNKNOWN or DEGRADED and name the recheck trigger.

9. Semantic conventions are not an eternal schema

OpenTelemetry system conventions still contain Development-status metrics and process conventions include Release Candidate metrics. Treat those names as evolving reference conventions, not as an immutable MAKNOW schema. Version the MAKNOW telemetry contract so upstream changes do not silently rewrite historical meaning.

10. Version drift is its own evidence dimension

Compare service.version and inventory generation with the local desired/expected version policy. Version drift does not necessarily make the node dead, but it can degrade health or restrict admission. Record it explicitly rather than confusing “the service responds” with “the expected version is running.”

11. Resource pressure is evaluated against local policy

This lesson invents no global CPU, memory, or disk threshold. The fixture or deployment policy supplies the relevant floor/ceiling. If disk free is below the local floor or memory pressure crosses the local ceiling, preserve DEGRADED even when heartbeat and service health are fresh. A number without policy is only an observation.

12. Health synthesis must fail closed or degrade

A bounded decision may be HEALTHY only when identity generation is current, heartbeat is fresh, inventory is current, the service is SERVING, and decisive observations exist and satisfy local policy. A stale, missing, or conflicting decisive dimension yields a typed non-success state such as STALE_REJECTED, INVENTORY_STALE, UNKNOWN, or DEGRADED. Averaging signals must not hide failure.

13. The Evidence Matrix makes the decision reproducible

For each case retain Worker ID, Identity Generation, Heartbeat Sequence, Observed/Received At, Heartbeat Age/Liveness, Inventory Generation/Age, Service Instance/Version/Health, CPU, Memory, Disk Free, Process Uptime, Missing Signal, Version Drift, Resource Pressure, Health Decision, and Reason. A reviewer should be able to reproduce the decision without hidden reasoning.

14. Decisive failure probes

Break four shortcuts: (1) heartbeat fresh + service NOT_SERVING must not be HEALTHY; (2) heartbeat/service fresh + inventory older than its TTL preserves INVENTORY_STALE; (3) a fresh heartbeat from the previous identity generation is rejected or quarantined; (4) all visible signals look good but required disk_free is missing, so the result remains UNKNOWN/DEGRADED rather than imputed. The learner lab does not expose final labels.

15. Transfer from push heartbeat to pull/streaming collection

Change transport without changing the evidence contract. Pull/streaming telemetry must preserve source identity, identity generation, observation time, received time, freshness windows, inventory generation, service health, and missing-signal semantics. A new sampling cadence changes ages, so the health decision must be recomputed from the new evidence instead of copied from the baseline.