Device Telemetry for GPU and Media Workloads

1. The real problem: low utilization does not prove a healthy GPU

Assume N-GPU-2/GPU-0 reports only 38% GPU utilization. It may look healthier or more available than another GPU at 91%, yet the first device can simultaneously have 95% framebuffer usage, near-warning temperature, and a recent XID sample. The second can be busy while memory, thermals, power, and error evidence remain healthy. The owned artifact is GPU/Media Device Telemetry Health & Scheduling Signal Record v1: a bounded evidence contract that prevents one convenient percentage from becoming a global health claim.

2. Lesson boundary: own telemetry evidence, not scheduling policy or remediation

AIE-017 owns metric identity, collection evidence, availability, freshness, provenance, and a bounded derived telemetry state. Static feasibility and placement policy remain AIE-016 territory. Power transitions and remediation remain outside this lesson; AIE-019 owns power orchestration. This lesson may emit explicit scheduler and operations signals, but downstream systems must consume them under their own versioned authority and may not treat telemetry as autonomous remediation permission.

3. Every sample needs identity, unit, time, and window

A numeric value without a contract is difficult to audit. At minimum, preserve node_id, device_id, driver_version, workload_label, metric_id, metric_unit, sample_timestamp, sample_window, metric_value, metric_available, and telemetry provenance. Collector/exporter version and configuration should also be traceable when they affect the result. A value such as 50 becomes meaningful only when bound to an exact metric such as GPU temperature in degrees Celsius with a known sample time.

4. The current DCGM Exporter surface is broad, but it is not your product policy

Current NVIDIA DCGM Exporter documentation lists shipped metrics including GPU temperature, board power, total energy, PCIe traffic, GPU utilization, encoder/decoder utilization, framebuffer free/used, and XID errors. That inventory describes a tool surface, not a universal definition of health. MAKNOW should preserve exact metric semantics and then apply an explicit versioned product policy; a vendor metric alone does not authorize HEALTHY, PLACE, or remediation.

5. Configured does not mean emitted: preserve UNKNOWN

NVIDIA explicitly notes that selecting a field does not guarantee runtime emission on every system; GPU/entity support, driver, DCGM, permissions, and deployment configuration affect availability. Therefore, an absent metric is not zero. If temperature disappears after a collector change, do not substitute 0 or silently reuse the last healthy value. Record UNKNOWN/UNAVAILABLE and require requalification of collection or mapping evidence.

6. Collection cadence, scrape cadence, and freshness are separate concepts

DCGM Exporter watches selected fields and serves Prometheus metrics at /metrics; its collection interval controls the requested DCGM watch cadence, and optional workload labels can be attached. Scraping the endpoint does not necessarily force a fresh hardware query. The MAKNOW contract must therefore preserve sample timestamp/window and calculate freshness_state under an explicit policy. A still-visible value becomes STALE once it exceeds the policy's maximum age.

7. Keep utilization separate from capacity, pressure, and health

GPU utilization tells you about recent engine activity, not free framebuffer, thermal state, power pressure, or device incidents. A scheduler may use utilization as load evidence, but health must be multidimensional. The lab intentionally rejects the shortcut lower utilization = healthier: the 38% device becomes DEGRADED because of memory/thermal/error evidence, while a 91% device can be BUSY_BUT_HEALTHY.

8. Model framebuffer memory pressure independently

AI and media workloads can fail to allocate or experience churn when framebuffer memory approaches saturation even while GPU utilization is modest. Preserve used/free memory or an explicitly derived ratio with unit and policy threshold. That threshold is a MAKNOW design choice, not an NVIDIA universal constant, and must be bound to policy_version. Missing or stale memory evidence remains UNKNOWN/STALE rather than silently healthy.

9. Model thermal and power pressure with fresh evidence and versioned thresholds

Temperature and board power are important pressure signals, but warning and critical thresholds belong to a declared product policy. In this synthetic lab, warning pressure produces DEGRADED, while critical thermal/power evidence produces QUARANTINED/UNSCHEDULABLE. These numbers are teaching fixtures, not official limits for a real GPU model. The engineering rule is to bind value, unit, sample time, and policy version together.

10. Media engines matter: encoder or decoder saturation can be the real bottleneck

Video pipelines can have moderate GPU-core utilization while encoder or decoder engines are saturated. The current DCGM Exporter metric inventory includes encoder and decoder utilization fields. Preserve those signals separately from GPU utilization. In the synthetic policy, media-engine saturation without health pressure yields BUSY_BUT_HEALTHY: the device is not declared unhealthy, but scheduling should account for existing media load.

11. Do not collapse XID, ECC, or health evidence into utilization

Device incidents are a different evidence layer from load. A recent XID can require investigation even after utilization falls. In this lesson, an XID observed in the synthetic window yields DEGRADED and ops_signal=INVESTIGATE; a production policy may classify specific incidents more strongly, but that mapping must be explicit and versioned. Utilization dropped to zero is not evidence that an error condition disappeared.

12. Derived state must be deterministic, versioned, and no stronger than evidence

The lab uses a deterministic precedence: required metric unavailable → UNKNOWN; available but stale → STALE; critical thermal/power evidence → QUARANTINED; error incident or warning memory/thermal pressure → DEGRADED; healthy evidence with high GPU/media load → BUSY_BUT_HEALTHY; otherwise → HEALTHY. This is a synthetic policy, not a vendor standard. Every result must trace to the exact telemetry_policy_version and input samples.

13. Emit scheduler and operations signals separately

Placement and operations need different signals. In the lab, HEALTHY → AVAILABLE, BUSY_BUT_HEALTHY → AVAILABLE_WITH_LOAD, DEGRADED/UNKNOWN/STALE → DEFER, and QUARANTINED → UNSCHEDULABLE. Operations may receive NONE, OBSERVE_LOAD, INVESTIGATE, CHECK_COLLECTOR, REFRESH_TELEMETRY, or QUARANTINE. These are bounded evidence outputs, not authority to power-cycle, repair, or otherwise remediate the node automatically.

14. Worked decisions and counterexamples

Case 1: 38% utilization, 95% framebuffer, warning temperature, and a recent XID → DEGRADED/DEFER/INVESTIGATE. Case 2: 91% utilization with healthy memory/thermal/error evidence → BUSY_BUT_HEALTHY/AVAILABLE_WITH_LOAD. Case 3: healthy evidence and low load → HEALTHY/AVAILABLE. Case 4: temperature and power disappear after a collector change → UNKNOWN/DEFER/CHECK_COLLECTOR, not last-value reuse. Case 5: temperature exists but is too old → STALE/DEFER. Case 7: synthetic critical thermal evidence → QUARANTINED/UNSCHEDULABLE.

15. Transfer and final checklist

The transfer challenge is deliberate: the collector changes, utilization keeps arriving, but temperature and power stop emitting. Rebuild the record from the changed evidence; mark the missing signals UNKNOWN and do not copy the previous decision. Final checklist: exact device/workload identity; metric ID and unit; sample time/window; availability distinct from value; versioned freshness policy; utilization separated from memory/thermal/power/media/error evidence; missing never coerced to zero; deterministic derived state; separate scheduler and ops signals; provenance and recheck trigger; no autonomous remediation claim from telemetry alone.