Capability-aware Scheduler

1. The real problem: “a GPU is available” is not enough for placement

Assume AI/media workload W-GPU-7 requires exact artifact sha256:D-GPU, runtime profile R3, at least 24 GiB of device memory, a compatible accelerator, license state ALLOW, and a policy of no external egress. Three nodes advertise GPU capacity: N1 has only 16 GiB; N2 has 48 GiB but the required artifact replica is QUARANTINED; N3 has 24 GiB, the verified artifact, a compatible runtime, and valid policy clearance. A scheduler that sees only “GPU present” or a single aggregate score can place incorrectly. The owned artifact is Capability-aware AI Workload Scheduling Decision Record v1.

2. Lesson boundary: consume requirements and policy; do not own telemetry or power

AIE-016 consumes immutable artifact/runtime requirements from AIE-013 and license/policy state from AIE-015 without redefining them. It does not create live GPU health or utilization semantics; AIE-017 owns Device Telemetry. It does not power nodes on or off; AIE-019 owns Side-node Power Orchestration. This lesson owns only bounded eligibility, scoring, placement, provenance, and recheck behavior.

3. The workload contract must be exact, versioned, and comparable

Before looking at nodes, normalize the request into explicit requirements: workload_id, artifact_digest, runtime_requirement, license_gate, accelerator_requirement, memory_requirement, policy_constraint, and any other genuinely hard constraint. Phrases such as “a good GPU” or “works with CUDA” are not auditable contracts. Each requirement needs a stable identity/version/policy reference so the final decision can explain why a node passed or failed.

4. Declared capability is not the same as live telemetry

Static capability says what a node has declared and qualified: accelerator type, admission-safe memory capacity, runtime/driver capability, policy domain, artifact replica state, and freshness of that metadata. Live telemetry says what the device is doing now. Low utilization cannot compensate for insufficient memory, incompatible runtime, or a quarantined artifact. Telemetry may later influence load-aware preference scoring, but it is not a substitute for hard capability evidence.

5. Kubernetes device plugins provide a useful specialized-resource advertisement pattern

Current Kubernetes documentation describes device plugins as a framework for advertising specialized hardware resources such as GPUs, NICs, and FPGAs to the kubelet, with workloads requesting those resources through the resource model. This is useful evidence that capability should be explicit; it does not prove that every label or resource name in your environment is fresh, trustworthy, or sufficient for qualification. MAKNOW therefore keeps provenance and freshness in its own capability record.

6. Labels, selectors, affinity, taints, and tolerations steer placement; they do not prove full qualification

Kubernetes can label nodes by accelerator type, use node selectors or affinity as requirements/preferences, and use taints/tolerations to reserve specialized nodes for suitable workloads. But accelerator=h100 alone does not prove memory size, artifact digest, runtime profile, license clearance, or freshness. A label is one evidence dimension, not a final authorization shortcut.

7. Core rule: hard gates first, score second

The current kube-scheduler model filters nodes that do not meet requirements, scores the feasible nodes, then binds the selected node. MAKNOW makes the separation explicit: artifact, runtime, license/policy, accelerator, memory, and required static capability are hard gates, and no soft score may compensate for a failed hard gate. If N2 fails because its artifact replica is quarantined, a perfect locality score cannot bring it back into competition.

8. Type hard-gate results: PASS, FAIL, UNKNOWN, and STALE

A gate is not merely Boolean. PASS means sufficient valid evidence; FAIL means a requirement is explicitly violated; UNKNOWN means required evidence is absent; STALE means evidence exists but exceeds the freshness policy. FAIL removes the node from the feasible set. UNKNOWN/STALE must not be coerced into PASS; if no other feasible node exists, the bounded outcome is usually DEFER until qualification or refresh completes.

9. Artifact locality matters only after integrity and replica state

A local copy can be a useful preference only when it is the exact required digest and the replica state is VERIFIED. QUARANTINED, digest mismatch, or unknown provenance is a hard failure. Once a node is feasible, verified locality may improve the score by reducing transfer/startup cost; locality never overrides integrity.

10. Evaluate runtime, accelerator, and memory independently

The right accelerator does not prove runtime compatibility, and a compatible runtime does not prove enough memory. Keep at least three independent checks: accelerator/type compatibility, runtime/driver capability, and memory capacity. The lesson fixture intentionally rejects N1 even though it has the right GPU class because it has only 16 GiB. Compressing these into one vague label destroys failure explanation and requalification triggers.

11. License and policy are eligibility gates, not preferences

If license state is REVIEW_REQUIRED or BLOCKED, or a no-egress workload conflicts with the node policy domain, the node must not win because it has a short queue or a stronger GPU. Permission and policy are not scoring preferences. Record policy version and source state in the decision record so downstream systems can audit the authority under which placement occurred.

12. Score only feasible nodes

After hard gates, you may score preferences such as verified artifact locality, queue pressure, expected startup latency, energy preference, or a separate load signal. The scoring policy should be versioned and explainable, and it must run only over feasible_nodes. If two nodes are fully eligible, score can choose the better one. If one node hard-fails, its score must not participate in selection at all.

13. Three bounded outcomes: PLACE, UNSCHEDULABLE, and DEFER

Emit PLACE when at least one node passes every hard gate and the chosen node has a complete binding record. Emit UNSCHEDULABLE when all candidates have decisive known hard failures and no decisive uncertainty remains. Emit DEFER when there is no feasible node now but at least one candidate remains unresolved because capability/policy evidence is UNKNOWN or STALE. This prevents “not yet known” from being misreported as “impossible.”

14. Lab: eight scenarios that break common shortcuts

Case 1 is the baseline: N1 fails memory, N2 fails artifact quarantine, and only N3 is placed. Case 3 is the transfer: N3 capability metadata becomes stale while N2 gains a verified replica and compatible runtime, so placement moves to N2. Case 4 shows that low utilization or a friendly label cannot replace fresh capability evidence. Case 8 is a score trap: an invalid node has a much higher preference score, but because it fails a hard gate it never competes in scoring.

15. Transfer and final checklist

For the transfer, do not copy the previous winner; rebuild the feasible set from the changed evidence. Checklist: exact workload contract; artifact digest and replica state; runtime/driver capability; accelerator and memory; license/policy gate; capability freshness; hard-gate result per node; explicit feasible/uncertain/hard-failed sets; scoring only over feasible nodes; bounded PLACE/UNSCHEDULABLE/DEFER; binding record with scheduler version and reason; recheck trigger; and no claim that live utilization substitutes for declared capability.