Side-node Power Orchestration

1. The real problem: idle does not mean safe to power off

Assume N-GPU-2 is ON_IDLE. It has no active job, has been idle for 25 minutes, and energy cost makes shutdown attractive. The dangerous shortcut is to let that timer directly trigger power-off. Several independent questions still matter: did new qualified demand appear, is drain actually clear, is decisive telemetry fresh, did minimum-on and cooldown windows expire, is a shutdown inhibitor active, is the requested power control supported and policy-approved, and did the host actually reach OFF after the request? The owned artifact is Side-node Power Orchestration State Machine & Safety Record v1.

2. Lesson boundary: consume scheduler and telemetry evidence without absorbing them

AIE-019 owns bounded node-power transition policy, control intent, confirmation, and audit evidence. Scheduler feasibility and placement belong elsewhere; telemetry collection and device-health semantics belong to AIE-017. The power controller consumes those versioned inputs, closes admission at the correct point, requests a transition, verifies observed power, and separately verifies runtime readiness. Motherboard, PSU, BMC, firmware, and electrical design are outside scope.

3. Never collapse desired state, observed state, and runtime readiness

desired_power_state=ON is controller intent. observed_power_state=ON means a trusted observation says the host is powered. runtime_readiness=READY is stronger: the workload runtime has passed the defined readiness probe. Likewise, an accepted shutdown request is not evidence that the machine is already off. Keeping these states separate prevents a control acknowledgement from becoming a false operational success.

4. Discover node capabilities and allowable actions before requesting a transition

When a system implements Redfish, the ComputerSystem.Reset action can expose values such as On, GracefulShutdown, GracefulRestart, ForceRestart, and ForceOff, and a service can advertise the allowable values for that system. The controller should therefore discover the node-specific capability and intersect it with local policy. Redfish support on one system does not prove Redfish support on another, and an action present in a standard is not automatically approved for your operational policy.

5. Wake-on-LAN is a capability surface; a wake request is not readiness proof

Linux ethtool netlink exposes Wake-on-LAN get/set operations and WOL mode data for network devices. That helps establish whether an OS-visible WOL capability is supported or enabled. It does not prove that a magic packet will traverse the network, wake the machine from the current power state, initialize the GPU, or make the runtime ready. After wake intent, confirm observed power and then run the readiness probe as a separate gate.

6. Safe shutdown begins with admission control and drain evidence

Before power-off, the controller must do more than observe active_job_count=0; it should recheck qualified demand and prevent new admission so a race is not created. Kubernetes provides a useful current reference pattern: during graceful node shutdown, kubelet stops accepting new Pods and terminates running Pods within configured shutdown grace periods. MAKNOW does not require Kubernetes; the engineering lesson is that admission closure and workload drain must precede an irreversible power transition.

7. Fresh telemetry is a gate, not dashboard decoration

If the idle timer expires while decisive telemetry is STALE, lack of evidence must not be coerced into a safe-shutdown claim. Record freshness with timestamp and policy/version provenance. The bounded decision is DEFER or BLOCKED until decisive evidence is refreshed. AIE-019 does not redefine health metrics; it only refuses to power-transition from stale evidence.

8. Minimum-on, minimum-off, and cooldown windows prevent thrash

Rapid power cycling can add latency, churn, and operational instability. A policy should separately define minimum time on, minimum time off, and cooldown after transitions. The idle threshold does not supersede those constraints. If the idle threshold is 20 minutes but the machine has not satisfied a 30-minute minimum-on window, the correct decision remains DEFER even when the queue is empty.

9. Treat shutdown inhibitors as operational safety evidence

systemd supports inhibitor locks that can block or delay shutdown and sleep. Normal shutdown paths should respect them unless an explicit exceptional policy authorizes otherwise. Forced shutdown paths can skip normal service shutdown, and stronger force modes can risk data loss. In this lesson, ForceOff is never the default energy optimization. An active inhibitor should normally produce BLOCK/DEFER, with any override separately authorized and auditable.

10. Recommended state model: intent → guards → request → confirmation

A useful state model distinguishes OFF, WAKING, ON_IDLE, ACTIVE, DRAINING, SHUTTING_DOWN, and protective states such as UNKNOWN/BLOCKED. For shutdown, evaluate demand/drain/freshness/timing/capability/inhibitor guards, close admission, recheck demand, create a transition intent, request graceful shutdown, and move to OFF only after observed confirmation. For wake, qualified demand plus timing/capability gates lead to a request, then WAKING, observed ON, a readiness probe, and only then reopened admission.

11. Handle races by re-evaluating before the irreversible step

Suppose shutdown has entered DRAINING and a high-priority job becomes runnable for N-GPU-2 during the grace window. The controller must refresh evidence before irreversible power-off. If the host is still recoverably on, cancel or abort the pending shutdown, verify the resulting observed state, and reopen admission only after readiness is confirmed. Continuing the old timer decision despite new demand is a race bug, not successful automation.

12. Bound timeout, retry/backoff, and UNKNOWN states

A power-control endpoint can acknowledge a request without producing the requested state. A host can also power on while the runtime never becomes ready. Each transition needs a deadline, retry budget, and backoff rule. At deadline, do not manufacture success. If retry budget remains, preserve DEFER/WAKING/UNKNOWN and record the next recheck; if capability or confirmation can no longer be established, use a bounded blocked/unavailable reason. Infinite retry only hides failure.

13. What the Safety Record must preserve

Record at least: node_id, power_intent_id, transition_id, desired/observed/readiness states, control method and allowable action, queued demand, active jobs, drain/admission state, telemetry freshness, idle/min-on/min-off/cooldown windows, inhibitor state, request timestamp, deadline, retry/backoff, request result, state confirmation, ready probe, race flag, manual override, final decision, reason, power_policy_version, and provenance. The record must show which evidence proves intent and which evidence proves reality.

14. Worked decisions and counterexamples

Baseline shutdown: node is ON_IDLE, demand and active jobs are zero, drain is clear, telemetry is fresh, idle/min-on/cooldown gates pass, inhibitor is clear, and GracefulShutdown is declared and approved; admission closes, demand recheck stays clear, the request is accepted, and observed state later confirms OFF → SHUTDOWN. Baseline wake: qualified demand exists, min-off/cooldown pass, control is approved, observed state reaches ON, and readiness probe passes → START and reopen admission. Counterexamples: stale telemetry → DEFER; active inhibitor → BLOCK; unsupported control → BLOCK; wake acknowledgement without readiness → DEFER, not START.

15. Transfer and final checklist

Transfer 1: new demand appears during the shutdown grace window. Rebuild the decision, cancel the pending shutdown while recoverable, and reopen admission only after verified ON+READY. Transfer 2: wake is acknowledged but readiness is not confirmed by deadline. Preserve WAKING/UNKNOWN or a bounded DEFER/BLOCK outcome and do not bind the job to that node. Final checklist: identity and policy version bound; demand/active-job evidence fresh; drain/admission explicit; telemetry fresh; min-on/min-off/cooldown passed; inhibitor clear; capability and allowable action proven; request separated from confirmation; deadline/retry bounded; race recheck performed; readiness independently verified; only then issue the final power decision.