AI Cost Governor and Circuit Break

1. The real problem: a correct workflow can still be economically unsafe

A workflow may be schema-correct, model-correct, and output-correct while retries, expensive fallbacks, or one tenant exhaust a budget. A monthly alert reacts after the spend. A circuit breaker alone does not constrain cost. This lesson therefore maintains two independent state families: budget/quota state for whether consumption is allowed and backend circuit state for whether a backend should receive traffic.

The owned artifact is AI Budget Governor & Backend Circuit State Policy v1. It prechecks the allowed worst case against remaining budget, bounds retries/fallbacks, and reconciles actual usage after a response.

2. Separate request, job, consumer, and backend identity

A request_id identifies one attempt; job_id aggregates a chain of attempts/fallbacks; consumer_id owns a quota or allocation; backend_id identifies the serving path. Collapsing these counters into one number makes it impossible to distinguish a costly job, an exhausted consumer quota, and an unhealthy backend.

Every decision should also carry policy_version and the exact counter scope so an audit can reproduce which rule applied to which identity.

3. Budget is not quota

A budget is an economic or consumption boundary for a request/job/consumer. A quota may have its own time window and counter. A fixture can have a healthy backend with a CLOSED circuit while the consumer quota is exhausted. That must be QUOTA_BLOCKED, not CIRCUIT_OPEN and not MODEL_INELIGIBLE.

Azure API Management currently documents separate token rate and token quota behavior in llm-token-limit; exceeding the rate returns 429 and exceeding the quota returns 403. This is a vendor example, not the universal definition of our policy. cite:AZURE-LLM-TOKEN-LIMIT

4. Never collapse estimated and actual usage

Before a call, the governor usually has only an estimate. After a response, valid provider usage can update state with actual tokens. Azure documents both estimated prompt-token behavior and actual usage from model responses, including accuracy/concurrency caveats. cite:AZURE-LLM-TOKEN-LIMIT

MAKNOW rule: estimated_tokens supports precheck; actual_tokens supports post-call reconciliation. Provenance must state where each number came from.

5. Hard and soft budgets create different decisions

A hard_budget is a boundary a new call must not exceed under the allowed worst case. A soft_budget changes behavior: shorten/degrade, select a cheaper route, defer, or require approval. Reaching the soft threshold is not a backend failure.

Keep remaining budget reconstructable from committed actual usage and any reserved worst-case amount. If the next call's allowed worst case does not fit, hoping for lower actual usage is not an authorization mechanism.

6. Retry is bounded, not a right

Retry requires a retryable failure class, retry_count < retry_budget, enough budget for another attempt, and a circuit state that permits that attempt. Retry-After can inform timing; it does not increase retry budget or hard budget.

If a backend returns a retryable 429 with Retry-After but retry budget is exhausted, the governor must stop retrying.

7. Keep circuit state independent from spend state

Use at least CLOSED, OPEN, and HALF_OPEN. CLOSED allows normal calls from the circuit perspective; OPEN blocks normal traffic; HALF_OPEN permits only a bounded probe. Azure API Management documents backend circuit-breaker failure conditions, trip duration, and accepting Retry-After. cite:AZURE-AI-GATEWAY cite:AZURE-BACKEND-CIRCUIT

A closed circuit does not prove model qualification or budget availability. Those are separate gates.

8. Decision order: precheck before call

A practical flow is identity → quota → hard/soft budget → route eligibility input → circuit → retry/fallback budget → call → actual usage/failure evidence → state update → final action. Preserve each gate's reason.

Quota exhaustion should not be relabeled as circuit failure. Circuit-open blocks a primary call even when money remains. A worst-case budget miss is a cost-governance decision even when quota and circuit are healthy.

9. Fallback can hide amplification

Fallback is often treated as rescue, but it may use longer prompts, larger context, or a more expensive path. Track fallback_count and fallback_amplification. A fallback is new consumption, not a free continuation of the original request.

Before switching, compare the fallback's allowed worst case with remaining hard budget. Healthy and qualified are necessary but not sufficient.

10. Worked example: retryable backend, retry not allowed

A job has a 30k hard budget and has consumed 22k actual tokens. The primary backend returns 429 with Retry-After=10s. Policy allows at most two total attempts and both have already been used. Budget may remain, but retry_state=EXHAUSTED.

The correct result is a policy-bounded DEFER or another eligible route, not a third retry. Record RETRY_BUDGET_EXHAUSTED rather than mislabeling it as circuit-open.

11. Worked example: quota exhausted, circuit healthy

A consumer has exhausted its daily allocation while the backend circuit is CLOSED and healthy. Reporting BACKEND_FAILURE would corrupt telemetry and remediation. Correct state is quota_state=EXHAUSTED, circuit_state=CLOSED, final_action=HARD_STOP.

A later quota window or explicitly changed policy may allow new work; repeated attempts must not bypass the current quota.

12. Lab: eight deterministic fixtures

The lab covers: allowed baseline, quota exhausted, retry budget exhausted despite Retry-After, circuit open, actual usage crossing soft budget, fallback worst-case larger than remaining hard budget, a worst-case precheck that blocks a call, and a bounded HALF_OPEN probe.

Implement evaluate(case) returning exactly eight fields: budget_state, quota_state, circuit_state, retry_state, usage_evidence_state, fallback_state, final_action, and reason_class. The oracle stays admin-only.

13. Forbidden shortcuts

Three shortcuts fail: “Retry-After means retry no matter what,” “estimate is below the limit so the call is safe,” and “fallback is healthy so cost does not matter.” Each substitutes one gate for another.

The cost governor also must not invent model qualification. If an owning route/qualification lesson supplies BLOCKED or UNKNOWN, this lesson consumes that state rather than converting it to success.

14. The audit record must reconstruct the decision

Record request/job/consumer/backend IDs, policy version, estimated and actual tokens with provenance, hard/soft/remaining budget, quota/window, retry count/budget, fallback count/amplification, failure signal, Retry-After, circuit before/after, final action, and reason class.

If a reviewer cannot reconstruct why a call was stopped, degraded, or deferred from the record, governance is not evidence-closed.

15. Exit checklist and transfer

Before PASS: is hard budget enforced before a call? Are estimate and actual distinct? Are quota and circuit reasons independent? Are retry/fallback bounded? Do OPEN and HALF_OPEN differ? Is the decision bound to a policy version and usage evidence?

Transfer: the fallback is healthy and qualified, but its worst case is 8k tokens while only 6k hard budget remains. Decide from policy without launching it. Use DEGRADE if a cheaper allowed path exists, DEFER if waiting is allowed, or HARD_STOP if neither is allowed. “Call it and hope actual usage is lower” is not valid authorization.