LLM Gateway and Provider-neutral Abstraction

1. The real problem: hiding an SDK is not semantic abstraction

When business logic sees provider request types, model identifiers, message blocks, exceptions, and usage objects, provider semantics have leaked into the product core. A thin generate(prompt) method can hide SDK names while still flattening important differences in refusal, tool calls, structured output, streaming, usage, and failure behavior. This lesson builds a boundary that stabilizes product-owned semantics while keeping genuine provider differences observable.

2. Define the product operation first

A gateway should begin with a bounded operation, not every capability an LLM might expose. In the lab, the operation is summarize_support_case: it accepts support text and returns a typed summary with declared fields. The domain service knows only that operation and its contract. Whether Provider Alpha receives a response-format schema and Provider Beta receives a tool-like schema belongs to adapters, not domain logic.

3. The canonical request contains only product-owned semantics

A canonical request should retain fields the product actually depends on: operation, input, required capabilities, timeout, trace id, and contract version. Provider-specific knobs enter the shared contract only when the product deliberately owns their meaning. A vendor-only reasoning or verbosity parameter should not leak into the public domain interface merely because one adapter exposes it; it can remain private adapter configuration.

4. The canonical response must expose terminal state

A shared response is more than text. It should carry terminal state, data payload, normalized error, raw provider evidence, usage state, provider/model provenance, and trace id. A refusal must not become empty text, and an error must not resemble a successful response with missing data. Domain code should distinguish COMPLETE, REFUSAL, and ERROR without importing a provider SDK.

5. Declare capabilities before translation

Each adapter has a versioned capability declaration: text, tools, structured output, streaming, and any other operation-relevant feature. The gateway checks required capabilities before making a provider call. Missing capability remains UNSUPPORTED_CAPABILITY. Silent fallback from structured output to free text or to a tool call is allowed only when an explicit degraded contract exists and has its own test evidence.

6. Adapters map; they do not invent semantics

An adapter translates the canonical request into provider shape and maps the provider response back. It must not fabricate a state that neither provider evidence nor policy supports. If Beta returns a refusal block, preserve refusal. If usage arrives asynchronously, do not record zero as final usage. If a field has no equivalent, UNAVAILABLE or UNKNOWN is stronger evidence than an invented value.

7. Lowest common denominator is not the goal

Provider-neutral design fails when every capability is reduced to the smallest common set. That can remove useful provider features or manufacture fake equivalence for another provider. Prefer operation-scoped contracts: state what this operation requires, and let adapters retain extra provider capabilities outside the domain interface.

8. Unsupported, degraded, and rejected are different states

When a provider lacks a required capability, the workflow may reject the operation, run under a declared degraded contract, or defer until new evidence or an adapter exists. The choice must be policy-driven and auditable. An automatic fallback that silently changes semantics while returning the same response type is dangerous because callers believe the original guarantee still holds.

9. Normalize errors without deleting raw provenance

The gateway can map provider failures into stable classes such as RATE_LIMIT, TIMEOUT, AUTH, PROVIDER_ERROR, REFUSAL, and UNSUPPORTED_CAPABILITY, so business code does not parse exception strings. But raw provider code/message/request id should remain available for debugging and governance. Normalization is a decision layer, not deletion of primary evidence.

10. Usage normalization must tolerate UNKNOWN

Providers report token and cost metadata with different fields and timing. Canonical usage should have a state such as FINAL, PENDING, or UNAVAILABLE. If usage is delivered later, the gateway must not fabricate a final number at response time. Later cost-control lessons can consume this evidence; AIE-001 only owns the normalization boundary.

11. Authentication and timeout are part of the boundary

The domain service should not see provider credentials. The adapter or gateway owns the credential boundary and returns only typed authentication outcomes. Timeout is also part of the canonical request policy and is mapped to provider-specific mechanisms. Dynamic retry strategy is outside this lesson; the requirement here is that timeout and provider failure remain distinguishable and traceable.

12. Provenance is incomplete without provider identity

Each response evidence record should retain adapter version, provider/model id, provider request id when available, trace id, contract version, and capability profile. Without them, “the gateway returned a result” is not enough for debugging or requalification. If an alias or backend changes, provenance must still reveal which mapping actually executed the request.

13. Contract tests run one domain workflow across two providers

The lab's domain service calls one fixed function. Alpha and Beta have different wire shapes, but both must produce a valid canonical response. Tests cover success as well as refusal, rate limiting, timeout, and raw provider errors. PASS means the domain workflow contains no provider-specific branching and provider differences remain visible instead of being flattened.

14. Lab: eight deterministic fixtures and 64 assertions

The lab includes Alpha success, Beta success, Beta refusal, Alpha rate limit, Alpha timeout, Beta unknown provider error, a third provider without structured capability, and a third provider with a declared tool-based degraded mapping and non-final usage. Each case produces eight fixed outputs: capability result, request mapping, response state, normalized error, usage state, provenance, contract-test result, and decision. The reference run checks exactly 64 assertions.

15. Exit checklist and transfer

Before exit, confirm that operation and contract version are explicit; business logic imports no provider-specific types; capability gaps remain explicit; refusal/error/unsupported states are not flattened; raw provider evidence is preserved; unknown usage is not invented; auth and timeout boundaries are visible; and contract tests pass across two provider shapes. Then add a third provider that supports text and tools but lacks the same structured-output mode and reports usage asynchronously. Without changing the domain interface, decide whether the operation is PASS, CONDITIONAL, DEFER, or REJECT and record the residual portability gap.