Structured Outputs and Schema Validation

1. The real problem: valid JSON is still not a valid result

When a model returns text that parses as JSON, only one boundary has been crossed. A dependable workflow separates at least four layers: model terminal state, parseability, schema conformance, and domain invariants. A failure at any layer must not be coerced into a plausible object with defaults. This lesson keeps refusal, incomplete, parse error, schema error, and semantic/domain error as distinct, testable states.

2. Define the application contract before the provider

Write the application contract independently of the provider: required fields, field types, enums, extra-property behavior, nullability, schema version, and business invariants. In the lab, the classifier must produce {category, priority, requires_human, evidence_ids}. That is the application contract. Whether a provider emits it through json_schema, a tool schema, or another mechanism is a separate capability. Provider changes therefore do not silently redefine business logic.

3. Check terminal state before parsing

Before calling a JSON parser, determine whether the response is data completion or another terminal state. Keep REFUSAL, INCOMPLETE, and COMPLETE separate. A refusal is not classifier data, and an incomplete response is not a partial object that the application may guess into completion. The harness classifies terminal state first and only attempts parsing for COMPLETE responses.

4. Parseability proves syntax only

If the text parses, you know the JSON syntax is readable. You do not yet know that required fields exist, enums are allowed, types are correct, or extra properties are forbidden. Parse status therefore gets its own explicit result such as PARSE_OK or PARSE_ERROR. The failure record also retains raw payload identity and the parse error class so downstream code does not need to interpret arbitrary exceptions.

5. Schema validation must be deterministic

After parsing, validate the object against a specific schema version. Required properties, types, enums, array-item types, and extra-property policy must be enforced deterministically. Lab schema v1 requires four core fields and rejects unknown properties. The goal is not to reimplement a provider feature; it is to create independent, repeatable evidence for the application's own contract.

6. A provider strict feature does not replace application validation

Structured-output or strict-tool features can reduce structural failures, but they do not remove application responsibility. A provider may support only a subset of JSON Schema, capabilities can vary across models, and non-data terminal states can still occur. Keep provider-facing and application-facing schemas explicitly mapped. If the required feature is unsupported, preserve UNSUPPORTED_CAPABILITY; do not silently fall back to JSON mode or free text unless policy explicitly allows it.

7. Domain invariants come after schema validation

A schema can assert that evidence_ids is an array of strings, but it cannot by itself prove that every ID belongs to the allowed evidence set. The application may also require requires_human=true whenever priority=high. Those are application invariants and run after schema validation. A schema-valid object that violates an invariant is DOMAIN_INVALID, not partial success.

8. Typed failures are better than scattered exceptions

Each failure needs a stable class, retryability, and promotion decision. Examples include PROVIDER_REFUSAL, RESPONSE_INCOMPLETE, PARSE_ERROR, SCHEMA_INVALID, and DOMAIN_INVALID. Downstream code should branch on these types instead of parsing exception text. Retry behavior must follow the failure class: some failures may justify a bounded regeneration, while others require schema, prompt, model, data, or human/application correction.

9. Treat additional properties and enum drift as first-class failures

Unexpected properties and enum drift are common integration failures. If downstream code branches over a closed enum, a new value can break behavior unless versioning catches it. Extra properties can also signal prompt or provider-shape drift. A strict contract using additionalProperties: false or an equivalent deterministic rule makes drift fail early and visibly instead of leaking into downstream logic.

10. Nullable is not the same as optional

An optional field may be absent. A nullable field may be present with null. In this lesson's transfer, schema v2 adds an optional nullable due_at field. The harness preserves the difference between absence, null, and a valid string. Automatically converting absence to null, or null to absence, is a semantic mutation unless the contract explicitly permits it.

11. Schema version is part of provenance

Every validation record must retain the schema version. Without it, the claim “this output validated” is incomplete. When v2 narrows the priority enum, a payload using priority=low may be valid under v1 and invalid under v2. That incompatibility must be recorded explicitly. Evidence for v1 does not automatically transfer to v2; the payload must be evaluated against the new version.

12. The promotion gate has four ordered stages

This lesson's promotion gate runs in order: 1) terminal state must be complete data; 2) payload must parse; 3) object must conform to the declared schema; 4) domain invariants must pass. Each stage emits its own failure and later stages do not run when an earlier gate fails. This prevents domain checks on malformed payloads and preserves an auditable evidence chain.

13. Separate retry, remediation, and human correction

Not every failure is retryable. INCOMPLETE may justify a bounded retry; a refusal should not trigger blind repetition. SCHEMA_INVALID can trigger prompt/model/schema-mapping remediation. DOMAIN_INVALID often points to source data or a human/application decision. Record retryability and remediation class with every failure so a workflow cannot enter an unbounded retry loop or hide cost behind repeated attempts.

14. Lab: eight deterministic fixtures

The lab contains eight fixtures: contract-valid baseline, malformed JSON, missing required field, a multi-violation structural case with type/enum/extra-property errors, provider refusal, incomplete response, schema-valid output with an unsupported evidence ID, and a schema-v2 transfer case with a narrowed enum. Each case produces eight fixed outputs: terminal state, parse result, schema conformance, domain result, promotion decision, failure class, retryability, and compatibility result. The reference run must pass exactly 64 assertions.

15. Exit checklist and transfer

Before exit, verify that the contract is versioned; terminal states are separate from data; parsing, schema validation, and domain validation are distinct; failures are typed; unsupported capability is not coerced into success; retries are bounded; schema version is retained in evidence; and promotion occurs only after all four gates pass. Then run the transfer: schema v2 adds nullable due_at and narrows priority. Re-run old and new fixtures and classify which failures justify retry, prompt/model/schema change, or human/application correction.