Model Qualification: Task/Locale/Risk Benchmark

1. The real problem: one aggregate score can hide a critical failure

A model that looks excellent on a public leaderboard or a few English demos is not automatically qualified for your workflow. Real requirements vary by task, locale, risk tier, schema, refusal behavior, and failure cost. If a global average hides a critical Persian high-risk failure under strong scores elsewhere, the benchmark is producing numerical comfort rather than a defensible decision.

2. Qualification is not a global property of a model name

QUALIFIED must bind to an exact candidate and exact scope: model version, configuration, prompt version, dataset version, and task × locale × risk slice. The same model can be qualified for low-risk English summarization and not qualified for high-risk Persian extraction. A familiar model name does not authorize implicit inheritance across those slices.

3. Declare the evaluation contract before seeing results

Before running candidates, declare slices, metrics, hard gates, soft metrics, failure budgets, and how UNKNOWN is handled. A threshold changed after scores arrive is not an auditable qualification rule. The contract must say which failure closes qualification by itself and which metric is merely comparative or diagnostic.

4. The dataset must represent the product, not benchmark convenience

The evaluation set should cover representative product tasks, locales, and risk tiers. Record coverage explicitly: which slices were tested, how much evidence each slice has, and which slices remain unobserved. Do not fill a missing Persian high-risk slice with English low-risk evidence; retain that state as UNKNOWN or out of scope.

5. Keep row-level evidence beside aggregate results

Aggregates help reveal trends, but root causes and critical failures live at row level. The matrix should retain candidate/version/slice, hard-gate result, metric scores, human rating, judge state, and uncertainty. When an aggregate is green and a declared hard gate is red, the hard gate wins.

6. Separate hard gates from soft metrics on purpose

Schema validity, forbidden unsafe output, a required refusal, or a critical extraction field can be hard gates; fluency or style may be soft metrics. A strong soft average cannot compensate for a hard failure. Conversely, a low soft metric should not silently become NOT_QUALIFIED unless the predeclared contract made it decisive.

7. Do not treat a judge model as ground truth

A model-based judge can scale evaluation, but it remains a measurement instrument. Calibrate it against human ratings or deterministic references on sensitive slices and preserve disagreement. If judge and calibrated human ratings diverge on a locale-sensitive subset, keep the state CONDITIONAL/OPEN rather than choosing whichever score is more convenient.

8. Use human review only where it adds distinct evidence

Not every dimension needs a human rating. JSON schema validity, exact field matches, and forbidden tokens can be checked deterministically. Nuanced usefulness, cultural fit, or ambiguous refusal may need human review. The goal is reproducible evidence plus accountable human judgment where it is genuinely required.

9. Three bounded states: QUALIFIED, NOT_QUALIFIED, CONDITIONAL

QUALIFIED means all in-scope required slices and declared hard gates have sufficient passing evidence. NOT_QUALIFIED means at least one decisive hard gate fails. CONDITIONAL means there is no decisive failure but an evidence gap, judge disagreement, or scope limitation remains. None of these states means the model is globally good, bad, or safe.

10. Worked example: the best average is not necessarily the winner

Suppose Candidate A has a 0.94 mean but violates a schema/safety hard gate on Persian high-risk extraction. Candidate B has a 0.86 mean and passes every hard gate in the declared scope. B may be QUALIFIED for that scope while A is NOT_QUALIFIED; the aggregate cannot override the critical row.

11. Do not close UNKNOWN by silent inference

If a new candidate has English evidence but Persian/high-risk has not been tested, that slice remains UNKNOWN. Similar naming, nearby versions, or strong performance elsewhere is not sufficient inheritance evidence. Preserve the gap and state the requalification trigger.

12. Version binding is part of the evidence

Each row should bind at least model version, configuration version, prompt version, and dataset version. A change in one decisive component may invalidate only certain slices. Requalification does not always require rerunning the whole benchmark, but reuse is allowed only when the evidence binding is genuinely unchanged.

13. The qualification record constrains downstream behavior

The output of this lesson is not a public ranking; it is a versioned qualification matrix that downstream routing can consume. The record must expose scope, status, hard failures, evidence gaps, conditional limits, and requalification triggers so AIE-005 cannot reinterpret a bounded status as global permission.

14. Lab: eight deterministic fixtures

The lab includes an all-pass baseline, aggregate trap, judge-human disagreement, missing high-risk slice, schema hard-gate failure, lower soft mean with intact hard gates, prompt-version mismatch, and a transfer case where model v2 changes refusal behavior. The evaluator returns coverage, hard failures, UNKNOWN state, judge disagreement, aggregate quality, and a bounded decision; hidden expected labels are never placed in learner files.

15. Exit checklist and transfer

Before leaving, verify explicit scope and versions, representative slice coverage, predeclared hard gates, retained row-level evidence, no aggregate override of hard failures, judge calibration, preserved UNKNOWN state, and a bounded auditable status. For transfer, a new version that improves English quality but changes refusal behavior and lacks Persian high-risk evidence reruns only the necessary slices and never inherits prior qualification blindly.