AI Tutor Grounding and Evaluation

Fluent is not the same as grounded

A tutor answers a learner question with a clear, confident explanation. The answer earns a high model-grader score and sounds pedagogically useful. But the retrieval trace points to an obsolete lesson revision, one sentence introduces a claim that no permitted source supports, and the prompt accidentally exposes the hidden answer to a graded checkpoint. A spot check can miss all three failures.

In this lesson you build Grounded AI Tutor Evaluation Harness & Evidence Ledger v1. Every evaluation case binds the tutor output to a versioned learner-state fixture, an explicit allowed/forbidden context boundary, a source snapshot, retrieval evidence, claim/citation alignment, deterministic checks, model grading, human-expert review where required, assessment-integrity/refusal behavior, and model/prompt/retrieval versions. Promotion is a per-dimension evidence decision—not a reward for fluent prose or a single aggregate score.

Core rule: trace the answer back to permitted evidence, then evaluate behavior and regression separately.

For: learning-platform engineers, AI tutor evaluators, retrieval/RAG engineers, instructional QA, safety/governance reviewers. Not for: releasing a tutor from one good conversation, treating an LLM judge as ground truth, exposing answer keys for evaluation convenience, or letting aggregate score erase a blocking grounding/integrity failure.

Route orientation and ownership boundary

MKN-023 owns only AI Tutor Grounding and Evaluation. Its frozen prerequisites are MKN-021 (hard) and MKN-022 (soft). MKN-022 owns notification delivery; MKN-024 owns portfolio/profile compilation. This lesson may consume their bounded context but does not absorb their outcomes.

Foundation learners use a field map and one guided failure. Repair learners rerun only the failed dimension and preserve the original trace. Experienced learners defend dataset composition, grader calibration, human-review triggers, and regression policy without collapsing everything into one score.

Checkpoints: F1 state the ownership boundary and one non-goal; F2 diagnose one grounding/integrity failure by exact field/state; F3 rerun the fixed suite plus targeted failure-cluster cases after a model change and decide from dimension evidence.

Mental model: fixture → trace → dimension evidence → promotion gate

Use this ordered pipeline:

versioned learner/source fixture → tutor run + retrieval trace → claim/source alignment → deterministic integrity checks → rubric/model graders → human review where required → per-dimension regression report → PROMOTE / REVIEW_REQUIRED / HOLD

Keep six things separate: permitted learner state, source authority, generated claims, deterministic violations, probabilistic quality judgments, and release governance. A model grader can help measure a rubric. It cannot authorize access to forbidden context, repair a missing citation, or convert a failing integrity check into a pass.

Foundation contract: version every decisive input

Preserve the Bible dimensions: eval_suite_version, test_case_id, learner_state_fixture, allowed_context_ref, forbidden_context_ref, source_snapshot_id, retrieval_query, retrieved_source_ids, citation_alignment, grounded_claim_id, unsupported_claim, contradiction_state, expected_behavior, rubric_version, deterministic_grader, model_grader, human_expert_review, assessment_integrity_state, refusal_state, model_version, prompt_version, retrieval_version, regression_delta, promotion_state, failure_cluster, and trace_ref.

Minimum invariants:

1. A retrieved source must belong to the declared source snapshot; semantic similarity does not excuse a stale revision.
2. Allowed and forbidden context are explicit fixtures; hidden assessment solutions never migrate into allowed context.
3. Unsupported claims remain typed even when the rest of the answer is good.
4. Deterministic grounding/integrity failures are blocking and cannot be averaged away by a model score.
5. Model-grader results record grader/rubric version; high-impact pedagogical judgments require human review when the contract says so.
6. Model, prompt, retrieval, and eval-suite versions are recorded together.
7. A changed model/configuration requires regression evidence; the prior promotion result is not an oracle.
8. Promotion state is no stronger than the weakest blocking dimension.

Deep explanation: provider evals, graders, and MAKNOW authority

OpenAI Evals. The current API describes an evaluation as a set of testing criteria plus a data-source configuration that defines the data schema. The same evaluation structure can be run across different models or model parameters. That is useful for a repeatable harness, but it does not define learner-state access policy, pedagogy, or MAKNOW passing thresholds.

Grader types. Current OpenAI grader APIs expose string checks, text similarity, model-based grading, Python grading, and multi-grader composition. Use deterministic graders for exact properties—IDs, forbidden-context exposure, required citation references, version bindings—and model graders for rubric qualities that genuinely need judgment. Model grading is evidence, not authority.

NIST AI RMF Generative AI Profile. NIST AI 600-1 frames trustworthiness considerations across generative-AI design, development, use, and evaluation. It is voluntary risk-management guidance. It does not provide a tutor acceptance score, a MAKNOW instructional rubric, or a release decision.

Therefore the dataset, allowed context, source authority, rubric, blocking dimensions, human-review triggers, thresholds, and release gate are MAKNOW DESIGN. External documentation informs mechanisms and risk framing; it does not replace local governance.

Worked examples: one good trace and two deceptive wins

Baseline. Case ALG-07 uses learner state LS-07, source snapshot COURSE-ALG-v5, and retrieved sources ALG-v5-S2 and ALG-v5-S3. Every factual claim maps to those sources, forbidden assessment context is absent, deterministic checks pass, the rubric grader is acceptable, and a human reviewer still owns the pedagogical-release decision. Result: REVIEW_REQUIRED, not automatic promotion.

Deceptive win 1 — obsolete source. Retrieval swaps ALG-v5-S2 for semantically similar ALG-v4-S2. The answer may remain factually plausible and receive a high rubric score, but source_snapshot_id and retrieved_source_ids conflict. Result: HOLD_SOURCE_REVISION_MISMATCH.

Deceptive win 2 — aggregate score hides a citation failure. Average model score rises from 0.86 to 0.93, while deterministic citation_alignment fails for a central claim. The aggregate score is not allowed to overrule the blocking dimension. Result: HOLD_GROUNDING_FAILURE.

Failure modes that must remain visible

• Stale-but-similar retrieval: obsolete course revision is treated as current.
• Forbidden-context leakage: hidden solution or instructor-only state appears in allowed context.
• Unsupported fluent claim: the answer sounds right but no retrieved permitted source supports the claim.
• Contradiction laundering: a source contradiction is present but output is marked grounded.
• Judge monoculture: one model grader is treated as both evaluator and release authority.
• Version drift: model changes while eval suite, prompt/retrieval versions, or regression evidence stay unrecorded.
• Refusal failure: the expected behavior is to refuse/redirect but the tutor provides the protected answer.
• Average-score masking: a blocking dimension gets hidden inside a strong aggregate.
• Failure-cluster blindness: targeted low-resource or edge cases are omitted after a global-score improvement.

All of these are evidence failures, not invitations to fabricate a success state.

Alternatives and trade-offs

A tiny deterministic suite is reproducible but can miss pedagogical nuance. A model judge is flexible but can drift, share blind spots with the system under test, or reward style. Human review is valuable for pedagogy and high-impact interpretation but is costly and can vary without calibration. A large benchmark improves coverage but becomes stale if cases no longer represent learner traffic or new failure clusters.

Use a layered evaluation stack: deterministic checks for invariants, representative fixed cases for regression, targeted cluster cases for known failures, calibrated model graders for rubric dimensions, and human review for pedagogical/high-impact decisions. Keep each layer's authority explicit.

Experienced bridge: evaluation is a versioned product surface

Treat the eval harness as production infrastructure. Dataset revisions need change control; grader prompts and models need version identity; retrieval fixtures need source-snapshot identity; failure clusters need owners and retirement criteria; and promotion receipts need reproducible trace references.

Useful reporting is not “score = 91%.” It is a matrix: grounding, contradiction, integrity, refusal, pedagogy, and regression by learner/source segment, with blocking failures listed before aggregates. A high mean with concentrated unsupported claims in low-resource lessons is a regression, not a success story.

Guided and independent practice

Guided. Run MKN-023-LAB1 on the baseline. Fill the evidence ledger from visible synthetic fields. Do not prefill promotion_state; derive it from checks. Then mutate only the retrieved source revision and identify the first invalidated field/state.

Independent. Diagnose: hidden assessment solution in allowed context; unsupported claim despite high model score; model-version change without regression evidence; and a refusal-required case answered directly. For each, name the failed dimension, evidence, bounded terminal state, and minimal repair.

Transfer. A new model improves average rubric score but increases unsupported claims on low-resource lessons. Re-run the fixed suite plus targeted low-resource cluster cases. The aggregate improvement is secondary; promotion depends on the per-dimension regression evidence.

Verification and evidence

The offline Lab is synthetic and deterministic: JSON fixtures, Python checks, no network/API key, and no production learner data. It verifies source-snapshot membership, forbidden-context separation, unsupported claims, contradiction state, assessment-integrity/refusal behavior, grader layering, version bindings, regression evidence, and transfer recomputation.

A valid evidence ledger answers: Which learner state was permitted? Which sources were retrieved from which snapshot? Which claims were grounded? Which were unsupported? Which deterministic checks passed? Which rubric/model grader ran under which version? Where was human review required? What changed from the previous model/prompt/retrieval combination? Why is the promotion state bounded?

Assessment and transfer gate

Four blocking assessments:

1. Explain MKN-023 ownership and one non-goal.
2. Build the complete harness/evidence ledger for the baseline fixture.
3. Diagnose the obsolete-source mutation and propose the minimal repair.
4. Transfer: evaluate the new model whose average score improves while unsupported claims rise on low-resource lessons.

The transfer answer fails if it copies the baseline verdict, considers only the aggregate score, omits the targeted failure cluster, or lets an LLM judge override deterministic grounding/integrity evidence.

References, freshness, and authority boundaries

OpenAI — Create eval https://developers.openai.com/api/reference/java/resources/evals/methods/create FACT: evaluation objects combine a data-source configuration/schema with testing criteria/graders and can be run across model/config changes. Boundary: provider mechanism, not MAKNOW pedagogy or thresholds.

OpenAI — Grader Models https://developers.openai.com/api/reference/ruby/resources/graders/subresources/grader_models FACT: current grader model types include string checks, text similarity, model graders, Python graders, and multi-grader composition. Boundary: availability does not establish evaluator validity.

NIST — AI RMF Generative AI Profile (NIST AI 600-1) https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence FACT: voluntary risk-management guidance for trustworthiness across generative-AI lifecycle and evaluation. NIST's AI RMF page also states AI RMF 1.0 is being revised in 2026. Boundary: no MAKNOW tutor score or release policy.

Sources were rechecked 2026-09-13. OpenAI tool semantics are rechecked by 2026-10-18 or earlier on material API changes; NIST guidance by 2027-01-01 or earlier on material revision. Dataset composition, thresholds, integrity policy, human-review triggers, and release decisions remain MAKNOW DESIGN.