Diagnostic/Adaptive Assessment Design

When the same score should lead to a different next action

Two learners can show the same scalar estimate and still need different next steps. One may have broad evidence across prerequisite and transfer items; another may reach the same number through only a few high-quality transfer items. If an adaptive system hides item alignment, evidence composition, uncertainty, stop conditions and routing behind one score, it becomes impossible to explain why the next item was selected—or why the score must not silently become mastery.

This lesson builds Diagnostic & Adaptive Assessment Decision Matrix v1. It keeps diagnostic estimate, uncertainty, item selection, accommodation, route/stop logic and mastery effect as separate auditable decisions.

Lesson boundary and learning routes

MKN-005 owns diagnostic/adaptive assessment design. MKN-001 is a hard conceptual prerequisite and MKN-004 is a soft bridge for longitudinal mastery concepts. The adjacent result remains separate: this lesson does not redesign the longitudinal mastery model and does not issue credentials.

Foundation learners use the seventeen-field map. Repair learners isolate the first invalid field or state. Experienced learners must defend why two identical estimates can produce different routes when uncertainty or evidence composition changes. No live learner record or production assessment service is modified.

Mental model: target → item evidence → estimate + uncertainty → selection/stop → route → separate mastery review

The core flow is:

competency target → aligned item → response evidence + weight → estimate before/after + uncertainty → selection rule → next item | repair route | stop → accommodation-aware delivery → separate mastery effect/review

The diagnostic estimate summarizes evidence under a declared rule. Uncertainty says how much confidence the system should place in that estimate. The selection rule explains why another item or route is chosen. The stop condition explains why the assessment may end. Mastery effect records what the diagnostic result is allowed to do to mastery state—and in this lesson it is never silent auto-promotion.

Foundation contract: seventeen auditable dimensions

The Matrix records assessment_id, competency_id, item_id, item_alignment, item_difficulty_band, response_state, evidence_weight, estimate_before, estimate_after, uncertainty_band, selection_rule_id, next_item_or_route, stop_condition, accommodation_ref, mastery_effect, review_state, and provenance.

Identity and alignment tell us what was measured. Response and evidence weight describe what evidence the item added. Estimates and uncertainty expose the model state. Selection, route and stop make the adaptive policy inspectable. Accommodation must change access/presentation or eligible delivery without secretly changing the semantic value of identical evidence. Mastery effect and review state keep diagnostic routing separate from certification.

Deep explanation: FACT, INFERENCE, and DESIGN stay separate

FACT — QTI 3. Current 1EdTech QTI material describes an interoperability specification for assessment tests, sections, items and results, and QTI 3 provides more native support for Computer Adaptive Testing. This helps exchange assessment structures and CAT-oriented mechanisms; it does not define MAKNOW mastery thresholds or routing policy.

FACT — assessment alignment. Carnegie Mellon guidance emphasizes that assessments should align with the objectives they are intended to measure. An item that measures an adjacent competency cannot be treated as target-competency evidence merely because its score is convenient.

INFERENCE — one score is not an audit trail. If alignment, evidence weight, uncertainty, selection rule and route are explicit, a reviewer can reconstruct the decision. If they are collapsed into one scalar, identical numbers can hide materially different evidence states.

DESIGN — MAKNOW. Diagnostic estimates remain distinct from mastery state. Uncertainty and stop conditions are explicit. Accommodations cannot silently reweight evidence. The adaptive engine may route or request review; it cannot silently certify competence.

Worked examples: baseline repair, high uncertainty, accommodation, and transfer

Baseline. The learner answers three easy prerequisite items correctly, then misses a medium transfer item. The estimate falls from 0.72 to 0.64, uncertainty remains MEDIUM, and the policy selects route:repair-transfer before another transfer check. The learner has requested a keyboard/text accommodation, so the next delivery must honor it. The result is DIAGNOSTIC_REPAIR_ROUTE_REQUIRED; mastery remains a separate review.

Same estimate, higher uncertainty. Keep an estimate at 0.78 but increase uncertainty from LOW to HIGH. Copying the old “ready for review” route is invalid. The system must request more evidence: DIAGNOSTIC_CONTINUE_ASSESSMENT.

Accommodation without evidence mutation. Add an accommodation while holding item semantics, response and evidence quality constant. Delivery may change, but evidence weight and estimate must not be rewritten merely because access support changed.

Transfer. A second learner reaches the same estimate 0.78 through fewer, high-quality transfer items. Recompute uncertainty and next action. In the fixture, evidence breadth is narrower, so uncertainty is MEDIUM and one more transfer item is required even though the scalar estimate matches the baseline.

Failure modes that must fail closed

Fail or hold when assessment/competency identity is missing, item alignment is invalid, difficulty or response state is unknown, evidence weight is outside its allowed range, estimate values are impossible or not traceable to visible evidence, uncertainty is unknown, selection or next action is missing, stop state is incoherent, provenance is absent, or review state is already “approved” without a separate human decision.

Two failures are especially dangerous. Auto-promotion: a high diagnostic estimate must not set mastery to MASTERED. Accommodation contamination: adding an accessibility accommodation must not alter evidence weight or estimate unless the underlying response/evidence actually changes. Transfer also fails when a baseline decision is reused after evidence composition changes.

Alternatives and trade-offs: precision, burden, breadth, and fairness

More items can reduce uncertainty but increase learner burden and fatigue. Harder transfer items may be more informative for some competency claims but can also create accessibility or prerequisite confounds. Aggressive stopping reduces testing time but risks ending before decisive evidence appears. Conservative stopping preserves evidence quality but costs more time.

A single weighted score is operationally simple but hides evidence composition. A richer Matrix costs storage and review effort but makes routing reproducible. Accommodations may change presentation, timing or modality; they should preserve the intended construct unless the assessment explicitly documents a construct change. The point is not to maximize test length—the point is to make the evidence-to-action transition explainable.

Experienced bridge: separate estimator state from policy state

For experienced engineers, model at least two layers. The estimator layer owns response evidence, weights, estimate and uncertainty. The policy layer owns item eligibility, selection, route, stop and review. A third governance boundary owns mastery effect. This separation lets you replace a CAT algorithm without silently changing mastery policy.

Do not treat estimate_after as the oracle. A valid adaptive system can produce the same estimate from different evidence paths, yet choose different next actions because uncertainty, coverage, alignment or stop budget differs. Store the path evidence needed to reproduce that judgment.

Guided and independent practice

1. Run the baseline and explain why the medium transfer miss routes to repair instead of mastery. 2. Keep the score constant and raise uncertainty; verify that the next action changes. 3. Change item alignment to an adjacent competency and require recomputation. 4. Add an accommodation without changing response evidence and verify that evidence weight/estimate stay stable. 5. Hit the stop budget before decisive transfer evidence and explain why the result is review-required rather than mastered. 6. Rebuild the second learner case without using the baseline decision as an oracle.

In independent practice, expected decisions are removed. The learner sees only the seventeen fields, fixture facts and deterministic gate rules.

Verification evidence: what a reviewer must reconstruct

A reviewer must reconstruct the assessment and competency identity, the selected item and its alignment/difficulty, response state and evidence weight, estimate before/after, uncertainty, selection rule, next action, stop state, accommodation, mastery effect, review state and provenance. They must also show that changed alignment or evidence composition triggered recomputation and that accommodation did not silently rewrite evidence semantics.

The deterministic Lab covers baseline repair, evidence-review-ready, same-score/high-uncertainty, alignment mismatch, missing fields, invalid weight/estimate, missing selection/route/stop/provenance, forbidden mastery auto-promotion, accommodation contamination, stop-budget exhaustion and changed-context transfer with and without recomputation.

Assessment and transfer: same estimate, different evidence path

The assessment has four blocking parts: explain the MKN-005 boundary and one adjacent non-goal; build the baseline Matrix; diagnose “same score, higher uncertainty”; and rebuild the changed-context learner with fewer, higher-quality transfer items.

Transfer passes only when the new evidence composition is explicitly recorded, uncertainty is recomputed, the next action is derived from the new state, and baseline_oracle_reuse=false. If the estimate remains identical but uncertainty changes, copying the prior route is a failure.

References, freshness, and authority boundary

Official 1EdTech QTI and Carnegie Mellon assessment-alignment material were rechecked on 2026-09-14. QTI 3 remains the current QTI 3 final-release family and includes more native CAT support; CMU continues to frame assessment quality around alignment with intended learning objectives.

These sources provide FACT about mechanisms and design guidance. They do not define MAKNOW evidence weights, uncertainty bands, stop budgets, mastery effects or review authority. Evolving QTI/CAT semantics should be rechecked by 2027-01-15 or earlier after material change.