Mentor Quality and Escalation Operations

Why activity is not mentor quality

Imagine a mentor who looks excellent on the dashboard: high reply volume, fast response time, and strong satisfaction. A governed sample, however, contains unsupported technical advice, two reviewers disagree on a rubric dimension, and two learner complaints still lack traceable resolution evidence. If the system equates activity with quality, it can automatically clear the mentor while the consequential evidence says the opposite.

This lesson builds Mentor Quality Review, Calibration & Escalation Operations Ledger v1. Every reviewed interaction is bound to provenance; sampling policy and rubric are versioned; scores point to dimensions and evidence; disagreement and conflict stay visible; safety/policy/competence issues are typed; escalations have severity, owner, acknowledgment and resolution evidence; coaching never erases history; and automated signals remain advisory until an authorized human closes consequential mentor decisions.

The output is not a universal mentor score. It is an auditable operations contract that can justify REVIEW_REQUIRED, HOLD, INSUFFICIENT_EVIDENCE, or—only under bounded conditions—ACTIVE_BOUNDED.

For: learning-platform engineers, mentor/reviewer operations, QA teams, and reviewers who must reconstruct a quality decision from visible evidence. Not for: covert personnel ranking, automated HR decisions, or use of live private learner data without authorization.

Route and lesson boundary

Foundation learners start with the field/state map and the baseline. Repair learners isolate the first broken evidence dimension and rerun the smallest mutation. Experienced learners start from conflicting evidence and defend the boundary between activity, quality, learner outcome, and risk.

MKN-029 owns quality review, calibration, and escalation operations. MKN-028 supplies only prerequisite context-sharing boundaries; MKN-030 owns portable packaging next. This lesson does not issue legal conclusions, expose live learner context, approve a production mentor decision, or make an employment decision. The executable lab is synthetic and offline.

Three checkpoints drive the route: F1 restates scope, F2 rejects an invalid shortcut with exact evidence, and F3 rebuilds the transfer decision after a changed rubric. The baseline answer is never the oracle.

Mental model: interaction to resolution, with activity in a separate lane

The quality path is:

interaction snapshot → governed sample → rubric-versioned review → evidence-bound dimension scores → reviewer disagreement/calibration → typed issue classification → escalation owner + acknowledgment → resolution/corrective action → follow-up evidence → human closure

Activity metrics—reply counts, sessions, speed, or satisfaction—stay in a separate lane. They can be signals or sampling inputs, but they are not quality verdicts. That distinction is where Caliper is useful: it provides consistent language and metric profiles for learning activity data; it does not define mentor competence or escalation thresholds.

Keep three authority classes separate: FACT (what an external standard/framework says), INFERENCE (what the evidence suggests for a quality operation), and DESIGN (the local rubric, severity matrix, SLA, coaching, and clearance policy). A public standard never becomes local decision authority by implication.

Foundational contract: identities, versions, and typed states

Decision-critical fields include mentor_ref, interaction_id, interaction_snapshot_hash, review_sample_id, sampling_policy_version, rubric_version, dimension_id, reviewer_ref, review_score, evidence_ref, disagreement_state, calibration_round, conflict_state, unsupported_advice_flag, policy_issue_flag, escalation_id, severity, escalation_reason, escalation_owner, acknowledged_at, resolution_state, resolution_ref, corrective_action_ref, followup_due, appeal_or_review_ref, and audit_event_id.

A score without rubric version and evidence is merely a number. A CLOSED label without resolution evidence is merely a label. Coaching without follow-up evidence is not clearance. Reviewer disagreement is not a data error; preserve it and record what a calibration round changed.

Use honest non-success states: UNKNOWN, STALE, DISPUTED, REVIEW_REQUIRED, HOLD, and INSUFFICIENT_EVIDENCE. Never coerce missing or disputed evidence into success.

Designing sampling, review, calibration, escalation, and automation boundaries

Sampling. The sample must be explainable and reproducible. sampling_policy_version records why an interaction entered review. Activity signals may prioritize review, but sample selection must stay auditable.

Rubric versioning. Each dimension has a meaning, evidence expectation, and bounded scale. If the rubric changes, an old score cannot be relabeled current. Re-score only the governed sample under the new version and preserve the old decision as history.

Calibration. Preserve disagreement. A calibration round may refine dimension interpretation, but it must not erase prior reviewer evidence. calibration_delta records what changed.

Escalation. Type the issue: unsupported advice, policy concern, learner risk, or competence gap. Severity, owner, acknowledgment, resolution evidence, corrective action, and follow-up are distinct fields. Satisfaction cannot substitute for resolution_ref.

AI assistance. Models may summarize or prioritize, but an automated quality signal cannot silently penalize or clear a mentor. NIST AI RMF is useful for governing AI-assisted risk workflows; it does not define MAKNOW's mentor rubric or decision authority.

Worked baseline: a high-activity mentor with an open escalation

The baseline fixture uses mentor M-17. Activity and satisfaction are high, but governed sample S-204 is bound to an immutable interaction snapshot. Under rubric RQ-7, dimension evidence_grounding has a weak score. A second reviewer disputes severity, unsupported_advice_flag=true, and escalation ESC-91 is HIGH, owned and acknowledged, but still OPEN.

The bounded result is ESCALATED_REVIEW_REQUIRED. The evidence contract is valid and the escalation is traceable, yet resolution evidence does not exist. High activity changes none of those facts.

Compare baseline, mutation, and transfer in MKN-029-T1, then run lab://MKN-029/MKN-029-LAB1. The starter fixture contains no prefilled final decision; visible rules provide the oracle.

Failure modes that must fail closed

1. Rubric drift: review was performed under RQ-7, but the record labels the old score current under RQ-8 without re-scoring → HOLD.
2. Satisfaction-as-resolution: escalation is CLOSED from a satisfaction score without resolution evidence → HOLD.
3. Lost snapshot: reviewer disagreement remains but the original interaction snapshot/hash is missing → HOLD.
4. Automated penalty: a model signal directly changes the mentor to a punitive state without human authority → HOLD.
5. Missing owner/ack: a severe escalation lacks an owner or acknowledgment → HOLD.
6. Coaching equals clearance: corrective coaching exists but follow-up evidence does not → not cleared.
7. Activity proxy: volume/speed is used as a quality score → invalid evidence use; quality state must not change.

Repair the smallest failed dimension. Never repair a record by inventing authority or evidence.

Experienced bridge: model quality as a replayable decision system

Treat the quality system like event-sourced governance rather than a final dashboard. Interaction snapshot, sample inclusion, rubric version, reviewer evidence, disagreement, escalation transition, resolution, and follow-up each become auditable events. Current state is a projection of that history, not a replacement for it.

At scale, keep three lanes: telemetry for activity, review for quality evidence, and governance for escalation/appeal. Join them through stable identity and provenance instead of collapsing them into one score. Analysts can study activity, reviewers can assess quality, and authorized owners can close consequential cases without mixing accountability.

Sampling creates an operations trade-off: broader coverage costs reviewer time and latency; narrow sampling leaves blind spots. The answer is a versioned policy with review triggers and stop/pivot conditions, not an opaque weighted score.

Alternatives and trade-offs

Review everything: maximum coverage, high cost and latency. Random sampling: lower selection bias, but rare high-risk cases may be missed. Risk-prioritized sampling: better efficiency, but prioritization must remain advisory and auditable. Satisfaction-triggered review: useful signal, not quality evidence; sentiment can punish correct but difficult guidance. Automated scoring: useful for triage and consistency checks; unsuitable for consequential closure without human review.

No pattern is universally best. Make sample coverage, severity exposure, reviewer capacity, dispute rate, follow-up burden, and authority boundary explicit. When decisive evidence is incomplete, choose a weaker state.

Guided and independent practice

Guided: open the baseline. Ignore activity metrics first. Locate snapshot hash, sampling policy, rubric version, dimension, reviewer evidence, and escalation owner/ack. Derive the decision from visible rules. Then run satisfaction_closed.json and identify the exact field that forces HOLD.

Repair: in stale_rubric.json, repair only the rubric mismatch and rerun. Do not change unrelated fields. Determine whether another gate still limits the state.

Independent transfer: the transfer fixture moves to RQ-8 and records completed coaching. Re-score only the governed sample, retain the old decision as history, bind follow-up evidence, and refuse any automatic clearance from activity or model summaries. Recompute from the changed fixture; never copy the baseline label.

What belongs in an acceptable evidence package?

At minimum, preserve mentor/interaction identity, snapshot hash, sample/policy version, rubric/dimension, reviewer/evidence/score, disagreement/calibration, issue flags, escalation/severity/reason, owner/ack, resolution reference, corrective action, follow-up, appeal/re-review, and audit event.

Producer tests independently execute the major mutations. The learner path exposes rules, fixtures, and a submission template; producer evaluation logic stays outside the learner-publishable root so the starter does not pre-solve the skill.

A human reviewer must be able to reproduce the bounded state from visible evidence without hidden model reasoning. HOLD and REVIEW_REQUIRED are honest operational states, not system failures.

Summative assessment and transfer

Four summative prompts live in derived_text/assessment/questions.json: S1 boundary, S2 baseline ledger, S3 stale-rubric diagnosis, and S4 transfer after calibration and coaching.

Five rubric dimensions are blocking: identity/contract, exact evidence binding, negative evidence, bounded decision, and transfer. A response that says “satisfaction is high, so escalation is closed” or “the model score is high, so clear the mentor” fails even if other fields are correct.

In transfer, RQ-8 changes one dimension. Re-score the governed sample only, preserve the old decision, keep coaching separate from follow-up evidence, and if human closure is still pending retain HUMAN_CLEARANCE_REVIEW_REQUIRED. The baseline state is not an oracle.

References, claim boundaries, and freshness

1. 1EdTech Caliper Analytics supports consistent capture/labeling of learning activity and metric profiles. It does not define mentor-quality rubrics or escalation thresholds.
2. NIST AI Risk Management Framework is an official risk-governance reference for AI-assisted workflows. AI RMF 1.0 uses Govern, Map, Measure, and Manage, and NIST is revising it; operational use therefore remains freshness-sensitive.
3. Sampling policy, rubric, severity, SLA, coaching, appeal, and clearance authority in this lesson are MAKNOW DESIGN, not external mandates.

The two official sources were rechecked on 2026-09-13. Practice-evolving claims must be requalified by 2027-01-15 or earlier after a material guidance change. An accepted package is never silently rewritten; updates require a new version.

Closing question: which shortcut in your current system creates the most false confidence—activity volume, satisfaction, automated score, or “coaching completed”—and which exact evidence field exposes it?