Recommendation Engine with Evidence & Confidence

A useful recommendation is not merely “choose A.” It must show which evidence supports A, what contradicts it, how fresh and reliable that evidence is, what is still unknown, what the confidence label is based on, and when the system must HOLD, request review, or abstain.

A common failure is to let the model emit a rank plus a number such as “92% confidence,” then let downstream automation treat the number as calibrated certainty. This lesson rejects that shortcut. The owned artifact is Evidence-Bound Recommendation & Confidence Record v1: a record that binds recommendations to traceable inputs and prevents confident presentation from substituting for evidence.

Core rule: a confidence label is an output of the evidence contract, not evidence for itself. If decisive evidence is missing, stale, weak, or contradictory, preserve a typed non-success state.

Lesson boundary and route

AGN-002 owns recommendation evidence and confidence basis. AGN-001 owns owner-exception management; AGN-003 owns the agent catalog and typed contracts. This lesson does not redesign either adjacent outcome.

AGN-001 is the hard conceptual prerequisite: pressure to produce an answer must not erase accountability when evidence is insufficient. We carry that boundary forward without absorbing the prerequisite lesson.

All examples are synthetic and offline. No real vendor is selected and no production or billing system is mutated.

Mental model: a recommendation is a record, not a sentence

For every candidate, use this sequence:

candidate → supporting + contradicting evidence → freshness + quality → missing evidence + assumptions → confidence basis → review/abstain gate → recommendation state → trace

Keep three things separate:

1. Rank: candidate ordering under the current evidence.
2. Confidence label: bounded strength of the recommendation under this evidence contract.
3. Decision state: recommend, recommend-with-review, HOLD/ABSTAIN, or unresolved.

A high rank does not create high confidence. High confidence does not grant action authority. A trace does not prove truth; it records workflow events.

HOLD, UNKNOWN, PARTIAL, DISPUTED, STALE, and BLOCKED are first-class outcomes when evidence is incomplete.

Foundations: separate FACT, INFERENCE, and DESIGN

FACT — NIST: NIST AI 600-1 is a cross-sectoral Generative AI risk-management profile that supports context-specific risk management, evaluation, documentation, and monitoring. It supports the principle that model output should not receive unqualified trust. It does not define MAKNOW confidence scales or ranking algorithms.

FACT — OpenAI Agents SDK: current tracing documentation says traces record workflow events including generations, tool calls, handoffs, guardrails, and custom events. Those events can provide a trace_ref. They do not prove that a quoted price, capacity signal, or recommendation is factually correct.

INFERENCE: confidence is more defensible when its basis explicitly reflects evidence quality, freshness, contradiction, and missing information than when it is a free-standing model self-assessment.

DESIGN: labels such as STRONG/MODERATE/WEAK/UNSUPPORTED, review/HOLD rules, and promotion thresholds in this lesson are local MAKNOW design choices.

The 23-field recommendation record

The working record contains twenty-three dimensions:

recommendation_id, decision_context, candidate_id, candidate_rank, evidence_ref, evidence_type, evidence_freshness, evidence_quality, supporting_evidence, contradicting_evidence, missing_evidence, assumption_id, confidence_label, confidence_basis, calibration_scope, uncertainty_reason, abstain_condition, recommended_action, alternative_action, review_required, trace_ref, decision_state, review_due.

Gate 1 — Candidate and context

A rank only makes sense inside a decision context. “Best vendor” for urgent delivery can differ from “lowest-cost vendor.”

Gate 2 — Evidence binding

evidence_ref must point to identifiable records. Keep support and contradiction separate so adverse evidence cannot disappear inside prose.

Gate 3 — Freshness and quality

A reliable source can still be stale. Track source quality and time relevance separately.

Gate 4 — Missing evidence and assumptions

Missing evidence is not a neutral value. Record it explicitly. Give assumptions identifiers so a reviewer can distinguish observation from inference.

Gate 5 — Confidence basis

A confidence label is valid only when confidence_basis names the evidence and limitations that justify it. “The model said 92%” is not enough.

Gate 6 — Calibration scope

Do not imply statistical calibration unless it exists. NOT_STATISTICALLY_CALIBRATED is a valid and useful boundary.

Gate 7 — Contradiction and uncertainty

Do not average away a decisive contradiction. A current capacity signal against the top candidate should change uncertainty and often require review or hold.

Gate 8 — Abstention, alternative, and review

A mature system defines when not to recommend. Each record should state an abstain condition, an alternative path, and whether human review is required.

Gate 9 — Trace and review due

trace_ref supports observability; it is not a truth certificate. review_due prevents a recommendation from becoming permanent after price, capacity, or source conditions change.

Worked examples

1. Baseline Vendor A

Two recent delivery records support A, but one current capacity signal contradicts immediate selection. A safe result is not “A at 92% confidence.” A may remain rank 1, but the confidence label is MODERATE and the decision is RECOMMEND_WITH_REVIEW because the contradiction remains visible.

2. Vendor B with stale price

B appears cheaper, but its price record is stale. If price is decisive, the engine must not use the stale advantage as a current fact. Hold, weaken confidence, or refresh the evidence.

3. Remove the strongest support but keep the label

If the strongest A delivery record disappears while the same confidence label remains, confidence has become detached from evidence. The evaluator returns HOLD_UNSUPPORTED_CONFIDENCE.

4. Model percentage as the only basis

model_confidence_pct=92 plus confidence_basis="model says 92%" is self-assessment, not an evidence contract.

5. True transfer

A's capacity signal becomes unavailable and B receives a new verified current price. Rebuild the ranking and confidence basis. Reusing A because it was the baseline winner must fail.

Failure modes

• Naked confidence: a model percentage with no evidence basis.
• Rank-as-oracle: the baseline winner survives every mutation.
• Contradiction erasure: adverse evidence is omitted for a cleaner story.
• Stale-is-current: reliable but old evidence is treated as current.
• Missing-is-neutral: absence of evidence is treated as absence of risk.
• Trace-as-truth: a recorded workflow event is mistaken for factual verification.
• Quality collapse: weak and strong evidence receive identical meaning.
• Hidden assumptions: assumptions are buried in prose instead of recorded.
• Forced success: HOLD/UNKNOWN/PARTIAL/DISPUTED are coerced into a recommendation.
• No review trigger: the engine never defines when changed evidence requires recomputation.

Alternatives and trade-offs

You can use categorical confidence only, or combine it with a genuinely calibrated statistical score. Categories are easier to audit but have less resolution. Numeric scores are expressive, but without real calibration they create false precision.

You can HOLD on every decisive contradiction, or allow RECOMMEND_WITH_REVIEW for bounded cases. HOLD is more conservative; review can be more operationally flexible but needs an accountable reviewer and visible consequence.

This lesson uses one controlling rule: the decision must not be stronger than the weakest decisive evidence.

Experienced bridge

Review a recommendation with seven questions:

1. What exact candidate and decision context are we evaluating?
2. What is the strongest supporting evidence?
3. What is the strongest contradicting evidence?
4. Are decisive records current and sufficiently high quality?
5. What decisive evidence is missing?
6. What exactly supports the confidence label, and is statistical calibration being claimed?
7. What change invalidates the result, and when is review due?

If question 6 resolves only to “the model thinks so,” confidence is not yet auditable.

Guided and independent practice

Run the offline lab:

python -B lab/AGN-002-LAB1/learner/run_checks.py

Then apply four mutations: remove the strongest support, make a decisive source stale, add a contradiction against the top candidate, and replace the confidence basis with only a model percentage.

For transfer, remove A's usable capacity evidence and add a verified current price for B. Recompute candidate and confidence from the changed evidence; do not preserve the baseline rank.

Verification evidence

Acceptance requires the learner's decision to be reproducible from visible deterministic evidence. The lab must catch at least:

• unsupported confidence;
• stale decisive evidence;
• missing decisive evidence;
• contradiction without appropriate review;
• transfer that copies the baseline winner;
• trace availability with inadequate factual evidence.

No network, API key, production credential, or real customer/vendor data is required.

Assessment and transfer

The learner must explain the lesson boundary, construct the baseline record, diagnose “strongest supporting evidence removed but confidence unchanged,” and recompute the changed-context transfer.

A strong answer identifies the invalid field, consequence, and minimal repair: for example, evidence_freshness=STALE → confidence basis invalid → HOLD → refresh evidence and recompute.

When A loses usable capacity evidence and B gains new verified price evidence, the rank is allowed to change. The goal is not stable conclusions; it is stable decision rules.

References and freshness

• NIST AI 600-1 — Generative AI Profile: official source for GenAI risk-management, evaluation, documentation, and monitoring boundaries; not a local confidence scale.
• OpenAI Agents SDK — Tracing: current source for traced generations, tools, handoffs, guardrails, and custom events; not proof of factual truth or calibrated confidence.
• Frozen Lesson Bible AGN-002: local authority for fields, state vocabulary, negative cases, and evidence contract.

NIST and Agents SDK sources were rechecked on 2026-09-14. Recheck the volatile SDK semantics by 2026-10-18, and practice-level claims by 2027-01-01 or earlier after material change.