Repair Diagnostics and Isolated Reproduction

1. Problem: fixing inside the full system can hide causality

Repairing a failure directly in the full system can change multiple variables at once and erase the original causal boundary. This lesson produces Isolated Reproduction & Diagnostic Evidence Packet v1: a diagnostic packet that keeps failing input, environment, oracle, trace, suspect range, and boundary owner explicit and auditable.

2. Lesson boundary

PLN-027 does not own release/canary decisions; PLN-028 owns that outcome. It also does not absorb the PLN-026 canon boundary. The owned task is to reduce a failure to the smallest reproducible boundary and distinguish application/SDK-owned behavior from behavior that requires a real model, network, or sandbox provider.

3. External FACT versus local DESIGN

git bisect uses binary search between known old/good and new/bad revisions to narrow the revision that changed a property, but it depends on a test oracle that can classify candidates. OpenAI Agents SDK testing provides deterministic provider-neutral doubles that exercise application/SDK orchestration without real model or sandbox-provider requests. Those doubles do not prove real provider behavior. Packet fields and typed states here are MAKNOW DESIGN.

4. Separate incident and symptom from hypothesis

incident_id and symptom_id identify what was observed; diagnostic_hypothesis is provisional explanation. Changing the hypothesis must not rewrite incident identity. Diagnosis starts from observable failure evidence, not a preferred cause story.

5. Freeze the failing input

The failing input needs stable identity or a hash. If the prompt, tool input, attachment, or event sequence changes during testing, the new result is not the same reproduction. Bind failing_input_hash to a fixture and trace.

6. Environment fingerprint and dependency lock

Record runtime, dependency lock, config hash, agent-definition hash, and tool-contract hash. Changing two dependencies at once destroys attribution. If the reproduction environment does not match the failing environment, do not emit success; keep the state at HOLD.

7. The oracle must be explicit and stable

An oracle defines expected versus actual behavior. “The answer looks wrong” is not a reproducible oracle. It must be deterministic enough to classify bisect candidates consistently. A flaky oracle yields INCONCLUSIVE.

8. Trace and fixture make evidence replayable

fixture_id and trace_id must be traceable. The trace shows calls and state transitions leading to failure. Keep the fixture minimal; copying production state increases privacy risk and diagnostic noise.

9. Last known good and first bad candidate

With a valid revision boundary, last_known_good and first_bad_candidate define a suspect_range. git bisect can reduce that range only when candidates are buildable/testable and the oracle remains stable.

10. Name the boundary owner

Useful owners include APPLICATION_SDK, MODEL_PROVIDER, SANDBOX_PROVIDER, NETWORK/PROTOCOL, and UNKNOWN. If a scripted double is clean while a real provider reproduces the failure, reclassify the boundary to the provider; an offline green test cannot exonerate provider-owned behavior.

11. What does a test double prove?

A test double is strong evidence for tool routing, handoffs, guardrails, retries, normalized sandbox calls, and application orchestration. Real sandbox lifecycle, filesystem fidelity, process execution, resource limits, and isolation belong in provider integration tests. Do not promote SESSION_LOCAL success into provider or production proof.

12. Baseline — failure inside the application/SDK boundary

Incident INC-517 reproduces after a dependency upgrade with the same failing input, same environment fingerprint, and a stable oracle. The scripted model also reproduces the tool-argument normalization failure; last-known-good=r140, first-bad=r147. With one changed variable, matching environment, and stable oracle, the packet reaches REPRODUCED and is bisect-ready.

13. Dangerous shortcuts

(1) Change two dependencies together → INCONCLUSIVE. (2) Use a scripted sandbox to claim a real provider is fixed/broken → EXTERNAL_UNTRUSTED/HOLD. (3) Feed a flaky oracle into bisect → a narrowed range is not reliable causal evidence. (4) Ignore environment mismatch → HOLD.

14. Transfer — failure exists only with the real sandbox provider

The scripted double is clean, but the same fixture fails with the real sandbox provider. Do not copy the offline conclusion. Reclassify boundary_owner to SANDBOX_PROVIDER, set integration_required=true, and design the smallest integration reproduction that preserves the relevant lifecycle/filesystem/process/isolation behavior. Until that exists, the state remains HOLD.

15. Lab, assessment, and freshness

Run python run_checks.py under lab/PLN-027-LAB1/learner/. Eight fixtures cover Baseline, Two Dependencies Changed, Provider-double Overclaim, Flaky Oracle, Environment Mismatch, Missing Provenance, Stale Evidence, and Real-provider Transfer. Git and OpenAI Agents SDK FACT claims were rechecked on 2026-09-16; canonical changes require a new version.