Debug systematically and find the causal failure mechanism

Estimated active time: 75 minutes Defensible output: Root Cause Record v1

From “something is broken” to a bounded causal claim

A checkout system calculates the final amount incorrectly only at a specific coupon/tax boundary. Three code regions look plausible. A recent commit tempts you to blame it, and a warning appears in the log at the same time as the failure.

None of those observations is a cause yet. In this lesson, move from a repeatable symptom to a bounded Root Cause through a reviewable evidence chain: reproduction, competing hypotheses, discriminating probes, reduction, a bounded causal mechanism, a corrective change, and a recurrence sensor that turns red when the defect is deliberately restored.

The goal is not to prove that every failure has exactly one cause. A local regression check is also not a complete Test Strategy. If the evidence supports only a causal set or contributing condition, record that narrower boundary rather than enlarging the claim.

At the end, your Root Cause Record v1 must answer: “What exactly did we observe, which explanations did we test, what did we reject, what bounded mechanism remained, what change corrected it, and what evidence shows that the defect can be detected again?”

Choose support, not a lower standard

There are three routes, but all three finish at the same evidence bar.

• Foundation: use explicit definitions, one worked trace, and a partially structured Root Cause Record before the support fades.
• Repair: revisit only the fragile dimension, such as correlation versus cause or probe discrimination. The final acceptance criteria do not change.
• Experienced: move quickly through familiar explanations, but still produce the same reproduction, hypothesis ledger, claim boundary, transfer evidence, and human decision.

The practical prerequisite is a bounded repository map from the previous lesson. You should be able to connect an observed behavior to a few plausible code anchors rather than search the entire repository by guesswork. If that link is fragile, make a short bridge: input, observable execution path, three plausible regions, and one place where behavior can be measured.

Mental model: symptom is not cause

Keep five concepts separate:

1. Symptom: an observable mismatch you can describe and repeat.
2. Correlation: two events appear together. Correlation can guide investigation; it does not establish causation.
3. Hypothesis: a falsifiable explanation that predicts an observation.
4. Cause / Root Cause: the smallest causal mechanism or set of conditions supported by the current evidence. “Root” does not license an absolute claim.
5. Corrective action: a change aimed at the supported mechanism. Symptom disappearance alone does not prove that the action was causal.

Use this reasoning loop:

Observation → competing hypotheses → discriminating probe → update/reject → reduction → bounded causal statement → corrective change → recurrence evidence.

A useful probe does more than collect data. It should separate at least two live explanations. If both hypotheses predict the same result, the probe has little information gain for the current decision.

Quick check

If reverting the newest commit makes the failure disappear, you still need to show what mechanism the revert changed and obtain independent evidence that weakens competing explanations.

Foundation: bound the failure before explaining it

1) Reproduction note

A useful reproduction includes:

• Input: the exact data that activates the failure.
• State / preconditions: configuration, fixture, version, or state required.
• Expected behavior: what the governing policy or contract requires.
• Observed behavior: the actual measurable result.
• Repeat condition: how often and under what conditions it repeats.
• Boundary: the nearest cases that do not fail.

“Checkout is sometimes wrong” is not yet a bounded failure surface.

2) Hypothesis ledger

Write at least two competing hypotheses before the corrective edit. For each one, record its prediction, probe, result, status, and next discriminating probe. Chronology matters because it prevents the final story from being retrofitted onto the investigation.

3) Reduction

Remove dimensions while preserving the symptom. If the failure remains with an in-memory repository instead of a database, persistence becomes less plausible for this reproduction. Reduction is not “make the code as small as possible”; the symptom must survive.

4) Root Cause Record v1

The final artifact contains the reproduction, hypothesis history, rejected evidence paths, reduced case, causal statement, corrective change, recurrence sensor, limitations, support/AI disclosure, and final human decision. A polished narrative without an inspectable evidence trail is insufficient.

Deep explanation: increase confidence without jumping

Write hypotheses that can lose

Weak: “The tax code is probably broken.” Better: “If the tax calculation itself is causal, calling the tax function directly with the same taxable base should reproduce the mismatch.”

The second form predicts an observation. If the function returns the correct result under a controlled input, that hypothesis becomes weaker and its ledger status should change.

Design discriminating probes

Suppose you have three explanations:

• H1: the discount is calculated incorrectly;
• H2: the tax function applies the wrong rate to a correct base;
• H3: wiring sends the wrong base into the tax function.

A generic log dump may leave all three alive. Capturing subtotal, discounted_subtotal, and taxable_base_passed at the tax boundary can separate H3 from H1 and H2. Tie each probe to a decision boundary.

Use reduction to remove irrelevant explanations

A minimized case with one coupon, one rate, and one boundary is more useful than replaying the entire checkout stack. If the failure survives after the repository is replaced, correlated cache noise is weaker causal evidence even if the warning still looks suspicious.

Keep the causal statement small

Weak: “The tax system was broken.” Better: “Under condition X and state Y, wiring passed the pre-discount amount to the tax function. Direct probes showed that discount and tax calculation were independently correct, and changing that base selection repaired the reproduction.”

If two conditions are jointly necessary, record the causal set. Root Cause does not have to be one filename or one commit.

Make recurrence evidence detect the defect

After the correction, add a sensor that would fail before the fix and pass after it. Then reintroduce the defect in a controlled mutation. If the sensor remains green, it demonstrates only the current state, not recurrence detection.

Diagram text equivalent — “evidence-to-recurrence cycle”: one observation branches into several hypotheses. Each branch makes a prediction. A discriminating probe rejects some branches and keeps others. Reduction removes irrelevant dimensions. The remaining evidence supports a bounded causal statement. A corrective change targets that mechanism, and a recurrence sensor is checked against a controlled defect mutation. No branch wins merely because its commit is recent or its warning is noisy.

Worked example: one continuous evidence trace

In a training checkout, coupon WELCOME5 should discount the subtotal by 5%, then tax should apply to the discounted amount. For a subtotal of 200, the expected total is 205.20, but the system returns 206.00.

Reproduction: the input and policy are fixed; the mismatch repeats five times. Another coupon family does not fail.

Competing hypotheses:

• H1: the discount engine calculates the wrong amount.
• H2: the tax function has a rate or rounding defect.
• H3: a persisted policy for this coupon family selects the wrong taxable base.

Probe 1: observe the discount boundary: 10.00. H1 is rejected for this case.

Probe 2: call the tax function directly with base=190: 15.20. H2 is rejected for this input.

Probe 3: inspect the loaded policy object. An old fixture sets tax_after_discount=false. Replacing persistence with an in-memory repository containing the correct policy removes the failure; restoring the old policy brings it back.

Reduction: database behavior, cache warning, and the recent commit are removed from the reduced case. The old policy object is sufficient to reproduce the mismatch.

Bounded causal statement: in this fixture, the persisted policy selects the pre-discount taxable base. This does not claim to explain all tax failures or all coupon families.

Corrective action: repair/migrate that bounded policy family rather than adding a broad bypass.

Recurrence evidence: add a test that seeds the old policy and requires migration to correct it. Reintroducing the old value makes the sensor fail; the correction makes it pass.

This example demonstrates the method; it is not the answer to the lab. The lab requires new evidence and a different causal explanation.

Three failure modes that feel convincing

1) “The newest commit is probably the cause”

Recency is salient, not causal. Require a probe whose result would differ if the commit were truly causal versus at least one competing explanation.

2) “It is green, so Root Cause is fixed”

A flag can route around the failing path. That suppresses the symptom while the defect remains. Run a recurrence sensor against a defect mutation and explain the mechanism before closing the record.

3) “Root Cause must be one unique cause”

Some failures require interacting conditions. If the evidence supports only “state A + boundary B,” record that causal set. Compressing the story to one factor is an overclaim.

Formative check

A cache warning appears only during failing runs. What is missing? At least one probe that independently changes cache state or replaces the repository and observes whether the symptom persists. Co-occurrence is not enough.

Experienced bridge: convert familiar tools into evidence

You may already use debuggers, tracing, git bisect, profiling, log correlation, or feature flags. These are probes or reduction tools; they are not verdicts by themselves.

• git bisect can locate a behavior change, but you still need the mechanism.
• tracing can expose a wrong boundary value, but tracing without competing hypotheses may only create more noise.
• a feature flag can contain impact, but containment is not causal correction.
• a unit test that is green after the fix is useful, but recurrence evidence should show that it fails on the prior defect.

Fast path means less explanation, not less evidence. If you can build the reproduction and competing hypotheses quickly, move directly to discrimination and reduction; still deliver the Root Cause Record, limitation, transfer evidence, and human decision.

Trade-offs: change the probe, defer, or escalate

No single troubleshooting technique is best for every failure.

• Black-box reproduction is useful when the behavioral boundary is not yet stable.
• Instrumentation/tracing is useful when several components can transform a value and you need to observe boundaries.
• Reduction is useful when noise is high and dependencies can be removed while preserving the symptom.
• History tools such as bisect are useful when a behavior change is stable across versions, not merely because a recent commit exists.
• Containment/workarounds may be operationally necessary, but they should be recorded separately from causal closure.

If the failure is nondeterministic, required data is unavailable, or external systems cannot be controlled, the correct decision may be bounded defer or escalation. State exactly what is not proven and what evidence would be needed next.

Guided practice, then independent transfer

Guided pass

For a small failure, fill six items:

1. exact symptom and boundary;
2. two competing hypotheses;
3. one discriminating probe;
4. one rejected alternative with evidence;
5. a reduced case;
6. a provisional causal statement plus a non-claim.

The Foundation route starts with a partially structured artifact and ledger: a few already-known observations are labeled and there is exactly one suggested next probe. After the first pass, remove that suggested probe and continue in the blank artifact using your own decision. No prefilled prompt or structure remains for independent transfer.

Local checkout lab

The local checkout fixture contains a boundary defect. Three code regions are plausible, a recent-commit cue is intentionally misleading, correlated log noise is present, and a compatibility option can hide the symptom.

You must:

• reproduce the failure from a stated input/state;
• record at least two hypotheses before modification;
• reject at least one alternative with a discriminating probe;
• reduce the case enough to constrain the boundary;
• apply a bounded corrective change and record the original reproduction result after correction;
• create a recurrence sensor;
• deliberately reintroduce the defect and show that the sensor fails on the mutation and passes after correction;
• disclose documentation/AI/tool support, independently validate the evidence, and make the final decision yourself.

Support timing boundary: AI or documentation may suggest additional search/probe options only after the required retrieval/first-diagnosis probe has been closed. Support does not replace closed evidence, independent validation, or the human final decision.

No external network, production data, secrets, or destructive commands are required.

Self-check before independent transfer

Before S5, inspect the scaffold-faded Root Cause Record v1. Evidence, decision, and limitation fields must be reviewable, and no unsupported inference may be promoted to fact. If this self-check fails, repair only the failing dimension and inspect the record again; the final evidence bar is unchanged.

Independent transfer with an executable shipping fixture

The independent transfer_shipping/ fixture reproduces a different failure at a 10.0 kg weight boundary. A recent UI cue and a database warning are present, but neither is predeclared as causal. Use that fixture's tools to produce a new reproduction and new boundaries, record at least two new hypotheses, choose probes appropriate to this context, and create new recurrence evidence. Do not copy checkout paths, evidence, or answers.

At this stage, Foundation scaffolds and the suggested-next-probe cue are not allowed. Preserve only the method invariants: reproduction, hypothesis/refutation, reduction, bounded claim, and mutation-backed recurrence.

What counts as reviewable evidence

A reviewer should be able to follow this chain without guessing:

1. Reproduction quality: input/state/expected/observed/repeat/boundary are explicit.
2. Hypothesis discipline: at least two real explanations are recorded and one is rejected by evidence.
3. Reduction quality: at least one irrelevant dimension is removed while the symptom remains.
4. Bounded causal statement: the statement is tied to evidence and includes limitations.
5. Corrective change: the change targets the supported mechanism rather than cosmetic symptom suppression.
6. Mutation-backed recurrence: the sensor fails on the reintroduced defect and passes after correction.
7. Human decision: you choose accept / modify / reject / bounded defer with rationale.

Closure is invalid when the record is narrative-only, when green output replaces causal explanation, when AI/tool output is treated as final authority, or when transfer reuses old evidence instead of producing new evidence.

A local regression sensor demonstrates recurrence detection for the bounded defect. It does not prove that the system has a complete Test Strategy.

Assessment and transfer

S1 — Retrieval

A failure has both a recent commit and a simultaneous warning. Identify what is an observation, what is a hypothesis, and what is not yet a cause. Name two evidence steps that would raise causal confidence.

S2 — Application

From the supplied bounded case, produce the central Root Cause Record v1: reproduction, at least two hypotheses, probes/results, a rejected alternative, reduced case, bounded causal statement, corrective change, recurrence sensor, and limitation. Narrative-only output is insufficient.

S3 — Diagnosis

A team enables a compatibility flag, the main test becomes green, and the team declares the Root Cause fixed. Identify the first unsupported promotion and propose one discriminating repair probe.

S4 — Decision

Choose accept, modify, reject, or bounded defer for the candidate conclusion. Record evidence, limitations, one explicit non-claim, support disclosure, and your human rationale.

S5 — Transfer

In the executable shipping fixture, the shipping-cost service fails only at a 10.0 kg boundary. A recent UI commit and a database warning are also present. Without copying checkout paths, evidence, or answers, execute the method with probes appropriate to this context and create new evidence.

Passing condition: the five frozen critical dimensions are Reproduction quality, Hypothesis/refutation discipline, Reduction quality, Bounded causal statement, and Mutation-backed recurrence evidence. Failure in any one is blocking. Evidence traceability, the human decision with limitations/support disclosure, and changed-context transfer are also blocking hard conditions. No arithmetic average can compensate for failure in any of these requirements.

Reference, freshness, and claim boundary

The methodological foundation aligns with Google SRE — Effective Troubleshooting: troubleshooting can use observations and system understanding to form hypotheses, test/refute them, and simplify or reduce the failure surface. The source is not used to claim that every failure has exactly one Root Cause.

• Recorded retrieval: 2026-09-02
• Review due: 2027-09-02
• Freshness class: foundational stable

For this lesson, closure requires a bounded causal statement plus recurrence evidence. This is an instructional design rule for the lesson, not a fact attributed to the external source.

Preserve four non-claims: every failure does not have one unique cause; recent change does not equal causation; a green rerun alone does not prove the causal mechanism; a local recurrence check is not a complete Test Strategy.

Provenance: if documentation, AI, or other tools suggest hypotheses or probes, disclose that support. Blocking evidence and the final sufficiency decision must remain independently defensible by you.