Chaos and Failure Testing

1. Problem and provable outcome

Breaking a system on purpose is not, by itself, a resilience experiment. A defensible experiment starts with a falsifiable hypothesis about steady state, bounds its blast radius, injects one declared fault into one declared target, watches explicit guardrails and stop conditions, and closes with recovery evidence. The owned artifact is Controlled Failure Experiment Record v1.

The lab is fully local and synthetic. It touches no production system, real customer traffic, real credential, live endpoint, or external target.

2. Lesson boundary and prerequisite

OPS-008 is the hard prerequisite. This lesson does not absorb the outcomes of Incident Response/Postmortems or rollout/canary control. The narrow outcome is to design and interpret one bounded failure experiment. A successful run is evidence about this hypothesis, scope, and observation set—not proof that the whole system is resilient.

3. Mental model: baseline to claim

Use this chain:

steady-state baseline → falsifiable hypothesis → bounded target/fault → guardrail monitoring → continue/stop → rollback/recovery check → supported/refuted/open claim → next experiment

Each arrow is an evidence boundary. Without a baseline, the hypothesis is not testable. Without a precise target, blast radius is not defensible. If recovery is reduced to “the process is alive,” service state remains unproven.

4. Steady state and a falsifiable hypothesis

Steady state describes acceptable behavior before the fault; it is not a vague statement that the system is healthy. In this fixture it can combine success rate, latency, and queue depth. A useful hypothesis can be disproved, for example: “If dependency delay rises for 20 seconds, success remains above 98% and queue depth remains below 40.”

“System is resilient” is not measurable enough to test and must be rewritten before execution.

5. Scope, target, and blast radius

Scope identifies the environment and service graph included in the experiment. Target identifies the exact synthetic component receiving the fault. Blast radius is the maximum permitted impact. In this lesson the blast radius is always restricted to the local fixture.

AWS Well-Architected guidance emphasizes clear scope, rollback capability, and stop conditions. We use that principle but do not invent organization-specific thresholds; every number in the exercise is fixture-specific.

6. Do not collapse action, target, and stop condition

AWS FIS models Actions, Targets, and Stop Conditions as separate experiment-template components. The separation matters: the fault action is what changes, the target is where it changes, and the stop condition defines when continuation is no longer acceptable.

The lab uses no AWS resource. It practices only the evidence model on local synthetic data.

7. Guardrails and the stop branch

A guardrail is a signal that must remain under observation during the experiment. A stop condition is a predeclared threshold or state that requires the experiment to stop. It must not be invented after seeing the result to make the run look successful.

When a guardrail is crossed, continuing merely to collect more data is methodologically wrong. Record the abort reason and the rollback/recovery state.

8. Observability and recovery evidence

Recovery is demonstrated by returning to steady state, not by process liveness alone. If the process is running again while latency remains out of bounds or queue depth keeps rising, recovery is UNPROVEN.

If a required signal is missing, the result remains OPEN/INCONCLUSIVE. Missing evidence cannot be replaced by a guess or a green summary badge.

9. Guided example

Baseline: success=99.6%, p95 latency=85ms, queue=12. The hypothesis says a bounded dependency delay will keep success above 98% and queue below 40. The fault targets dependency-B only. Guardrails watch success and queue.

If the run holds success=99.0% and queue=31, and all three signals return to the baseline window after fault removal, the hypothesis is SUPPORTED_BOUNDED for this scope. “The system is resilient” would overclaim.

10. Failure probe: crossing the stop threshold

In another fixture, dependency delay drives queue=57 while the predeclared stop threshold is 45. The correct decision is STOP_REQUIRED. If fault removal lowers queue only to 49, recovery is still not demonstrated.

This probe falsifies a common shortcut: “continue unless the process crashes.” Guardrails and steady state—not process liveness—govern the decision.

11. Control condition, repeatability, and unknowns

A control condition helps attribute the observed change to the injected fault. Preserve a pre-run snapshot, fault duration, target selector, and signal window so the experiment is repeatable. Repeatability does not eliminate every unknown.

Keep unknowns explicit: production interactions, real traffic, dependencies outside the fixture, and missing observability remain outside the claim.

12. Controlled Failure Experiment Record v1

Each row includes at least: Hypothesis, Steady-State Signal, Fault, Target, Blast Radius, Guardrail, Stop Condition, Observation, Recovery Evidence, Supported/Refuted/Open, and Next Experiment.

Keep Direct Evidence, Inference/Decision, and Unknown separate. If a decision requires organization policy that the fixture does not provide, leave it open.

13. Transfer practice

The second service graph changes topology, fault location, steady-state signal, and threshold. Do not copy the first hypothesis. Start from the new graph's observations and rebuild the record. Transfer passes only when the decision is independent of first-case identifiers and numbers.

14. Assessment and acceptance criteria

You must distinguish steady state, hypothesis, blast radius, and stop condition; construct the record before fault injection; reject false recovery based on process liveness; decide continue/stop/rollback/defer from guardrail evidence; and design a new experiment for the changed graph.

Three blocking failures are unsupported operational claims, happy-path-only evidence, and missing independent transfer.

15. Sources, freshness, and claim limits

This lesson uses current AWS Well-Architected, AWS Prescriptive Guidance, and AWS FIS documentation. On 19 September 2026, AWS guidance still describes controlled experiments, hypothesis/stop conditions, starting small, and bounded impact; AWS FIS still separates Actions, Targets, and Stop Conditions, and defines stop conditions through CloudWatch alarms.

Controlled Failure Experiment Record v1 is a MAKNOW instructional artifact, not an AWS-required document. Source review is due 17 October 2026. No result from the local fixture is generalized into a production-survivability claim.