Experiment Model: Separate Observation, Hypothesis and Change

1. The real problem: observe first, invent the story later

A team sees landing-page completion fall from 48% to 39% in a bounded synthetic window and immediately proposes changing the headline, navigation, and signup form together. If the metric later improves, the team cannot tell which change mattered or whether the treatment caused the movement at all. This lesson turns an experiment from “change things and explain later” into an auditable evidence model.

2. Lesson boundary

This lesson owns Experiment Hypothesis & Change-Isolation Card v1. A prior observation may enter from the preceding lesson, but trend/pattern discovery is not owned here. Event/dimension/metric-regime semantics are expanded in the next lesson. The bounded question is: what was observed before exposure, what is the falsifiable hypothesis, what exactly changes, and what evidence permits a limited causal interpretation?

3. Freeze the observation before the hypothesis

An observation is descriptive: which metric moved, for which population, during which window, under which query and instrumentation. Example: “In the synthetic fixture, completion for new users is lower in W2 than W1.” The statement must not smuggle in a cause. “The headline caused the drop” is already a hypothesis. Freeze the window, query, population, and provenance before designing the treatment.

4. Make the hypothesis falsifiable

NIST describes experimental design as planning controlled factor changes in advance and observing their effects on responses [NIST-DOE]. A useful hypothesis names an expected mechanism and what evidence would weaken it. Example: “If headline ambiguity contributes to lower completion, changing only the headline to a clearer value statement should improve completion versus control while the form and navigation stay fixed; otherwise the hypothesis is not supported.”

5. Isolate the treatment change

Changing multiple independent variables at once makes the learned claim about a bundle rather than a specific mechanism. If headline, form, and navigation all move, the result cannot isolate the headline hypothesis. In this lesson the treatment contains one decisive change, or any bundled change must be explicitly declared as the treatment itself. More than one independent treatment variable routes the decision to ISOLATE_TREATMENT.

6. Define control and experimental unit

Control preserves the condition that does not receive the treatment. The experimental unit is what gets assigned: user, account, session, device, or cluster. The wrong unit can expose the same person to both arms through different identities. The card therefore records the unit, the reason for choosing it, and any dependence or interference constraint.

7. Assignment, randomization, and interference

NIST describes completely randomized designs as randomly assigning factor levels to experimental units [NIST-RAND]. In digital systems, assignment may be randomized or implemented through a stable hash, but the rule must be fixed before exposure and cross-arm leakage must be controlled. If one user can enter both control and treatment through multiple identities, assignment is not valid until the interference path is repaired.

8. Lock the primary metric and guardrails before exposure

The primary metric should directly operationalize the hypothesis, while guardrails protect against a local win that causes unacceptable side effects. Completion rate may be primary while form-error rate and time-to-complete are guardrails. Switching the primary metric after seeing the result is post-hoc interpretation. The card declares the metric, direction, denominator, guardrails, and each metric's role before exposure.

9. Sample plan, exposure window, and analysis plan

This lesson does not invent a universal sample-size or significance threshold. The sample plan should state population, unit, minimum exposure window, and practical constraints; the analysis plan should state the comparison and how uncertainty will be reported. Statistical thresholds must be context-specific. Missing design evidence should remain missing rather than being replaced by a generic number.

10. Stopping rule and peeking risk

If a team repeatedly inspects results and stops only after the preferred metric rises, the stopping rule has been contaminated by the outcome. Predeclare the stopping condition: a fixed window, a sample-plan boundary, or an explicitly defined safety trigger. Early stopping can be legitimate when the trigger existed beforehand; “stop when it looks good” is not a valid rule.

11. Result is not the same as causal interpretation

Microsoft Research describes controlled/A-B experiments as a way to evaluate the impact of product changes and emphasizes trustworthy experimentation [MS-EXP]. A metric difference alone is still not a causal license. Instrumentation, assignment, interference control, uncertainty, and predeclared rules must remain credible. The card therefore keeps Result separate from Causal Claim Boundary.

12. Decision gate

The table below shows the main evidence states. The goal is not a single boolean; the state identifies the broken or satisfied design dimension and the next bounded action.

13. Worked example: landing page

Frozen observation: completion for new users is lower in W2 than W1. Hypothesis: headline ambiguity creates friction. Treatment: change only the headline; keep form and navigation fixed. Unit: stable user. Assignment: randomized 50/50. Primary metric: completion rate. Guardrail: form-error rate. Stop rule: predeclared window plus a safety trigger. If the result is measured with valid instrumentation and reported uncertainty, the team may report a bounded treatment effect—not the universal claim that the headline is always the cause of lower conversion.

14. Negative cases

Three shortcuts are rejected deliberately: (1) change headline and form together, so treatment is not isolated; (2) keep checking results and rewrite the stopping rule after the preferred metric rises, so predeclaration is broken; (3) let the same user enter both arms through different identities, so assignment and independence are compromised. A fourth failure is changing the hypothesis or primary metric after observing the outcome, which blocks a causal claim.

15. Transfer: email subject-line experiment

A user may appear in multiple campaigns, so copying the page-test unit is unsafe. Rebuild the unit and interference control—for example, stable assignment at a user-cluster level during a defined window, no cross-arm exposure, subject line as the only treatment, predeclared open/click/conversion metrics, and unsubscribe as a guardrail. The experiment model remains stable while context-specific design details are recomputed.