Pilot Design and Stop/Pivot Criteria

1) A pilot is a decision instrument, not a small production project

A pilot exists to reduce uncertainty about a risky assumption. It is not a reward for activity, a demo backlog, or permission to keep building because money has already been spent. This lesson builds Pilot Hypothesis & Stop-Pivot Contract v1 so the final action is bounded by evidence.

2) Start with the riskiest assumption

GOV.UK alpha guidance recommends focusing on the riskiest assumptions and building only enough prototype to test the idea. Translate that into one falsifiable statement: if operators use the assisted workflow, verified review time will fall below the target without breaching the human-review safety guardrail.

3) Shrink to the minimum testable scope

Scope should include only what is necessary to expose the assumption. Authentication polish, broad integrations, billing, migration, and production hardening do not belong in this pilot unless one of them is the risky assumption being tested. More scope can increase sunk cost without increasing learning.

4) Freeze the baseline before interpreting the result

A result needs a comparison point. Record the current review time, current error/escape rate, and the exact measurement method before the pilot starts. A baseline rewritten after results are visible cannot support a clean decision.

5) Separate the primary measure from the guardrail

The primary measure answers whether the hypothesized benefit appeared. The guardrail asks whether the benefit was obtained safely. Faster verification is not success if the safety/error boundary fails. A guardrail breach has veto power over a flattering primary metric.

6) Bound the sample, timebox, and decision date

Name who or what is included, who is excluded, how long the pilot runs, and when the decision is made. Private beta guidance uses limited participation to learn and improve before broader opening; MAKNOW uses the same bounded-learning principle without importing GOV.UK sample-size rules.

7) Precommit STOP, PIVOT, and CONTINUE

Write the thresholds before results are known. CONTINUE means the hypothesis and guardrail both pass inside the declared boundary. STOP means a safety/guardrail or unrecoverable condition fails. PIVOT means the evidence rejects the current mechanism but supports a defined, testable change. HOLD means evidence is not yet sufficient.

8) Protect the decision from sunk cost

Money already spent is not evidence that the pilot should continue. The contract should be readable on the decision date without asking how much effort the team invested. If a criterion would change only because the result is disappointing, it was not a real precommitment.

9) Keep the production claim separate

A pilot can support learning and still be unfit for production. Security, reliability, capacity, compliance, operations, cost and release ownership may remain unproven. CONTINUE in this lesson means continue the bounded learning path—not “ship to production.”

10) Baseline Pilot Hypothesis & Stop-Pivot Contract

The baseline asks one question: can operators verify assisted outputs in at most 60 seconds while unsafe escapes remain at or below 2%? The pilot is four weeks, the sample boundary is explicit, and the decision date is fixed before execution.

11) Wrong shortcut: primary metric wins while the guardrail fails

Suppose median verification time falls from 90 seconds to 48 seconds, but unsafe escapes rise to 5%. The primary metric improved, yet the guardrail failed. The correct bounded action is STOP, not a celebratory average score.

12) Scope and threshold mutations invalidate the original contract

If the team expands the pilot after kickoff without rebinding the hypothesis, the evidence becomes PARTIAL. If it changes the success threshold after seeing results, the contract becomes DISPUTED. A pilot cannot remain auditable while the rules move underneath the data.

13) Lab: eight fixtures and 64 assertions

The offline lab evaluates baseline continuation, guardrail stop, recoverable pivot, scope mutation, missing evidence, post-hoc threshold change, stale evidence, and the integration transfer. Eight visible output fields are checked for every fixture; no API key or live tenant is required.

14) Transfer: integration reduces review time after an earlier guardrail breach

Recompute from new evidence. If the integration brings verification to 44 seconds and the safety rate to 1.5% under the same declared sample and thresholds, the bounded result can return to CONTINUE. The earlier failure is not erased; the new decision is supported by a new evidence record.

15) Delivery checklist

One risky assumption is explicit; minimum scope is bounded; baseline and measurement method are frozen; primary and guardrail measures are separate; sample/timebox/decision date are named; STOP/PIVOT/CONTINUE criteria are precommitted; missing, stale, disputed or changed evidence stays visible; transfer is recomputed; and no pilot outcome is presented as production readiness.