Audience Fit and Drift Detection

1. The real problem: the audience label can stay stable while the people change

A product has an audience named “returning learners.” The label and dashboard stay unchanged, but a new campaign brings a different acquisition mix and advanced-lesson completion falls. If planning assumes the historical segment is permanently representative, decisions drift away from actual users. This lesson builds Audience Fit & Drift Evidence Monitor v1, separating audience definition, version, baseline, observed window, composition, and behavior.

2. Lesson boundary and owned artifact

The owned artifact is Audience Fit & Drift Evidence Monitor v1. It emits FIT, DRIFT_SUSPECTED, DRIFT_CONFIRMED, or INSUFFICIENT_DATA, while preserving UNKNOWN, DEFERRED, REVIEW_REQUIRED, or REJECTED when evidence is incomplete. PLN-011 owns cadence and PLN-013 owns context radar; this lesson does not issue frequency recommendations or crisis actions.

3. Version the audience definition

Every comparison records at least audience_version, segment_condition, and membership_rule. If a membership rule changes from static to dynamic in the middle of the observed window without a version change, the before/after records no longer represent one compatible population. Drift must not be computed as if nothing changed; use REVIEW_REQUIRED or DEFERRED until a compatible window exists.

4. What GA4 audiences actually demonstrate

Current Google Analytics documentation says audience conditions can be evaluated statically or dynamically and can use time-windowed metrics [GA4-AUDIENCE]. Under dynamic evaluation, users are included when they meet the condition and excluded when they do not. This is a vendor implementation, not a universal definition of audience truth or drift.

5. Cohorts compare behavior over time

Official Cohort Exploration guidance defines a cohort as users sharing a common characteristic, with an inclusion criterion for entry and a return criterion for subsequent behavior across daily, weekly, or monthly periods [GA4-COHORT]. This is useful evidence for behavior change, but cohort movement alone does not prove audience drift; composition, windows, and confounders still matter.

6. Baseline and observed windows must be comparable

baseline_window and observed_window are more than date ranges. Definition version, seasonality, acquisition mix, and coverage must be comparable. A 28-day baseline compared with a seven-day promotion spike can create a misleading delta. The monitor closes window_compatible before assigning a drift state.

7. Separate composition change from behavior change

composition_signal describes who entered the segment, such as channel, device, or entry-cohort mix. behavior_signal describes what the represented population did after entry, such as completion or return. Composition-only movement can raise a drift hypothesis, but it does not confirm behavior drift.

8. Sample size prevents weak evidence from becoming certainty

Each window carries sample_size and a fixture-specific minimum. The lab deliberately halves the sample while keeping the same metric movement. The same delta with insufficient evidence must not retain the previous conclusion; it becomes INSUFFICIENT_DATA. Real product thresholds are context dependent and are not universalized by this lesson.

9. Treat confounders as first-class evidence

Campaigns, seasonality, pricing changes, and tracking gaps can move metrics without proving that the audience itself drifted. confounder stays explicit. A one-week campaign spike must not automatically become DRIFT_CONFIRMED; a bounded DRIFT_SUSPECTED state plus a review trigger is more defensible.

10. A drift metric is composite evidence, not an override

This lesson uses separate composition and behavior deltas. In the synthetic fixture, both can support DRIFT_CONFIRMED only when definition/version, window, sample, provenance, and confounder gates are healthy. A summarized drift metric never bypasses those gates.

11. The four primary monitor states

FIT means current evidence remains compatible with the baseline. DRIFT_SUSPECTED means meaningful movement exists but confirmation is incomplete or a confounder remains open. DRIFT_CONFIRMED means composition and behavior both diverge from a comparable baseline with sufficient evidence. INSUFFICIENT_DATA means sample or coverage cannot support a valid judgment.

12. Provenance and review triggers make decisions auditable

Every row needs provenance_ref and review_trigger. If source, window, or version cannot be traced, an attractive metric is still not defensible and may become REJECTED. The trigger must be finite, such as after 200 more observations, after a campaign ends, or after a new baseline is collected for version 2.

13. Worked example: returning learners after a new campaign

Baseline version 1 uses a stable membership rule, 28 days, and adequate sample. The observed window is also 28 days, but acquisition mix changes and advanced completion falls. If both composition and behavior deltas cross the synthetic gates and no live campaign confounder explains the movement, the monitor returns DRIFT_CONFIRMED. If the campaign remains an unresolved confounder, only DRIFT_SUSPECTED is allowed.

14. Deterministic negative cases and failure probes

Four shortcuts are rejected: 1) membership rule changes mid-window without versioning → REVIEW_REQUIRED; 2) sample is halved while keeping the old conclusion → INSUFFICIENT_DATA; 3) a one-week campaign spike → DRIFT_SUSPECTED, not confirmed; 4) missing provenance → REJECTED. Composition-only or behavior-only movement is also insufficient for confirmation.

15. Transfer: static to dynamic without pretending the windows are compatible

Now the audience moves from a static membership rule to a dynamic rule. The contract remains the same, but audience_version must advance and a fresh baseline must be collected under the new rule. Version-1 and version-2 windows must not be compared as one stable segment. Successful transfer versions the definition, re-baselines the window, and recomputes state from new evidence rather than copying the old label.