Component Catalog and Visual Regression

A green screenshot check can still describe the wrong product

A component can look perfect in one screenshot and still be poorly specified. The catalog may show only the happy path while loading, error, long-content, keyboard-focus, and direction-sensitive states are absent. A first screenshot can be saved as a baseline before anybody reviews whether it represents the intended design. A test can also fail every run because time, animation, fonts, viewport, or random data are drifting. Finally, a zero-pixel diff can tempt a team to declare the component correct even though keyboard behavior, semantics, accessibility, data rules, or business logic were never exercised.

WEB-018 turns visual testing into an explicit evidence contract. The bounded fixture is a reusable card component. You will create a component catalog whose stories make meaningful supported states inspectable, qualify a reference image as a reviewed baseline for one pinned environment, stabilize volatile rendering inputs, compare screenshots, classify changes as expected change, regression, or noise, and state what the visual oracle cannot prove. Completion means producing Component Catalog & Visual Regression Evidence Record v1 rather than merely generating images.

The durable lesson is not “use Storybook” or “use screenshot tests.” The durable lesson is to define the state inventory, control enough of the environment to interpret the output, give baseline changes an authority trail, and make an engineering judgment about diffs without pretending pixels are a universal oracle.

Scope, route, and evidence bar

This lesson assumes WEB-011's component/state foundation. A component story is useful here because it makes one supported rendered state addressable and repeatable. WEB-012 remains separate: if a card's data comes from a server or URL state, that ownership still matters, but this lesson does not redesign the query cache or router. Its concern is the catalog and visual evidence surface.

Use the foundation route if a component explorer and image baselines are new. Use the repair route if you already write stories but routinely accept the first snapshot or chase noisy diffs. Use the experienced route if you can skip syntax and go directly to coverage, determinism, baseline governance, and transfer. All routes must finish with the same evidence dimensions: implementation artifact, observed visual/browser behavior, mutation result, repair result, changed-context transfer, and residual limitation.

The lab is local and non-destructive. It contains an inspectable story inventory and a browser-based screenshot harness. The public checker can verify files and evidence shape, but it cannot fabricate observations. Browser evidence belongs in your record only after you actually run the harness and inspect the result.

Mental model: requirement inventory → stories → stable render → reviewed baseline → diff → decision

Visual regression becomes reliable when each arrow in the pipeline has an owner.

1. Requirement/state inventory. Start from supported product states, not from whichever stories happen to exist.
2. Stories. Give each important state a reproducible render input: default, loading, error, long title, focus-visible, and any other state justified by requirements.
3. Stabilized render environment. Pin browser family/version where practical, viewport, device scale, fonts/data, and volatile inputs such as time, random values, and animation.
4. Reviewed baseline. A reference screenshot is an approved expectation for that state/environment, not truth created automatically by first execution.
5. Screenshot comparison. Produce the current image under the same qualified environment and calculate the visible change.
6. Diff classification. Decide whether the change matches an intentional requirement, is a regression, or is noise from an unstable input/environment.
7. Action and non-visual evidence. Accept/update the baseline only with a reason, fix the regression, or remove the noise source. Then keep interaction/accessibility/business checks separate.

This model makes two failure boundaries visible. Missing stories are coverage failures before screenshot comparison begins. Uncontrolled environment changes are evidence-quality failures even if the component code is correct.

Foundation: five contracts that must stay distinct

1. Story-coverage contract

A catalog is not a list of files; it is an inventory of meaningful supported states. Storybook currently describes a story as a rendered state of a UI component and encourages multiple stories for interesting states. That does not mean every theoretical permutation deserves a story. Coverage follows risk and requirements.

For the card fixture, the required states are default, loading, error, long-title, and focus-visible. The state list is intentionally small enough to inspect and large enough to expose different failure classes. Loading and error exercise status presentation. Long-title challenges wrapping and spacing. Focus-visible makes a keyboard-oriented visual state addressable. If the real product later supports selected, disabled, RTL, compact, or dark-theme variants, those enter the inventory only when the contract requires them.

A useful review question is: “Which supported failure could happen without any story changing?” If the answer names a realistic state, the catalog has a coverage gap.

2. Baseline-authority contract

The first image generated by a tool is evidence of what the environment rendered, not evidence of what the product should render. A baseline becomes authoritative only after a human or governed process reviews it against the requirement and records why it is acceptable.

The evidence record therefore stores baseline status, environment signature, reviewed state set, and change reason. When an intentional redesign changes spacing, updating the baseline is legitimate after review. When a test fails because a developer accidentally removed card padding, clicking “accept” would erase the signal. The difference is not pixel count; it is requirement authority.

Baseline updates should be reviewable changes, not an automatic cleanup step after failures.

3. Visual-determinism contract

Screenshot comparison is useful only when repeated runs are comparable enough to interpret. Playwright's current documentation explicitly warns that host OS, browser/version, settings, hardware, headless mode, fonts, and related factors can alter rendering. The lab therefore records its environment instead of pretending images are universally portable.

Within one environment, remove avoidable volatility. Use fixed fixture data. Freeze time where time appears in the UI. Seed or remove randomness. Disable or settle animations. Use a fixed viewport and device scale. Make font assumptions explicit. Wait for the component's intended steady state before capture.

Determinism does not mean “every pixel is identical on every computer.” It means the tested environment is qualified tightly enough that a diff can support a bounded engineering conclusion.

4. Diff-judgment contract

A visual diff is a signal, not a verdict. The same changed pixels can mean three different things.

• Expected change: a reviewed requirement changed, the screenshot changed accordingly, and the new baseline should be approved with a traceable reason.
• Regression: the requirement did not change but the implementation drifted, such as lost spacing, clipping, incorrect color, or a missing focus indicator.
• Noise: the product contract did not meaningfully change, but an unstable clock, animation frame, font/environment mismatch, or random datum changed the image.

The job is to classify the diff using context. Blind rejection wastes time on noise. Blind acceptance launders regressions into new baselines.

5. Oracle-boundary contract

Pixels answer visual questions. They do not prove every other dimension of correctness. A screenshot can show a focus ring, but it does not prove Tab reaches the control in the correct order. It can show text that looks like a heading, but it does not prove semantic heading markup. It can show an error message, but it does not prove the business rule that triggered it is correct.

Keep visual evidence alongside, not instead of, interaction, semantic/accessibility, and business evidence. WEB-018 is complete only when the record states at least one residual limitation.

Deep explanation: why visual systems fail quietly

Visual testing often fails through process shortcuts rather than image algorithms. The first shortcut is story-by-example: developers write a Default story, maybe a Loading story, and conclude that the catalog “covers the component.” The missing Error or long-content state remains invisible because no test can compare a state that has no story.

The second shortcut is baseline-by-generation. A new screenshot appears, the tool says no baseline exists, and the image is committed. That workflow has established history, not correctness. If the first image already contains clipped content, future runs will faithfully protect the clipped design.

The third shortcut is tolerance inflation. When animation or font drift causes noise, a team may increase a pixel threshold until tests become quiet. Thresholds can be appropriate for known rendering variance, but using them to hide unstable inputs weakens the oracle. Stabilize the input first; tune comparison only after you understand what variability remains.

The fourth shortcut is universalizing one environment. “It passed in Chromium on Linux” is precise. “The visual is correct everywhere” is not. A baseline is qualified by the environment in which it was produced.

The fifth shortcut is confusing visual sameness with behavioral sameness. A refactor can preserve pixels while breaking keyboard activation. Conversely, a rendering engine update can change anti-aliasing pixels while behavior and intended appearance remain acceptable. Engineering judgment sits between raw diff and product decision.

Worked example: a reusable card with five states

The guided fixture contains a card with a title, body/status region, and action button. Build five stories.

Default uses a normal short title and ready content. It establishes the ordinary spacing and typography baseline. Loading replaces content with a stable skeleton/status surface. It must avoid animated shimmer during screenshot capture unless the animation is deliberately frozen. Error shows a bounded error presentation. LongTitle uses content long enough to wrap and stress vertical rhythm. FocusVisible renders the action with the visible focus treatment required by the design contract.

Before creating baselines, record the environment: Chromium family, fixed 900×640 viewport, device scale 1, fixed fixture data, stable clock, animation-disabled capture policy, and the font assumption used by the fixture. Generate images, inspect all five, and mark the baseline review as approved only after verifying each state against the requirement.

Now inject a real spacing regression by changing the card padding from the approved token to a larger or smaller value. Capture again. The diff should affect the card geometry across relevant states. Because neither the requirement nor baseline authority changed, classify this as regression, repair the token, and rerun until the evidence returns to the approved expectation.

Then inject volatile noise by allowing a live timestamp or random value into the captured card. Two consecutive captures can differ even with identical component code. Classify this as noise, restore fixed data/time, and rerun. Do not accept a new baseline for each random frame.

The final evidence record separates what was observed from what was inferred. “The comparison reported 8,412 changed pixels after the padding mutation” is an observation. “The padding mutation created a genuine layout regression in this pinned environment” is an inference connected to the unchanged requirement.

Failure probes: make the system prove it can fail

Mutation A — remove an important story

Delete LongTitle or FocusVisible from the catalog. Predict the failure before running checks: the visual system can no longer protect that supported state because there is no addressable case to capture. The repair is to restore the state with explicit fixture data, not to rename Default and claim coverage.

Mutation B — change a spacing token

Alter the card padding while leaving requirements and baseline authority unchanged. A deterministic screenshot comparison should produce a meaningful diff. Classify it as regression and repair the token. If the test does not react, the capture surface or comparison may be too weak.

Mutation C — introduce time/random/animation volatility

Allow a live clock, unseeded random value, or in-progress animation into the image. Consecutive captures should expose instability. The correct repair is to stabilize or deliberately mask that volatile input with a documented reason. Automatically blessing every new screenshot converts noise into baseline churn.

Mutation D — replace the baseline without review

Change the reference image or baseline metadata while leaving no requirement/change record. The bytes may now match, but baseline authority is broken. The checker should reject the unreviewed replacement even if the visual diff becomes zero.

Experienced bridge: treat the catalog as a state model and the baseline as governed data

For an experienced engineer, the useful abstraction is not “Storybook page plus snapshots.” The catalog is a product-state model. Stories are named fixtures for states the team agrees to support. Visual baselines are governed data derived from those fixtures under a qualified render environment.

That framing improves review. Ask whether state coverage is risk-driven, whether fixture data is stable, whether environment signatures are reproducible, whether baseline changes have reasons, and whether the diff classifier can distinguish requirement change from implementation drift and infrastructure noise.

It also clarifies what to store in version control. Story definitions, render configuration, baseline artifacts, and review metadata should change together when the contract changes. A baseline-only update with no requirement explanation deserves scrutiny.

Alternatives and trade-offs

Storybook is a concrete catalog surface, not the only possible one. A project can use another component explorer or a small dedicated fixture app. The durable requirement is that supported states are individually addressable and reproducible.

A cloud visual-testing service can provide browser matrices, hosted baselines, review UI, and CI integration. A local Playwright-style screenshot harness can be simpler and fully offline. The trade-off is operational: local systems make environment ownership explicit but require you to manage artifacts and review workflow; hosted systems can simplify collaboration but add service configuration and external execution. WEB-018 does not require a vendor.

Pixel thresholds and masks are also tools, not escapes. Masking a volatile third-party timestamp can improve signal if the timestamp is outside the owned visual contract. Masking the card body because it frequently fails would destroy useful coverage. Every tolerance or mask needs a reason tied to the contract.

DOM snapshots answer a different question. Storybook's current guidance notes that markup snapshots can become noisy and that visual or interaction tests may be a better fit depending on the concern. Use the oracle that matches the question rather than accumulating every testing mode by habit.

Practice: repair the guided fixture, then transfer the method

Open lab/WEB018-LAB1/learner. Start with the requirements and story inventory before editing the visual harness. The starter intentionally omits required states and leaves the baseline unreviewed. Add the missing stories with stable fixture data. Review the baseline metadata only after inspecting the rendered states. Set the deterministic capture policy so repeated runs are interpretable.

Run the local harness in baseline mode, then compare mode. Record the environment signature and actual observations. Apply the spacing mutation, predict the class, run the comparison, and repair it. Apply the volatile-noise mutation, capture twice, and explain why the resulting difference is noise rather than a product regression.

The independent transfer uses a notification/status component with the same evidence method but different content and layout. It still needs representative states, a qualified baseline, deterministic fixtures, diff classification, and an oracle-boundary statement. Do not copy the guided card's state names mechanically if the transfer requirement uses different meaningful states.

Verification: what the Evidence Record must contain

The record is the inspection surface for the lesson. It contains the state inventory and rationale, the pinned environment, baseline authority and review reason, observations from screenshot runs, diff decisions, one deliberate mutation and repair, the transfer result, and a residual limitation.

Separate observation from inference. “FocusVisible screenshot exists and the capture report hashes it under environment signature X” is observation. “The focus visual is protected against this class of appearance regression in that environment” is inference. “Keyboard navigation is accessible” is not justified by that screenshot alone.

A valid residual limitation might state that the fixture does not prove screen-reader announcements, keyboard order, business-rule correctness, or portability to another OS/browser/font stack. Precision strengthens the evidence; it does not weaken it.

Assessment and transfer

Retrieval: state the five contracts and one non-goal. Application: complete the guided card catalog and evidence record. Diagnosis: predict and detect one spacing regression and one volatility failure. Decision: defend a baseline update, rejection, or noise-removal decision using the requirement and environment record. Transfer: solve the changed notification/status component without decisive hints.

Blocking failures include missing a required state, treating first-run screenshots as approved, accepting an unreviewed regression baseline, leaving volatile inputs uncontrolled while claiming deterministic evidence, or using visual success as sole proof of interaction/accessibility/business correctness.

Completion is not “all screenshots are green.” Completion is a catalog with justified state coverage and an evidence trail that reacts correctly when you intentionally break coverage, rendering stability, baseline authority, or styling.

References and freshness boundary

Current Storybook documentation defines stories as rendered component states, uses Component Story Format to describe those states, and distinguishes visual tests from DOM/HTML snapshot tests. Current Playwright documentation describes reference screenshot comparison and warns that rendering varies with OS, browser version/settings, hardware, headless mode, fonts, and other environment factors. These tool-specific claims were rechecked on 2026-09-13 and remain bounded to the cited guidance.

The stable method survives tool changes: enumerate meaningful states, render them reproducibly, review baseline authority, classify diffs with context, and keep the visual oracle inside its evidence boundary. Re-review tool details by the recorded review date or earlier if Storybook/Playwright guidance or the lab environment changes.