TTS, Forced Alignment & Word Sync

When a timestamp looks like truth

The same paragraph is synthesized in two browsers. Browser A emits some word boundary events with charIndex; Browser B emits sentence-level boundaries only. A separate forced aligner estimates token timings for another audio asset, but a token near punctuation drifts by 180 ms. If the UI highlights that token without further checks, it is inventing precision.

This lesson builds Speech Timing & Word Synchronization Contract v1. Text and token identity must be explicit; every timing value needs provenance and a known timebase; confidence and drift are separate evidence; and when word-level timing cannot be defended, the system degrades to sentence/cue synchronization instead of promoting weak timestamps to “exact sync.”

Route orientation and ownership boundary

MKN-010 owns TTS, Forced Alignment, and Word Sync. MKN-007 is the hard conceptual prerequisite and MKN-009 is soft. This lesson does not design the updater scheduler from the previous lesson and does not absorb the end-to-end multimedia accessibility work owned by MKN-011. The bounded question is: what evidence is required before a UI may claim that a token is synchronized to audio, and how should granularity degrade when that evidence weakens?

Foundation learners get a field map and one complete baseline. Repair learners isolate only the broken dimension such as text_version or drift_ms. Experienced learners lose the supplied decision label and must defend state and fallback from evidence. The checkpoints are F1 boundary retrieval, F2 rejection of an invalid shortcut, and F3 recomputation for recorded human narration.

Mental model: text version → token → timing sources → timebase → confidence/drift → sync or fallback

The decision chain is:

text version → token map → TTS boundary | forced alignment | timed-text cue → normalize timebase → confidence + drift checks → word sync | sentence/cue fallback → human review

First establish which text version the audio represents. Tokenize and preserve character offsets from that same version. A timing source may be a browser event, a forced-alignment estimate, or a timed-text cue; these sources are not equivalent. Normalize all timestamps to one timebase before comparison. Then inspect confidence and drift. If word evidence is weak but cue evidence is sound, a lower-granularity fallback is valid. If neither word evidence nor a defensible fallback exists, the state remains HOLD.

Foundation contract: sixteen auditable dimensions

utterance_id and text_version identify the text/speech execution under review. token_id, char_start, and char_length preserve text identity. audio_asset_ref binds evidence to audio. timing_source records provenance. start_ms, end_ms, and elapsed_ms hold normalized time values. confidence records trust in alignment or mapping. cue_id links lower-granularity timed-text evidence. drift_ms captures disagreement or offset from the expected position. sync_state records the current decision, fallback_mode records the lower granularity, and review_owner preserves accountability.

For this deterministic teaching artifact, local MAKNOW thresholds are alignment confidence >= 0.85 and maximum word drift <= 120 ms. These values are not web standards; they are explicit design rules for this lab.

Deep explanation: keep FACT, INFERENCE, and DESIGN separate

Web Speech API — FACT. A synthesis boundary event can be fired at word or sentence boundaries when the speech engine provides it. charIndex only approximates the speaking position in the original utterance and is not guaranteed to sit on a word boundary. charLength and elapsedTime may be 0 when the engine cannot provide or the user agent cannot determine them.

W3C Timed Text — FACT. Timed text is text associated with timing information and is used for subtitle/caption and other synchronized-media workflows. A timed-text cue is useful temporal representation, not proof of word-level alignment.

Product interpretation — INFERENCE. Browser events, forced-alignment timestamps, and timed-text cues therefore provide different evidence with different granularity and uncertainty. The presence of a timestamp is not sufficient evidence for a word highlight.

MAKNOW policy — DESIGN. Confidence/drift thresholds, the state machine, and fallback selection are local project authority. External specifications do not authorize us to convert an estimate into universal ground truth.

Worked examples: baseline, invalid shortcut, and transfer

Baseline. Token alignment has forced-aligner confidence 0.94, drift 52 ms, and an exact text-version match, so its word evidence is defensible. A punctuation-adjacent token has confidence 0.91 but drift 180 ms, so that token falls back to its cue. The utterance is therefore a mixed-granularity decision rather than “every word is precise.”

Invalid shortcut. Browser word-boundary support disappears and a team interprets charIndex=0 as “the first word starts at time zero.” That inference is invalid because zero can represent an unsupported value. If no independent word alignment is available, the contract must degrade to sentence/cue fallback.

Transfer. TTS is replaced by human narration and one phrase is paraphrased. Because audio_asset_ref changes, absolute offsets from the old TTS asset do not remain valid automatically. 'Preserve valid timings elsewhere' means timings that have been revalidated/recomputed against the new human-audio asset, or valid cues belonging to that new asset. The changed phrase must be remapped; if confidence is weak, only that segment degrades to cue fallback.

Failure modes that must fail closed

Do not use word timing when text_version does not match the audio. Do not claim precision when timing_source is unknown. Do not compare timestamps from unknown timebases as if they were equivalent. Downgrade word-level state when confidence is below threshold or drift is over the limit. If cue fallback is required but no cue_id exists, do not force the release path. If review_owner is missing, the result is not review-ready. During transfer, never reuse the baseline decision as an oracle.

States such as UNKNOWN, DRIFTED, FALLBACK_REQUIRED, and HOLD are valid results; they prevent fabricated precision.

Alternatives and trade-offs: browser events, forced alignment, and timed-text cues

Browser TTS events are low-friction for feedback during synthesis in the same engine, but support and granularity vary. Forced alignment can produce word-level estimates for recorded or generated audio, but remains dependent on text, pronunciation, model behavior, and matching. Timed-text cues usually offer lower granularity but can provide a stable, auditable fallback.

The best architecture is not always the highest granularity. If maintaining word timings is fragile across engines or edits, sentence/cue highlighting can be more reliable. The decision should expose provenance and state rather than compressing everything into an unexplained score.

Experienced bridge: treat synchronization as a state machine

In a production design, each token can have its own state: WORD_ELIGIBLE, DRIFTED, LOW_CONFIDENCE, CUE_FALLBACK, TEXT_VERSION_MISMATCH, or REVIEW_REQUIRED. A voice change, punctuation edit, text edit, or regenerated audio may invalidate only dependent segments rather than the entire utterance. This enables selective re-alignment and partial regeneration.

Automation can check version equality, numeric thresholds, cue presence, and deterministic drift. It should not silently invent the semantic judgment that a paraphrase still matches instructional intent or that final review is approved.

Guided and independent practice

1. Open the baseline fixture and identify each timing_source and its text_version. 2. Locate the drifted token and explain why word sync is not defensible without changing the threshold. 3. Run the no-browser-boundary mutation and derive the fallback from evidence. 4. Inspect the recorded-narration transfer; remap the paraphrased segment and retain outside timings only after they are validated against the new human-audio asset. Do not reuse absolute offsets from the old TTS asset.

For independent practice, remove the final decision label. Reconstruct the contract and state from visible fields rather than from the fixture name.

Verification evidence: what must be reproducible?

A reviewer must be able to reproduce this chain without hidden reasoning: does text version match? Do token offsets belong to that text? What is the timing source? Has timebase been normalized? Are confidence and drift within the local word-level rules? If word sync fails, is sentence/cue evidence available? Is a review owner named?

The offline lab probes missing browser boundaries, 180 ms drift, text-version mismatch, unknown timebase, low confidence, missing fallback cues, missing owner, and baseline-oracle reuse in transfer.

Assessment and transfer: human narration with one paraphrased phrase

Assessment has four parts: retrieve the lesson boundary; build the baseline Contract; diagnose removal of browser boundary support; then replace TTS with human narration containing one paraphrase. The transfer cannot reuse the baseline label. The changed segment needs a new token mapping, confidence, and fallback; unchanged segments retain only timings revalidated against the new audio_asset_ref. An absolute offset from the old TTS file is not evidence for the new human-audio file merely because the text is unchanged.

A strong answer does not have to end in WORD_SYNC_REVIEW_REQUIRED. If evidence supports only mixed granularity, that bounded result is more correct.

References, freshness, and authority boundaries

Web Speech API and W3C Timed Text primary sources were rechecked on 13 September 2026. The current Web Speech document still states that boundary events depend on engine support, charIndex is approximate, and unsupported timing-related fields may be returned as 0. W3C Timed Text still represents time-associated text for synchronized-media workflows. Recheck these claims by 2027-01-01 or earlier if relevant semantics materially change.

The 0.85 confidence threshold, 120 ms drift threshold, states, and fallback rules are local MAKNOW DESIGN authority, not W3C requirements.