Technical SEO for a Content Site

1. First screen: useful content can still send contradictory signals

A publisher has a strong article at /articles/agent-evaluation, but the same body is also reachable through tracking and sorting variants. The article is linked from the home page, a sitemap lists several URLs, robots.txt contains crawl rules, and every HTML page emits metadata. Nothing looks obviously broken in a browser. Yet the system can still be internally contradictory: the sitemap may advertise a tracking URL while the HTML points to a different canonical; a crawl rule may block a path that the team mistakenly treats as “deindexed”; a duplicate page may declare itself canonical; or a title may exist in a template but be missing from the rendered response.

The goal of this lesson is not “make Google rank the site.” The bounded engineering goal is to build a consistent crawl-and-index contract that can be inspected locally. You will decide which URLs are intended representatives, which paths crawlers may access, which URLs appear in the sitemap, which metadata is present in the rendered HTML, and which internal links make important pages discoverable. Then you will prove that the pieces agree with each other.

Your evidence artifact is Technical SEO Crawl & Index Contract Record v1. It records observations, not predictions about future traffic. Passing the lab proves that the fixture has internally consistent technical signals; it does not prove that a search engine will crawl, index, select, or rank a URL in a particular way.

2. Route orientation: reduce explanation, never reduce evidence

Foundation route: start with the supplied URL inventory. For each URL, write the intended canonical representative, crawl access, sitemap membership, rendered title/description state, and a crawlable internal-link source. Treat each row as a contract that must be supported by files you can inspect.

Repair route: if you already know the vocabulary but have a failing site, isolate one contradiction. Do not “SEO the whole site.” Ask whether the defect is in crawl access, canonical consistency, sitemap coverage, rendered metadata, or link discoverability. Repair the smallest failing boundary and rerun the evidence checks.

Experienced route: skip the introductory definitions. Review the URL matrix as if it were a production change. Require evidence for every canonical choice, every sitemap entry, and every blocked path. Run the mutation suite and explain why each failed mutation is a contradiction rather than merely a style preference.

Every route preserves five critical dimensions: crawl_index_distinction, canonicalization, sitemap, rendered_metadata, and no_ranking_promise.

3. Mental model: five layers, one evidence chain

Technical SEO is easier to reason about when you stop treating it as a bag of tags. Use one evidence chain:

URL inventory → crawl access → rendered metadata → canonical relationship → sitemap/discovery support → observed audit evidence.

The URL inventory answers “what exists?” before you decide what should be discoverable. Crawl access answers whether the crawler is allowed to fetch a path under the local policy. It does not answer whether that URL will or will not appear in a search index. The distinction matters because robots.txt is a crawl-control mechanism, not a universal no-index command. Google’s current robots documentation explicitly keeps those concepts separate. [S-GSC-ROBOTS]

Rendered metadata is what the response actually contains. A source template is not sufficient evidence if the generated HTML differs. In this fixture you inspect the final <title>, description metadata, canonical link and ordinary crawlable links.

Canonicalization answers which URL is the preferred representative among duplicate or highly similar URLs. Google documents canonical preference as a signal, not an absolute instruction; redirects, sitemap inclusion and rel="canonical" can contribute to the decision. [S-GSC-CANONICAL]

The sitemap is a discovery and coverage aid. In this lesson, its engineering contract is simple: list the URLs you intend to be canonical representatives and avoid advertising known tracking variants as if they were independent canonical content.

4. Foundation layer: crawl access is not indexability

The most important conceptual separation in this lesson is between whether a crawler may fetch a URL and whether a search system may ultimately include a URL in an index. The local fixture can verify the first directly: parse robots.txt, match a path and report whether the path is allowed under the fixture rules. The fixture cannot verify the second because no live search engine is part of the acceptance run.

That is why the evidence record uses three labels: Observed, Inferred, and Not observed. “Observed: /articles/agent-evaluation is not disallowed by the fixture’s robots rules.” “Inferred: the page is intended to be crawlable under this policy.” “Not observed: Googlebot fetch, index state, or Search Console result.”

This wording prevents a common failure: blocking a URL in robots.txt and writing “therefore it is deindexed.” Current Google crawling guidance says a disallowed page’s content cannot be crawled, while the URL may still be indexed and shown without a snippet. [S-GSC-ROBOTS] The safe engineering conclusion is narrower: the path is blocked from crawling under that robots policy. Nothing in the offline fixture upgrades that fact into a guaranteed index outcome.

The same discipline applies to metadata. A page may emit an indexability-related directive, but the local lab only proves that the directive is present in the rendered HTML. It does not claim a downstream search engine outcome.

5. Deep explanation: canonical signals must agree

Assume a canonical article exists at /articles/agent-evaluation, while a campaign URL such as /articles/agent-evaluation?utm_source=newsletter exposes equivalent content. The fixture treats the clean article URL as the intended representative. The tracking variant should point its canonical annotation to the clean URL; the sitemap should list the clean URL rather than the tracking variant; internal editorial links should also prefer the clean URL.

Why require agreement? Because contradictory signals make the system harder to reason about. A sitemap that promotes a tracking URL while the page’s canonical points elsewhere forces the operator to explain which source of intent is authoritative. A duplicate page that canonicals to itself creates another contradiction. The lab does not pretend to reproduce Google’s canonical-selection algorithms; it checks whether your own declared signals are coherent.

Google’s current canonicalization documentation states that Google selects a representative URL and that signals can include redirects, sitemap inclusion and rel="canonical". It also states that Google may choose a different canonical from the site owner’s preference. [S-GSC-CANONICAL] Therefore the evidence record must use language such as “preferred canonical declared by the site” rather than “canonical forced on Google.”

Rendered HTML remains the inspection surface. For each important page, inspect one absolute canonical URL, a meaningful title, a description where your content policy requires one, and ordinary links that expose the canonical URL without requiring client-only state. The lesson does not claim that metadata quality compensates for weak content.

6. Worked example: one article and one tracking variant

The reference fixture contains a canonical article and a tracking variant. The canonical page renders this head fragment:

<title>Evaluating AI Agents: A Practical Checklist</title>
<meta name="description" content="A bounded checklist for evaluating agent behavior and evidence." />
<link rel="canonical" href="https://example.test/articles/agent-evaluation" />

The duplicate tracking fixture points to the same preferred canonical. The sitemap contains only the canonical URL:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.test/articles/agent-evaluation</loc></url>
</urlset>

The home page contains an ordinary <a href="/articles/agent-evaluation"> link. The baseline robots.txt does not disallow the article path. These four observations support a bounded statement: the local fixture has a crawlable canonical article, a duplicate that declares the same representative, sitemap membership aligned to that representative, and an HTML discovery path.

They do not support “the page is indexed,” “the page will rank,” or “Google must select this canonical.” Those outcomes are outside the offline lab.

7. Failure modes: contradictions must be observable

Mutation A — robots shortcut. Disallow the article path and then claim the page is guaranteed deindexed. The audit must reject the conclusion because crawl blocking and index outcome are distinct.

Mutation B — conflicting canonical. Make the duplicate tracking page canonicalize to itself while the canonical article still declares the clean URL. The audit should report canonical disagreement for equivalent fixture content.

Mutation C — misleading sitemap. Add the tracking URL to the sitemap while the page declares the clean canonical. The sitemap now advertises a URL outside the intended canonical set.

Mutation D — client-only discovery. Remove the ordinary HTML link to the article and leave the article reachable only through a synthetic client-only state marker. The local audit should report that the required crawlable link evidence disappeared. The lesson does not claim that JavaScript links are never discoverable; it only enforces this fixture’s explicit link-evidence contract.

Mutation E — ranking promise. Write “all checks pass, therefore this page will rank.” The record fails because technical consistency is not ranking evidence.

Each mutation changes a specific evidence dimension. If a mutation can be introduced without changing your conclusion, your evidence method is too weak.

8. Experienced bridge: review URL policy as configuration

In mature teams, technical SEO problems are often configuration drift. Content systems generate new URL shapes, analytics append parameters, migrations leave alternate hosts reachable, and sitemap jobs operate independently from page templates. The useful review unit is therefore not a single tag. It is a URL-policy diff.

When a new URL pattern appears, ask five questions. Is it intended to be crawlable? Is it intended to be a representative URL or a duplicate/variant? Which canonical does its rendered response declare? Should it appear in the sitemap? Where is the ordinary internal link that makes the representative discoverable?

The evidence matrix turns those questions into change-control data. A CMS change that introduces /archive?page=2 or a versioned documentation path should not be approved merely because it renders. It needs a declared place in the URL policy. This is especially valuable when multiple systems own robots rules, page metadata, sitemaps and navigation.

The fast path still requires one mutation and one changed-context transfer. Experience reduces tutorial time; it does not remove the need for observable proof.

9. Alternatives and tradeoffs

A redirect can consolidate users and crawlers onto a new URL when a URL has actually moved. A canonical annotation can express a preferred representative while keeping alternate URLs reachable. Those mechanisms are not interchangeable in every product situation. This lesson does not teach a universal migration strategy; it asks you to make the local representative relationship explicit and internally consistent.

Sitemap inclusion is useful as discovery/coverage support, but it should not become a second URL policy that contradicts page metadata. If your sitemap generator and page canonical generator use different normalization rules, fix the shared URL policy rather than patching individual entries.

Blocking crawl may reduce crawler access to a path, but it is the wrong mechanism to use as a guaranteed deindex statement. Likewise, a perfect metadata audit cannot prove content value or ranking. The bounded contract is intentionally modest: remove technical contradictions that you can inspect, then state what remains unobserved.

10. Guided and independent practice

Run node --test test/public-contract.test.mjs inside the learner fixture. Then run node src/audit-seo.mjs to regenerate the deterministic audit snapshot.

Complete the work in this order:

1. Read seo-plan.json and state the intended representative URL for each variant.
2. Inspect robots.txt and record crawl access separately from indexability inference.
3. Inspect the rendered HTML for title, description, canonical annotation and ordinary links.
4. Compare sitemap URLs with the intended canonical set.
5. Run the public checks and save the audit output in the evidence record.
6. Apply each mutation only after predicting which evidence row must fail.
7. Repair the fixture and rerun the checks.
8. Complete the documentation-site transfer independently.

The transfer case adds versioned documentation URLs such as /docs/v1/guide, /docs/v2/guide, and archive pages. Your task is not to copy the article solution. Decide which URLs represent distinct current content, which are historical but still valid, which should be discoverable, and how canonical/sitemap/link evidence stays coherent. Keep the conclusion free of ranking promises.

11. Verification and the evidence record

The record has six required evidence dimensions: implementation artifact, observable generated-output behavior, failure mutation result, repair result, changed-context transfer, and residual limitation.

A strong row looks like this: “Observed: tracking variant declares canonical https://example.test/articles/agent-evaluation; sitemap contains only the clean URL; home page links to the clean URL. Inferred: the site’s preferred representative is the clean URL. Not observed: Google-selected canonical or index state.”

A weak row says: “Canonical SEO is correct.” It does not show which file was inspected, what value was found, or what remains outside the lab.

Record the tested environment as well. Macro-A acceptance uses a dependency-free local Node model and static fixture files. It performs no outbound network request and does not submit URLs to a search engine. The official sources are rechecked separately for freshness, but the learner lab itself stays deterministic and offline.

Residual limitation is mandatory: passing this artifact does not prove live crawling frequency, index inclusion, canonical selection, rich-result eligibility, traffic, or ranking.

12. Assessment and transfer

Retrieval: explain the difference between crawl control, index outcome, canonical preference and sitemap discovery support without using ranking as the definition of success.

Application: make the article fixture internally consistent and complete Technical SEO Crawl & Index Contract Record v1.

Diagnosis: choose one mutation, predict the first failing evidence row, run the check, and explain why the failure matters.

Decision: defend one representative-URL choice against a plausible alternative. Your defense must mention user/content identity and evidence consistency, not “Google likes clean URLs” as a slogan.

Transfer: design the URL policy for versioned documentation and archive pages. Your result must identify canonical representatives, crawl-access intent, sitemap membership and discovery links. It must also state one residual uncertainty.

Completion requires evidence quality, scope discipline and transfer independence. A technically plausible answer that promises ranking or fabricates live index evidence does not pass.

13. References and freshness boundary

This lesson’s evolving external claims are bounded to three official Google sources from the frozen Bible. On 2026-09-13, the canonicalization documentation still describes canonicalization as selection of a representative URL and treats signals such as redirects, sitemap inclusion and rel="canonical" as preferences that do not force Google’s choice. [S-GSC-CANONICAL]

Google’s current robots documentation still defines robots.txt as crawler-access control and states that a disallowed URL may still be indexed without a snippet. [S-GSC-ROBOTS] The Search documentation updates page remains the freshness anchor for detecting relevant documentation changes; it is not evidence for a ranking claim. [S-GSC-UPDATES]

The frozen review date remains 2026-12-31. Recheck earlier if official guidance changes or if the deterministic fixture no longer represents the documented boundary. Do not expand the source scope silently. This lesson proves an inspectable technical-SEO consistency method, not search-engine behavior outside the evidence actually observed.