Push to production and manage indexing

Publish in waves, give each wave its own sitemap, and read the indexing of the first wave as a verdict on your template. The launch to avoid is the one that pushes every URL on the same day and learns afterwards that something was wrong with all of them.

Say you are launching a coworking directory with 2,000 city pages, built on the data model from the previous chapter.

Run the checks that a template makes cheap

A template error is multiplied by the number of pages, which also means a fix is multiplied the same way. Run these against the rendered HTML of a sample from each template, and script as much as you can:

• title and meta description are unique, and written for a click as much as for the keyword
• exactly one H1, and a heading structure that matches the page
• absolute canonical pointing at the page itself
• no empty block, no raw {{city}}, no undefined or null in the visible text
• JSON-LD parses, matches what is visible on the page, and passes the Schema Markup Validator
• the page returns 200 and carries no leftover noindex
• it renders correctly on mobile, with the main content present in the HTML Google gets

For speed, audit one URL per template with Lighthouse or PageSpeed Insights and fix it once. Those are lab numbers. The field numbers in the Core Web Vitals report (LCP, INP, CLS) only appear once real Chrome users have visited, so new low-traffic pages may have none for a while.

Then answer the question no checklist asks: would you send a friend to this page for the city with the thinnest data? If the answer is no, go back to your threshold.

Test in pre-production, and protect it properly

A staging environment needs HTTP authentication. A Disallow in robots.txt stops crawling, but a blocked URL can still end up indexed if it gets linked. The opposite trap is worse: staging’s noindex or blocking rule travels to production with the release. Add that check to your deployment script.

Deployment itself should be boring and repeatable. On Webflow, a script or Make scenario pushes items through the CMS API and publishes them. On Astro, the build reads the rows marked ready and your CI deploys the result. In both cases the database decides what goes out, so rolling a page back means changing a status.

Release in waves you can learn from

Google publishes no safe number of pages per day and I won’t make one up. I stage releases because a small wave tells me whether the template gets indexed while a mistake is still cheap. It also keeps you honest with the scaled content abuse policy, which targets thousands of pages nobody needs, whatever pace they were published at.

Wave	Pages	Which ones	Go on when
1	100	Cities with the richest data	Most are indexed and earning impressions
2	400	Mid-sized cities	Indexing holds at a similar level
3	1,500	Everything that passed your threshold	No new error pattern in wave 2

These sizes are an example. Keep the principle: pick the first wave from your best pages, so that a poor result can only mean the template or the site has a problem.

Give Google a sitemap you can actually read

One sitemap per wave or per template, grouped in an index file:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemaps/cities-wave-1.xml</loc>
    <lastmod>2026-11-02</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemaps/cities-wave-2.xml</loc>
    <lastmod>2026-11-16</lastmod>
  </sitemap>
</sitemapindex>

The split pays off in Search Console, where the Page indexing report can be filtered by sitemap. You get an indexing rate per wave without any tooling.

Generate these files from the page table so they only list published URLs. A single sitemap is capped at 50,000 URLs or 50 MB uncompressed. Google ignores priority and changefreq, and it only trusts lastmod if it stays accurate, so update it when the content really changes. Submit the index in Search Console and declare it in robots.txt. Google retired its old sitemap ping endpoint in 2023, so those are the two routes left.

User-agent: *
Disallow: /search
Disallow: /*?sort=

Sitemap: https://www.example.com/sitemaps/index.xml

Check your CDN settings while you are there. Some offer one-click blocking of AI bots, and if you want these pages to be used in AI answers, the crawlers behind them have to get through.

What actually speeds up indexing

Internal links from pages that are already indexed are the strongest lever you control. Link the new wave from your hubs, your regional pages and your homepage if that makes sense, as described in the chapter on internal linking. A page present in a sitemap and linked from nowhere sends a weak signal. External links help the hubs too, and they are worth earning for the section as a whole.

URL Inspection lets you request indexing, and I keep it for a handful of strategic pages. The quota is small, and asking twice does nothing.

The Indexing API gets sold as a shortcut. Google limits it to job posting pages and livestream videos, so unless that is what you publish, leave it alone. On the Bing side, IndexNow and Bing’s URL Submission API are legitimate ways to notify new or changed URLs. Google doesn’t take part in IndexNow, so treat it as extra coverage for other engines and the AI products that read from them.

Read the Page indexing report status by status

Status	What it usually means	What I do
Discovered - currently not indexed	Google knows the URL and hasn’t fetched it	Strengthen internal links, wait, check server capacity
Crawled - currently not indexed	Google read the page and passed	Treat it as a quality signal, enrich the page or reconsider it
Duplicate, Google chose different canonical than user	Pages look too alike	Compare the two pages, add differentiating data
Excluded by 'noindex' tag	Expected, or a leftover	Confirm it is deliberate
Soft 404	Page looks empty to Google	Add content, or return a real 404

Pulling a sample through the URL Inspection API gives you exact status per URL when the report’s examples aren’t enough. Server logs complete the picture by showing how often Googlebot visits each wave. Verify that hits are really Googlebot, since the user agent is easy to fake.

A word on crawl budget, a term that gets overused. Google says it mainly concerns very large sites or sites whose content changes quickly. At 2,000 pages, a page that doesn’t get indexed almost always has a quality or discovery problem.

Keep the index clean over time

Every few months, go through the pages that stay unindexed or get no impressions. An orphan page goes back into the linking structure. A thin page gets enriched, or merged into its regional page with a 301 redirect, or removed with a 404 or 410 if nothing replaces it.

Set those pages to pruned in the database so they leave the sitemap and the links automatically. Use noindex when a page should exist for users but not in search, knowing Google keeps crawling it to see the tag. Don’t block a URL in robots.txt while relying on its noindex, because a page Google can’t fetch is a page whose tag it can’t read.

Get these foundations right and the pages can keep paying off on their own. In January 2025 the Derrick pages were still bringing 88 clicks per day, two years later and with no additional SEO work.

Your next step: pick your first 100 URLs, generate their sitemap and submit it. Give it a few weeks, then open the Page indexing report filtered on that sitemap and start tracking performance by wave.