Automate SEO audits at scale

On a programmatic site, audit templates and data, and let a schedule do the crawling. One bad template pushes the same error onto every page it generates, so an automated audit has one job: tell you quickly which template broke, on how many pages, and who has to fix it.

Here is the system I would set up, in the order I would set it up. To keep it concrete, say you run a job board with 40,000 URLs and three templates: /jobs/{role}-in-{city}, /companies/{company} and /salaries/{role}.

Why spot checks stop working past a few thousand pages

Manual checks are sampling. You look at the pages that get traffic and skip the long tail, which is the reason the site exists. They are also periodic, so a problem shipped on Monday waits until someone happens to look. And a crawl export read page by page shows 3,000 issues when you really have one bug in one template.

Crawl capacity is the other concern. Google’s documentation says crawl budget mostly matters for very large or fast-changing sites, and a programmatic launch can put you in that category overnight. Redirect chains, soft 404s and slow responses all spend crawling on URLs that bring you nothing.

Pick a crawler that can run unattended

Any tool on this list can work. Choose on scale, on your ability to export raw data, and on the possibility of scheduling. The tiers are rough, and I have left prices out because they change often.

Site size	Options	What to look at
Up to tens of thousands of URLs	Screaming Frog, Sitebulb, Ahrefs Site Audit	Screaming Frog’s paid licence adds scheduling and a command line, which makes it the most scriptable
Hundreds of thousands	Semrush Site Audit, Sitebulb Cloud	Crawl limits on your plan and cost as the site grows
Enterprise scale	Botify, Conductor Website Monitoring (formerly ContentKing), or a custom build	Log analysis and continuous monitoring

Size the choice for where the site is heading. Switching audit systems halfway through a rollout means losing your history.

Schedule crawls by template

Start with a weekly full crawl, then add a daily crawl for the sections you just deployed. A week is too long to discover that a new release made thousands of pages non-indexable. On the job board, I would run /jobs/ daily during rollout and everything else weekly.

With Screaming Frog on a Linux server, a cron job can run this:

screamingfrogseospider \
  --crawl https://www.example-jobs.com/ \
  --headless \
  --config /srv/audits/programmatic.seospiderconfig \
  --output-folder /srv/audits/crawls \
  --timestamped-output \
  --save-crawl \
  --export-tabs "Internal:All"

The settings that matter, whatever the tool: a smartphone user agent to match mobile-first indexing, JavaScript rendering if your pages depend on client-side code, a crawl rate your server can absorb, and robots.txt respected so you see what Googlebot sees. Keep every export, because trends and before/after comparisons need history.

Read the results per template

The most useful report groups problems by URL pattern, so “title too long on 3,000 pages” becomes one line and one ticket. Most tools offer segmentation. With a raw export, DuckDB can do it in one query:

SELECT
  regexp_extract("Address", 'https?://[^/]+/([^/]+)/', 1) AS template,
  count(*) AS pages,
  count(*) FILTER (WHERE "Status Code" <> 200) AS not_200,
  count(*) FILTER (WHERE "Indexability" <> 'Indexable') AS not_indexable,
  count(*) FILTER (WHERE "Title 1" IS NULL) AS no_title,
  count("Title 1") - count(DISTINCT "Title 1") AS duplicate_titles
FROM read_csv_auto('internal_all.csv')
WHERE "Content Type" LIKE 'text/html%'
GROUP BY template
ORDER BY not_indexable DESC;

Tools also give you a 0 to 100 health score and sort issues into errors, warnings and notices. That is a fine starting point for triage. I would still not steer by the score, because it treats a missing alt attribute and a template-wide noindex as items on the same scale.

Decide what deserves an alert

Rank issues by the pages affected, the harm to users, the effect on indexing, and the value of the template. Too many alerts and people stop reading them, so begin loose and tighten once the backlog is cleared. These are my starting values for the job board example, to be adjusted after a month of real noise:

Problem	Why it matters here	Starting rule
Indexable pages fall	A template shipped a noindex, or a robots.txt rule got too broad	Immediate alert if a template loses 10% between two crawls
Missing titles or canonicals	Usually a variable that stopped resolving	Immediate alert above 100 pages
Duplicate titles jump	A template change dropped the city or the role	Immediate alert above 100 new duplicates
Broken links in a template block	Repeated on every page generated	Immediate alert
Schema errors	Only where the markup is core to the template	Fix within two weeks
Redirect chains, 4xx, soft 404s rising	Crawling spent on useless URLs	Weekly digest
Core Web Vitals getting worse	Often a template or asset change	Weekly digest
Long titles, alt text, heading order	Small but cheap at template level	Backlog

Turn alerts into work

Send the immediate alerts to Slack or email, and have them create a ticket in Jira, Asana or Linear containing the URL pattern, the page count, a sample URL and the suggested fix. Everything else goes into a weekly digest of new and resolved issues. Stakeholders only need a monthly view by template.

Write a short runbook for each frequent issue, so that a ticket can be picked up without a meeting. For example:

Missing canonical on programmatic pages. Without one, search engines may treat similar pages as duplicates and split signals. Add a self-referencing tag to the template: <link rel="canonical" href="{{ page.url }}" />. Owner: frontend team. To verify: re-crawl the template and confirm each tag points to the right URL.

Block bad deploys before they ship

The cheapest bug is the one that never reaches production. This is where the audit plugs into your deployment process. Run a script in CI against staging, over a sample of URLs per template, and fail the build above a threshold. Besides the usual tags, it looks for unresolved variables, the error most specific to programmatic pages.

// check-templates.mjs: run in CI against staging, exit 1 blocks the deploy
import { readFile } from "node:fs/promises";

const BASE = process.env.STAGING_URL;
const MAX_FAILURE_RATE = 0.05;
// samples.json: { "jobs": ["/jobs/nurse-in-lille", ...], "salaries": [...] }
const samples = JSON.parse(await readFile("samples.json", "utf8"));

async function check(path) {
  const res = await fetch(BASE + path, { redirect: "manual" });
  const html = await res.text();
  const visible = html.replace(/<script[\s\S]*?<\/script>/gi, "");
  const errors = [];

  if (res.status !== 200) errors.push(`status ${res.status}`);
  if (!/<title>[^<]{10,}<\/title>/i.test(html)) errors.push("title");
  if (!/<link[^>]+rel="canonical"/i.test(html)) errors.push("canonical");
  if (/\{\{|\bundefined\b|\bNaN\b/.test(visible)) errors.push("unresolved variable");
  return errors;
}

let blocked = false;

for (const [template, paths] of Object.entries(samples)) {
  const failures = [];
  for (const path of paths) {
    const errors = await check(path);
    if (errors.length) failures.push(`${path}: ${errors.join(", ")}`);
  }
  console.log(`${template}: ${failures.length}/${paths.length} failing`);
  failures.slice(0, 5).forEach((line) => console.log(`  ${line}`));
  if (failures.length / paths.length > MAX_FAILURE_RATE) blocked = true;
}

process.exit(blocked ? 1 : 0);

After each release, run a focused crawl of the deployed sections within 24 hours and compare it with the previous baseline. The production chapter covers the rollout itself.

Add what a crawler can’t see

A crawler tells you what your server returns. Google’s reaction has to come from Search Console, and if you submit one sitemap per template, the Page indexing report can be filtered template by template. A concentration of “Crawled - currently not indexed” on one template often means Google sees little value in those pages. The Crawl stats report adds total requests, download size and average response time, and server logs show the same activity URL by URL. The tracking chapter covers how to read these over time.

Usefulness also needs its own checks, since Google’s March 2024 scaled content abuse policy targets mass pages with little value, and a page can pass every technical test while saying nothing. Measure how many data fields each page actually fills, and flag pages below your minimum for noindex or removal. Run near-duplicate detection per template. On the Deux.io build, a duplicate content checker and an AI content checker were part of the toolset. Then read a random sample by hand, because no metric replaces that.

AI-generated answers consume these pages too, so check that robots.txt reflects a deliberate choice about crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and look for them in your logs. Google-Extended is a separate control that doesn’t change your presence in Search.

Verify fixes and review the system

Closing a ticket proves nothing. After a fix, re-crawl the affected pages, compare before and after, and look for new issues on those same URLs. Then check a few pages by eye.

Once a month, ask where the system failed. Did an issue reach production that the checks should have caught? Which alerts were ignored? Which template keeps generating problems? The number to track is the time from detection to fix, by template. Alongside it, watch pages indexed and organic traffic per template to confirm the work is paying off.

Your next step: write your URL patterns down, set up one scheduled weekly crawl, and run the per-template query on the first export. Fix whatever tops the list, then add alerts and the CI check.