Google's Spam Guide Now Names Generated Pages. Robots.txt Is the Wrong Fix

On October 26 Google rewrote part of its guide to preventing user-generated spam. The changelog entry is dry: the prevent abuse guide now covers "blocking indexing of link-shared content and monitoring abuse in Search Console." The edit itself is more pointed than that summary.

The guide used to talk about comment forms and hosting platforms. Its opening paragraph now names "tools that let users publish shared pages, generated content, or web apps," and warns that spammers may create a large number of "pages or sites," where it previously said sites. Google also added a new section with a plain instruction: only index content meant for Search.

If you run a page generator, a client portal, a reporting tool that produces shareable URLs, or a programmatic SEO pipeline, this edit was written with your kind of system in mind.

What Google actually added

The new section reads, in full:

"If your service lets users create shareable pages, generated outputs, or interactive tools, consider whether that content is meant to be found through search engines or only accessed directly by people with a link."

"When pages are created for direct sharing, collaboration, or utility rather than public search discovery, apply the noindex rule (using a robots meta tag or X-Robots-Tag HTTP response header) or block crawling in robots.txt by default."

Two more additions sit under the monitoring advice. Google now tells platforms to verify their site or subdomain in Search Console so they get manual action and security alerts, and to watch the Page Indexing and Search performance reports for "unexpected spikes in indexed pages, impressions, or queries that don't match your platform's purpose." A new bullet asks for platform-level alerts on spikes in newly published public pages, or "a small number of users generating a large share of indexable URLs."

What Google did not do is announce an enforcement action. Marie Haynes asked publicly whether manual actions for AI tools and app-building platforms are coming, and Lily Ray named AI artifact pages and chat-built sites as the likely targets (Search Engine Roundtable has the diff and the reactions). That reading is plausible. It is still a reading, not a policy.

The trap in "noindex or robots.txt"

The new section offers two options as if they were interchangeable. They are not, and Google's own documentation says so.

Google's robots.txt introduction states that robots.txt "is not a mechanism for keeping a web page out of Google." A disallowed URL can still be indexed if other pages link to it. Google will not crawl the content, but the URL, and potentially the anchor text pointing at it, can still appear in results without a description.

The noindex documentation has the mirror-image problem. Google has to crawl a page to read its noindex rule. If robots.txt blocks the URL, "the crawler will never see the noindex rule."

So the common belt-and-suspenders setup, noindex on the page plus a robots.txt block on the same path, cancels itself out. The block hides the noindex. Search Engine Journal's write-up recalls that in July, Claude chat share pages turned up in Google even though the share path was disallowed in robots.txt. It also notes that Search Console has a status for exactly this outcome: "Indexed, though blocked by robots.txt."

The practical rule for share links, previews and generated outputs: use noindex, sent as a meta tag or an X-Robots-Tag header, and leave the path crawlable so Google can read it. Use robots.txt for crawl load, not for keeping pages out of the index.

Where agencies are exposed without realizing it

Most agencies will read "platforms" and assume this is about other people. Look at what a typical agency stack actually publishes.

• Staging and preview copies. A client site often has a staging host, a preview deployment per branch, and sometimes a second hostname the hosting provider assigns automatically. Cloudflare documents that every Pages preview deployment sends X-Robots-Tag: noindex by default (preview deployments). The project's production hostname on pages.dev is a different case: Cloudflare's headers documentation shows adding that rule yourself in a headers file. A full duplicate of a client site on a second hostname is the kind of page count spike Google now asks platforms to watch for.
• Shareable report links. Rank reports, audit exports and dashboards shared by URL are "generated outputs" in Google's wording. If they can be reached without a login and are linked anywhere public, they can be found.
• Thin programmatic variants. A location or service matrix where many pages differ only by a swapped noun is the pattern Google's spam policy calls doorway abuse: "substantially similar pages that are closer to search results than a clearly defined, browseable hierarchy."
• Unreviewed AI drafts. Google updated its generative AI content guidance on October 1. It says generating many pages without adding value may violate the scaled content abuse policy, and that review applies to titles, meta descriptions, structured data and alt text as well as body copy.

None of these rules are new. What changed is that Google now names generated content and shared pages together.

A one-hour audit for each client portfolio

Run this once per client, then add it to onboarding.

1. List every hostname that serves the site. Production, www and apex, staging, preview and platform-assigned domains. Fetch each one and read the response headers and the robots meta tag. Anything that is not production should return noindex.

2. Find conflicting rules. Search robots.txt for Disallow lines that cover pages you have also marked noindex. Remove the block on those paths so the noindex can be read.

3. Open the Page Indexing report. Look for "Indexed, though blocked by robots.txt" and for indexed URL counts that jumped without a matching publishing decision.

4. Sort generated pages into two lists. Pages a stranger searching would want to land on stay indexable. Pages that exist for one person with a link, or that only repeat a sibling page with one word changed, get noindex or get merged.

5. Set an alert. Google's suggestion translates directly to agency work: alert when the count of newly indexable URLs for a client moves well outside its usual weekly range.

Step 4 is the judgment call, and it is where a scoring step earns its place. In TrafficForge, QualityForge scores each generated page across 16 criteria, including doorway-page risk and thin-content risk, and gives a Publish, Improve or Discard recommendation. A page with high doorway risk cannot get a Publish recommendation whatever its total score. AuditForge's 27-point audit includes canonical and noindex checks for any URL you give it, which is a quick way to spot-check the live pages from step 1.

What to take from this

Google did not create a new penalty on October 26. It wrote down an expectation: if your system produces pages at scale, you decide which of them belong in Search, and you make that decision before Google has to.

For agencies, that is a reasonable standard anyway. Index the pages you would defend in a client meeting. Noindex everything that exists for convenience.