Article

A publisher's robots.txt is blocking what nobody audited

By Miguel Maio · Published 20 September 2026

At a news publisher, robots.txt and the CDN configuration are rarely audited in detail - and that's exactly where traffic, images and citations are lost without anyone noticing. The question "block or allow AI?" is the wrong one for a publisher. The right one is: which resource am I blocking, for which bot, and with what effect on Google News, Discover and AI answers?

A publisher's robots.txt doesn't govern "the site"

It governs access by resource type and by bot, and that's what almost nobody audits. A content site has at least four different planes to control: the news HTML, the images, the video, and the channels that depend on their own crawlers or their own eligibility - Google News and Google Discover. A Disallow: / aimed at the wrong user-agent doesn't "protect the site": it erases an entire channel, and the effect only shows up weeks later, in the traffic.

What I audit, resource by resource

When I take on a publisher, the access audit isn't a top-to-bottom read of robots.txt - it's a check of each resource type against the bot that serves it and the channel that depends on it. This is what I run through:

1. HTML: separate search from training. Googlebot serves Search and Discover; Google-Extended controls the use of content for Gemini training without affecting ranking. Blocking Googlebot removes the site from Google; blocking only Google-Extended does not.
2. Images: Googlebot-Image has its own user-agent. Blocking it removes photos from Google Images and weakens rich cards and Discover, which is highly visual. At a publisher, images are traffic, not decoration.
3. Video: confirm the video and its metadata (thumbnails, transcripts) aren't behind a block inherited from an old CMS.
4. News and Discover: both depend on content being crawlable and indexable. There's no magic switch - there's the absence of accidental blocks and the technical eligibility (structured data, large images, editorial policies).
5. AI bots, one by one: GPTBot (OpenAI training) and OAI-SearchBot (real-time ChatGPT citations) are different bots with different consequences. The same goes for PerplexityBot. Blocking training isn't the same as blocking citation - and swapping the two is the classic mistake.
6. CDN and rate limits: a too-tight 429 or a WAF rule cuts off legitimate bots allowed in robots.txt. The file says "come in"; the CDN says "no". I always audit both.

The mistake I find most often

The pattern repeats: a publisher wanted to "block AI" and, without knowing, threw away the citations that were bringing it traffic. I've found a Disallow: / for OAI-SearchBot set to "not feed OpenAI" - except OAI-SearchBot trains nothing, it serves ChatGPT's real-time citations. The publisher kept being trained by other bots and stopped being cited where it wanted to be. In another case, a blanket block on Googlebot-Image, inherited from a migration, had the photos out of Google Images for months, with a direct loss in Discover. None of these show up in classic ranking - which is why nobody connects the dots until someone audits.

What Cloudflare changed on 15 September - and what it doesn't fix

Until now, blocking AI training without losing search was hard in practice: the controls were mostly all-or-nothing. On 15 September 2026, Cloudflare launched the "Disallow AI Training" option, which keeps the crawler for search and blocks the same crawler for training. It now distinguishes three bot behaviours - search, training and agent - and calls "accountable" those that meet four conditions, including the assurance that opting out of training doesn't affect classic search results. It's managed by a new "Bot Preference Sync" that replaces the old Managed Robots.txt.

What it does not fix, and it needs saying: only Apple, Google and Microsoft are designated "accountable", and they've committed to comply within a set timeframe - meaning not all of them apply it yet. Every other training crawler stays in the all-or-nothing bucket. And Cloudflare is an intermediary, not a web standard: this holds for those behind Cloudflare and as long as these vendors honour the commitment. It's a real step, not a universal fix.

How to decide, by resource and not in one block

The right decision at a publisher isn't a line in robots.txt - it's a matrix. For each resource (HTML, images, video) and each channel (Search, News, Discover, AI citations, AI training), the question is the same: does this bring me traffic or authority, or does it just feed a model with no return? Model training is, for most publishers, the only honest candidate for blocking - and now you can block it without sacrificing search, at least with three vendors. Everything else - search, images, News, Discover, real-time citation - is visibility you'd be giving up for nothing.

A publisher that handles this by reflex - "AI is stealing from us, block everything" - doesn't protect the content; it cuts the channels that still bring it readers. The cost doesn't show in ranking, it shows in traffic, three months later, when nobody remembers the line that was changed in robots.txt.

Sources

• Cloudflare - Have it both ways: stay discoverable while disallowing AI training
• Google Search Central - Overview of Google crawlers (user agents)
• OpenAI - GPTBot and OAI-SearchBot

Frequently asked questions