AI crawlers and bots: who crawls, renders and indexes

By Miguel Maio · Published 6 October 2026

Crawl, render, index

1. 1Crawl: fetch the HTML
2. 2Render: execute the JavaScript
3. 3Index/use: search, training or answer
4. 4Almost no AI bot renders

If it needs JS, AI can't see it

"Block the AI bots" is the advice I hear most, given without distinguishing what each bot actually does. But crawl, render and indexing are three different things, and each bot combines them its own way. Without that map you decide blind: you block who brings you citations, let in who only copies you, and assume AI sees your site the way Google does. It doesn't. This is the matrix of what each crawler and AI bot really does.

Crawl, render and index: three things that aren't the same

The confusion starts here, and it contaminates everything else:

1. Crawl: the bot fetches the URL's HTML file. It's just the code download, not seeing the page the way you do.
2. Render: executing the JavaScript to build the final page. It's the step that turns an empty shell into visible content. It needs a browser engine and is expensive.
3. Index or use: storing the content for something - a search index, training a model, or an answer generated in real time.

The trap: a bot can crawl without rendering. If your content only appears after JavaScript runs (client-side rendering), a bot that doesn't render sees a nearly empty page, even though it crawled successfully and got a 200. That's exactly what happens with almost every AI bot.

The criteria for this comparison

For each bot I look at four things, based on each vendor's official documentation and third-party log data:

1. Does it render JavaScript? Executes the JS or only reads the raw HTML.
2. What it's for: search index, model training, or real-time fetch on user request.
3. Does it respect robots.txt? And with what caveats.
4. Operator: who's behind it, because blocking one can affect several products.

An honest note up front: almost no vendor publicly confirms whether it renders. The "doesn't render" comes from measurements on real logs (mainly the Vercel/Merj study), not an official statement. Where it's inferred, I say so.

The matrix: what each bot does

The bots that matter most today, grouped by type. "Render JS" = executes JavaScript; "Purpose" = the main use.

Notice the pattern: the "Render JS" column is "No" across almost everything below the search engines. That's the fact that changes how you build the site.

Who renders JavaScript (and who doesn't)

On the search-engine side, Googlebot renders in an evergreen version of Chromium, in a two-wave process: first the HTML, then rendering in a queue that can take a while. Applebot also renders, and uses the content for Siri and Spotlight search and to train Apple Intelligence. Bingbot renders, but with limits: Microsoft recommends serving critical content as HTML.

On the AI side, the picture flips. The Vercel/Merj study, run on real traffic, found that none of the major AI crawlers execute JavaScript: GPTBot, ClaudeBot, PerplexityBot and the Meta and ByteDance bots. They do fetch JavaScript files (ChatGPT requests JS in about 11.5% of requests, Claude in about 23.8%), but they don't run them. The exception is Gemini, which leans on Google's infrastructure and so benefits from Googlebot's rendering. Common Crawl, which feeds a large share of training datasets, doesn't render either.

In plain terms: if your content depends on client-side JavaScript to exist, Google probably sees it, but ChatGPT, Claude and Perplexity see an empty page.

Search, training or real-time answer

"AI bot" isn't a single category. The purpose changes what's at stake when you block:

1. Search index: Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot. Blocking them takes you out of results and of search-grounded answers. It's almost never what you want.
2. Model training: GPTBot, ClaudeBot, Bytespider, CCBot (plus the Google-Extended and Applebot-Extended tokens). Blocking stops your content entering the model's weights, but doesn't take you out of real-time search answers.
3. Real-time fetch: ChatGPT-User, Claude-User, Perplexity-User. They visit on user request, to answer a specific question. Blocking them cuts your chance of being the cited source in that moment.

Deciding whether to block or allow each group has nuances, and I cover it in depth in block or allow AI and, for publishers, in the robots.txt nobody audited.

The fake crawlers: Google-Extended and Applebot-Extended

This is the mistake I see most in robots.txt out there. Google-Extended and Applebot-Extended are not crawlers and fetch nothing. They are control tokens: they tell Google and Apple whether they may use the content they already crawled (with Googlebot and Applebot) to train their models. Blocking Google-Extended doesn't remove you from Google Search or AI Overviews; it only says "don't use my content to train Gemini". Mistaking this for a bot leads to blocks that don't do what people think.

What this means for your site

Three direct consequences, in order of impact:

1. Client-side rendering kills you in AI. If content only appears after JavaScript, you're invisible to almost every AI bot. Serve content with SSR or SSG, rendered on the server, on the pages you want cited.
2. Put the facts in the HTML and in structured data. Price, address, dates, definitions and FAQ in text and in JSON-LD, not hidden behind JavaScript. It's what a non-rendering bot can actually read.
3. Decide access by purpose, not by reflex. Blocking everything labelled 'AI' usually removes you from where discovery is growing. Decide bot by bot, with the map on the table.

It's the technical layer of GEO, and the exact point where technical SEO meets AI visibility.

Limits and shelf life of this reading

Two honest warnings. First, this moves fast: vendors launch new bots and change behaviour without notice. This matrix holds for October 2026 and has to be revisited. Second, most rendering capabilities aren't officially confirmed; the "doesn't render" rests on log measurements (Vercel/Merj) and can change when a vendor decides to invest in rendering. Treat this as a working map, not truth carved in stone.

How to confirm on your own site

Don't trust tables, measure. Your server log tells you exactly which bots visit you, how often and what they request. That's where a comparison like this stops being theory. I show how to read AI bots in your logs with Screaming Frog for GEO, and how to work the Bing and ChatGPT side in Bing, Copilot and IndexNow. If you want me to run that diagnosis on your site, let's talk.

Sources

• Vercel / Merj - The rise of the AI crawler
• Google Search Central - JavaScript SEO basics
• OpenAI - Overview of OpenAI crawlers
• Anthropic - Does Anthropic crawl data from the web?
• Perplexity - Crawlers (PerplexityBot)
• Apple - About Applebot

Frequently asked questions