Screaming Frog for GEO
TL;DR
- Screaming Frog audits, at scale, the signals that make content citable by AI.
- Since v20/v21 it connects to OpenAI, Gemini, Anthropic and Ollama, with up to 100 prompts over crawl data.
- The Log File Analyser separates training bots (GPTBot, ClaudeBot) from real-time citation bots (OAI-SearchBot, PerplexityBot).
- Most AI bots don't execute JavaScript like Googlebot - checking rendered HTML is critical.
GEO isn't done page by page, by hand. Screaming Frog lets you audit, at scale, the signals that decide whether your content is citable by AI - and, with the Log File Analyser, see what AI bots actually do on your site. Five fronts.
1. Audit citability with Custom Extraction
Under Config > Custom > Extraction, with XPath or CSS, you pull from each page the signals LLMs love: a short definition right after the heading, FAQ blocks, tables, lists, dates and stats, and author markup. It's how you measure, across every URL at once, how "liftable" your content is.
2. LLM prompts at scale (no code)
Since v21, under Config > API Access > AI, you set up to 100 prompts against crawl text, HTML or an extraction, with OpenAI, Gemini, Anthropic or Ollama. Use it to classify intent, extract entities, summarize, or build an E-E-A-T-style scoring - Screaming Frog gives the mechanism; you define the criteria in the prompt (it's not an official "E-E-A-T" feature).
3. Structured data and entity, at scale
Screaming Frog validates Schema.org/JSON-LD natively. For GEO it's how you confirm, in bulk, that you have Author, Organization, sameAs and types like FAQPage/Article where you should - the entity signals that underpin AI citation.
4. JavaScript rendering: see what AI sees
In JavaScript Rendering mode you compare raw vs rendered HTML. This is critical because, unlike Googlebot, most AI bots don't execute JavaScript: JS-dependent content can be invisible to them.
AI bots: training vs real-time citation
| Type | Bots | What they do |
|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended | Collect content to train models |
| Real-time citation | OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User | Fetch pages to answer and cite now |
5. Log File Analyser: what AI actually does
The Log File Analyser ships presets for AI user-agents. You see which URLs each bot hits, the response codes (a 4XX/5XX to a citation bot is a lost citation), and you verify bots by IP to catch spoofing. The SEO Spider audits the site; the LFA audits the real hits.
I do this in the GEO service, alongside embeddings work for topical authority.
Key data
| Major AI bots active | 21 |
|---|---|
| Sites blocking GPTBot | 5,89% |
| ClaudeBot block growth (YoY) | +32,7% |
| Referral traffic from AI | 0,1% |
| Top sources not shared across AIs | 86% |
Ahrefs data (third-party authority), samples of ~140M and ~35K sites.
Sources: Ahrefs — AI bot block rates · Ahrefs — AI traffic research
Common mistakes and how to do it right
Screaming Frog gives you the mechanism; the criteria are yours. Mistakes I've seen cost dearly:
| Avoid | Do |
|---|---|
| ✗ Auditing GEO on raw HTML only | ✓ Turn on rendering - most AI bots don't run JS |
| ✗ Blocking GPTBot/PerplexityBot by accident | ✓ Decide consciously which AI bots you let in |
| ✗ Trusting LLM scoring without validating it | ✓ Calibrate the prompt and check a sample by hand |
| ✗ Ignoring the logs | ✓ Use the Log File Analyser to see what AI bots actually fetched |
Tools I use and recommend
- Screaming Frog SEO SpiderDesktop crawler for deep technical and on-page audits.
- Google Search ConsoleOrganic performance, coverage and field Core Web Vitals.
- ChatGPTAI assistant for research, analysis and quick audits.
- PerplexityAnswer engine with sources, useful for research and GEO.
- GeminiGoogle's AI assistant, grounded in Search.
Sources
- Screaming Frog - How To Crawl With AI Prompts
- Screaming Frog - Monitor AI Bots in the Log File Analyser
- Screaming Frog - How to Crawl with ChatGPT
- Ahrefs - AI bot block rates
- Ahrefs - AI traffic research
Frequently asked questions
Does Screaming Frog read llms.txt?
Yes - it's just a URL, and the crawler reads any file. But there's no dedicated official feature for llms.txt; treat it like any crawl. What is llms.txt →
Do I need the Log File Analyser?
To see real AI-bot behaviour, yes. The SEO Spider tells you how the site is; the Log File Analyser tells you what bots actually accessed. GEO service →