Screaming Frog for embeddings and vectors
TL;DR
- Screaming Frog generates embeddings two ways: Custom JavaScript (v20, 2024) calling an API, or native Semantic Similarity (v22, 2025).
- An embedding turns each page into a vector; proximity (cosine similarity) reveals semantically close pages.
- It powers content clusters, cannibalization detection, low-relevance content and internal linking at scale.
- It's cheap: mapping ~50,000 URLs with OpenAI costs under 5 USD.
An embedding represents a text as a vector of numbers that captures its meaning. Pages with close meaning sit close together in that vector space - which is what lets you, at scale, cluster content, detect cannibalization and suggest internal links by semantics rather than exact keywords. Screaming Frog brought this inside the crawl.
Two ways to generate embeddings in Screaming Frog
Don't conflate them - they arrived in different versions:
1. Custom JavaScript (v20.0, May 2024). Under Config > Custom > Custom JavaScript > Add from Library there are ready-made snippets that call the OpenAI API during the crawl (just paste your API key). It needs JavaScript Rendering on. You export a CSV with URL + Embeddings and process it (e.g. Python/Colab) to compute similarity.
2. Native Semantic Similarity (v22.0, June 2025). It does everything inside the tool, under Config > Content > Embeddings, using the providers in Config > API Access > AI: OpenAI, Gemini or Ollama (local). Screaming Frog recommends Gemini. This is the no-code path.
Models, dimensions and cost
| Model | Dimensions | Price |
|---|---|---|
| OpenAI text-embedding-3-small | 1536 (shortenable) | $0.02 / 1M tokens |
| OpenAI text-embedding-3-large | up to 3072 | $0.13 / 1M tokens |
| Gemini / Ollama (local) | varies | free (Ollama) |
For scale: mapping about 50,000 URLs with OpenAI cost, in the workflow documented on Screaming Frog's own blog, under 5 USD. Cheap for what it solves.
How similarity is measured
Proximity is computed with cosine similarity, a score from 0 to 1. In the native feature, by default, pages scoring above 0.95 are treated as semantically similar (adjustable). For "low relevance content", the tool computes the centroid (the average of all crawl vectors) and measures each page's distance from it - handy to find pages drifting off-topic.
Real use cases
| Use case | What it solves |
|---|---|
| Content clusters | Group pages by topic and design topical architecture |
| Cannibalization | Find near-duplicate pages competing for the same intent |
| Low-relevance content | Spot off-topic pages that dilute topical authority |
| Internal linking | Suggest links between semantically close pages, at scale |
| Redirect mapping | Match old and new URLs by semantics during a migration |
I tie this straight to Screaming Frog for GEO and technical SEO: clean topical authority is half the battle to get cited by AI.
Key data
| text-embedding-3-small — dimensions | 1536 |
|---|---|
| text-embedding-3-small — price | $0.02 / 1M tokens |
| text-embedding-3-large — dimensions | 3072 |
| text-embedding-3-large — price | $0.13 / 1M tokens |
| Custom JavaScript (Screaming Frog) | v20 · 2024 |
| Native Semantic Similarity (Screaming Frog) | v22 · 2025 |
The ~50,000 URLs (<$5) figure is from the official guest-post author (Gus Pelogia), not a Screaming Frog benchmark.
Sources: Screaming Frog — v22 · OpenAI — embeddings
Common mistakes and how to do it right
Embeddings are powerful and easy to misuse. What separates a useful map from a pile of numbers:
| Avoid | Do |
|---|---|
| ✗ Generating embeddings without cleaning content (menus, footers) | ✓ Extract only the main content - noise distorts the vectors |
| ✗ Comparing vectors from different models | ✓ Always use the same model across the whole crawl |
| ✗ Treating 0.95 as a hard law | ✓ Tune the threshold to your site and eyeball the pairs |
| ✗ Assuming high similarity always means cannibalization | ✓ Confirm intent: close pages can be legitimate |
Tools I use and recommend
- Screaming Frog SEO SpiderDesktop crawler for deep technical and on-page audits.
- ChatGPTAI assistant for research, analysis and quick audits.
- GeminiGoogle's AI assistant, grounded in Search.
- Google Search ConsoleOrganic performance, coverage and field Core Web Vitals.
Sources
- Screaming Frog - SEO Spider 20.0 (Custom JavaScript)
- Screaming Frog - SEO Spider 22.0 (Semantic Similarity)
- Screaming Frog - Identify Semantically Similar Pages
- Map Related Pages at Scale (Gus Pelogia)
- OpenAI - New embedding models
Frequently asked questions
Do I need to code?
No. The «Add from Library» snippets do the work, and the native feature (v22) is no-code - you just paste the API key. Technical SEO checklist →
OpenAI or Gemini?
In the native feature Screaming Frog recommends Gemini; in Custom JavaScript snippets, OpenAI is most common. For local and free, Ollama. Screaming Frog for GEO →