Meta robots, robots.txt and X-Robots-Tag: the signals that contradict each other

By Miguel Maio · Published 3 November 2026

Most pages that vanish from Google weren't penalised - they were deindexed by a signal the team itself put there, without noticing. And the worst part is that it throws no error on the day: it shows up as a drop weeks later that nobody ties to the cause. robots.txt, noindex, canonical, the sitemap and the status code are five different players, and nearly every indexing disaster is two of them, each correct on its own, giving opposite orders.

Crawl, index and serve are not the same thing

Before any directive, three distinct verbs: crawl (the bot reaches the URL), index (Google stores and shows it) and serve (the server answers with a status code). Each signal acts on one of these planes, and confusing them is the root of everything that follows. robots.txt is about crawling. noindex is about indexing. The status code is the serving response that holds up both.

robots.txt does not control indexing

robots.txt blocks crawling, not indexing - and that distinction costs traffic every day. If you Disallow a URL, Google won't crawl it; but if there are links to it, it can still index it, with no content, as the "Indexed, though blocked by robots.txt" notice. Worse: if you block a URL that also has noindex, Google never gets to see the noindex - and the page stays indexed against your will. Google is explicit: for the noindex to work, the page "must not be blocked by a robots.txt file", and declaring noindex inside robots.txt is not supported.

meta robots vs X-Robots-Tag: the same order, different places

meta robots and the X-Robots-Tag header give the same order (noindex, nofollow) - the difference is where they live. meta robots goes in the HTML and serves normal pages. X-Robots-Tag goes in the HTTP header and is the only way to apply noindex to non-HTML files: PDFs, images, videos. Google confirms they "have the same effect", so the choice is convenience. But there's a catch almost nobody tests: if the X-Robots-Tag rule is generated by a wrong condition on the server, it deindexes at scale - and silently.

The canonical is a hint, not an order

rel=canonical tells Google which is the good version of duplicated content - but it's a signal, not a command. Google uses dozens of signals and may pick a canonical different from the one you declared. So the signals have to agree: the rel=canonical on the page, the redirect (the strongest signal), the sitemap (the weakest) and the internal links. When some say A and others say B, Google decides for you. And you never mix noindex with canonical: one asks to remove, the other to consolidate - opposite orders on the same page.

The sitemap should only list what you want indexed

The sitemap is an invitation, not a guarantee - so it should hold only what you actually want in the index: URLs that return 200, are indexable and self-canonical. Listing a URL with noindex, one that redirects, a 4xx or a non-canonical one is sending Google a signal you contradict somewhere else. The sitemap is also a (weak) canonical signal: if it lists a URL different from the one rel=canonical points to, you add noise instead of clarity.

The URL triangle

There's a simple way not to get any of the above wrong - what I call the URL triangle (your "golden triangle" of consistency, not to be confused with the SERP eye-tracking zone). The three URLs that describe the same page must be the same URL, and that URL must return 200 OK, with no redirect and no error.

When the triangle closes (the three meet on one self-canonical 200), the signals reinforce each other and Google has nothing to interpret. When one vertex drifts - the sitemap lists a URL that redirects, the canonical points to a third, or one of them returns 4xx - you've given Google three addresses for the same person. This triangle catches the error I see most: the sitemap, the canonical and the served URL drifting apart, each right in its own way.

Status codes are the floor under everything

No signal above is worth anything if the status code underneath is wrong - it's the floor it all rests on. The essentials: 200 is "it exists" (but a dead page answering 200 becomes a soft 404, a ghost page competing with you); 301 and 308 are permanent and a strong signal, the target replaces the source; 302 and 307 are temporary and a weak signal, the target replaces nothing - using 302 for a permanent move means not consolidating equity; 404 and 410 are "it doesn't exist" (Google treats them the same and removes from the index); 429 is "you're asking too much" and slows the crawler; a prolonged 5xx makes Google back off and eventually deindex. The 429 is not an access control, however much it sometimes plays that role by accident.

The table of error combinations

Laid out by combination, because that's how it shows up in the field: not one wrong signal, two correct signals in conflict. From the most common to the most treacherous.

Whoever manages the redirects doesn't always see the implications

The part almost no guide admits: this rarely fails from technical ignorance, it fails from governance. Whoever touches the redirects - an in-house team, an external agency, the CMS vendor - often doesn't see the downstream implications, and nobody validates the whole. That's how a simple rule becomes a "blocker": the X-Robots-Tag generated by a wrong condition, the sitemap pointed at the staging environment, the domain typo nobody reviewed. The fix isn't only technical: it's deciding who owns these signals and who validates before it ships.

AI makes it worse: JS, rendering and the bots that don't wait

AI crawlers follow robots.txt and server-side redirects (301/302), but barely execute JavaScript. If your canonical or noindex are injected by JS, Google still sees them in a second rendering wave (delayed), but AI bots may simply not see them - and the version AI learns of you becomes the wrong one. Put these signals in the initial HTML or the header, never only in JS. It's where technical SEO meets GEO and migration.

Where this comes from

I've run into these errors across very different setups, and the pattern repeats. I've seen a site in one market "crash" because of redirect loops. I've seen a pair of sites start being deindexed by a wrong rule in the X-Robots-Tag generation - the same rule replicated, pulling pages out of the index in silence. I've seen sitemaps listing URLs that redirected to another, with the canonical pointing to a third. And I've seen the classic nobody owns up to: domain typos in the sitemap, www swapped for non-www, the trailing slash coming and going. None threw an "error" on the day it was made. And there's a common thread: whoever manages the redirects rarely sees all the downstream implications. That's where "blocker" level is born from a simple mistake.

What you lose when the signals contradict each other

None of these errors shouts on the day it's made. It shouts weeks later, as a drop you'll blame on a Google update that never happened. The difference between a site that holds and one that "crashes" isn't the algorithm - it's whether the signals you control are all saying the same thing. Start with the triangle: the served URL, the sitemap one and the canonical, the same, at 200. The rest of the table follows from there.

Sources

• Google Search Central - Block indexing (noindex)
• Google Search Central - HTTP status codes and soft 404s
• Google Search Central - Consolidate duplicate URLs (canonicalization)

Frequently asked questions