Robots.txt, Noindex, and Canonicals: Answering the Crawl Control Questions Everyone Asks

I've been doing technical SEO long enough to know that three things confuse almost everyone, at every level: robots.txt, the noindex directive, and canonical tags. Not because they're genuinely complicated — individually, each one does something fairly simple. The confusion comes from the gaps between them, the edge cases, and the fact that a lot of SEO content gets these subtly wrong in ways that compound over time.

So let me just answer the questions. The ones I see in Slack channels, Twitter threads, and client calls. No filler.

The Robots.txt Questions

What does robots.txt actually do — and what doesn't it do?

Robots.txt tells crawlers which URLs they're allowed to visit. That's it. It doesn't tell them anything about indexing. A Googlebot that respects your Disallow: /private/ rule won't crawl those pages — but if someone links to a page inside /private/, Google can still index it based on that link alone, even without ever crawling the content. You'll get a title-less snippet and a URL in the index. That surprises people.

The key mental model: robots.txt controls access, not indexing.

Does Googlebot always obey robots.txt?

For standard crawling, yes. Google has said it respects the Robots Exclusion Protocol. But there are nuances. Google treats robots.txt as a strong suggestion, not a hard firewall. And other bots — scrapers, AI training crawlers, random spiders — may not respect it at all. If you need pages truly inaccessible, robots.txt is the wrong tool. Use authentication.

Can I block specific bots but allow Google?

Yes, using the User-agent field:

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

This blocks OpenAI's GPTBot from everything while allowing all others. You can name specific crawlers — AhrefsBot, SemrushBot, Bytespider — and set rules per agent. Just make sure the order doesn't matter (it doesn't in robots.txt; the most specific user-agent match wins, not the first listed).

My robots.txt has a mistake. How fast does Google pick it up when I fix it?

Google caches robots.txt, typically for up to 24 hours, sometimes longer. The old cached version may affect crawls during that window. You can request a faster re-fetch via Google Search Console's URL Inspection tool — submit the robots.txt URL directly and request indexing. It won't be instant but it's faster than waiting for the natural refresh cycle.

The Noindex Questions

What's the difference between blocking a page in robots.txt and adding noindex?

This is the big one. Here's a table in your head:

Robots.txt Disallow: Google can't crawl it → can't read the noindex on it → page might still be indexed (from links)
Noindex meta tag: Google crawls it, reads the tag, removes it from index

If you block a page in robots.txt and add a noindex tag, Google can't read the tag because you've blocked the crawl. The page may stay indexed. This is the most common mistake I see in site audits — devs block staging environments with robots.txt AND add noindex, thinking it's double protection. It can actually be weaker than just one or the other, depending on how many links point to those URLs.

The rule: For pages you want deindexed, use noindex without a robots.txt block on that URL. Let Google in so it can read the instruction.

How long does noindex take to work?

It depends entirely on when Googlebot next crawls the page. For a high-authority page that Google crawls daily, deindexing can happen within days. For a deep page on a low-crawl-budget site, it could take weeks or longer. There's no magic "submit noindex" button. If speed matters, use the URL Removal Tool in Search Console for a temporary 6-month suppression — then clean up properly with noindex for permanent removal.

Can I use noindex in an HTTP header instead of a meta tag?

Yes, and this is actually the right approach for non-HTML resources — PDFs, for instance. You can't add a <meta> tag to a PDF. But you can send:

X-Robots-Tag: noindex

in the HTTP response header. Google reads it. This works for any file type. For HTML pages, the meta tag approach is more common and easier to manage, but the header method is equally valid.

I noindexed a page by accident and it dropped from rankings. Can I recover it?

Yes. Remove the noindex tag, wait for Google to recrawl, and the page should return to the index. Your rankings may or may not fully recover depending on how long the page was deindexed and whether competitors moved in during the gap. For critical pages, check your implementation before pushing to production — a simple site audit or a crawl with Screaming Frog will catch stray noindex tags fast.

The Canonical Questions

What does a canonical tag actually tell Google?

A canonical tag (<link rel="canonical" href="...">) is a strong hint to Google about which version of a page you consider the "real" one. It's your way of saying: "if you're looking at this URL, the one that matters is over here." Google should consolidate ranking signals — links, engagement, authority — toward the canonical URL.

The word "hint" is important. Google can and does ignore canonical tags when it thinks you've made a mistake or when signals contradict each other. It's not a directive the way a noindex tag is.

When should I actually use a canonical tag?

The most common scenarios:

Duplicate content from URL parameters: /products?color=red and /products?color=blue showing the same product should both canonical to /products (or the main variant)
HTTP vs HTTPS, www vs non-www: Make sure every page canonicals to the preferred protocol and subdomain version
Paginated content: Debated, but many SEOs canonical page 2, 3, etc. to page 1 — though this can hurt if those pages have unique content worth indexing
Syndicated content: If you publish your content on another site (Medium, a partner publication), ask them to canonical back to your original URL

Can I use a canonical to redirect ranking power to a completely different page?

Technically you can set a canonical from any URL to any other URL, even on a different domain. Cross-domain canonicals exist and Google supports them. But Google is skeptical of canonicals that point to very different pages — if the content doesn't match closely, it may simply ignore the canonical. This is not a link-juice manipulation tool. Use it for genuine duplicate or near-duplicate situations.

What's the difference between a canonical and a 301 redirect?

A 301 redirect actually moves users and bots to the new URL — the old URL stops existing in any meaningful way. A canonical keeps the old URL technically accessible while telling Google which one to prefer in the index.

Use a 301 when you've permanently moved a page and want users to land on the new URL. Use a canonical when you want to keep both URLs accessible (maybe for UX reasons, session IDs, tracking parameters) but want Google to index only one.

If you can 301, do that. It's a stronger signal. Canonicals are the tool when redirects aren't practical.

I have a canonical on my page but Google is indexing a different URL. Why?

Google disagreed with your canonical. Common reasons:

The canonical URL has thin or very different content versus the page carrying the tag
More external links point to the non-canonical version, so Google trusts that one more
The canonical URL returns a non-200 status code
Internal links overwhelmingly point to the non-canonical version — internal linking is a powerful signal

Fix the inconsistency Google sees: make sure internal links point to the canonical URL, ensure the canonical version has strong content, and verify it's returning 200. Don't fight Google with just a tag when the rest of your signals say something different.

The "All Three at Once" Questions

I have a page blocked in robots.txt, with noindex, and a canonical pointing elsewhere. What happens?

Honest answer: chaos, and Google's behavior here can be unpredictable. The robots.txt block means Google can't read either the noindex or the canonical tag. The canonical elsewhere gives Google no signal it can process. You've essentially told Google "stay out" through one channel and "look here instead" through a channel it can't access.

Simplify. Pick the right tool for the job. If you want Google to stop indexing a page, noindex it with no robots.txt block. If you want to consolidate signals, use a canonical and make sure Google can crawl both pages. Don't layer all three on the same URL — you'll just create conflicting signals that Google has to guess at.

What's the fastest way to audit these across a site?

Screaming Frog is still the fastest for mid-size sites — run a crawl and filter for pages with noindex, check for canonicals pointing to blocked URLs, and look for self-referential canonicals vs. non-self ones. For larger sites, Sitebulb gives better visualizations of crawl depth and canonical chains. Google Search Console's Index Coverage report is your confirmation layer: it tells you what Google actually indexed and flagged, not just what your HTML says.

Check these quarterly minimum. Settings that were right six months ago may be wrong now if your site structure changed.

These three tools — robots.txt, noindex, canonical — cover most of what you need to control how search engines interact with your content. The problems come from using them interchangeably or stacking them without understanding that each operates at a different layer. Get the mental model right, and most of the weird indexing behavior you're seeing on your site starts to make sense.