🤖 Robots.txt Generator & Tester

Why robots.txt Still Matters in 2025 (And How to Get It Right)

Every web crawler that lands on your domain looks for one file before doing anything else: robots.txt, sitting quietly at the root of your site. It is, in a sense, the bouncer at the door of your content — you decide who gets in and who gets turned away. And yet, a surprising number of sites either have no robots.txt, a broken one, or one that accidentally blocks pages they desperately want indexed. This guide goes deep on the mechanics, the edge cases, and the rules that trip up even experienced SEOs.

The Specification Google Actually Follows

The original 1994 Robots Exclusion Protocol was informal and inconsistent across crawlers. Google eventually published and pushed the RFC 9309 standard, which became the authoritative spec in 2022. The key distinctions matter in practice:

• Case sensitivity on paths: /Admin/ and /admin/ are treated as different paths by most crawlers. Always match the exact case your server uses.
• Most specific match wins: If you have Disallow: / and Allow: /blog/, a request to /blog/post-1 checks both. The /blog/ rule is 6 characters and the / rule is 1 character, so the longer match wins — the page is allowed.
• Tie-breaking favors Allow: If two rules of equal specificity match, Allow takes precedence over Disallow. This is Google's implementation and it has practical importance when you're mixing wildcard patterns.
• Wildcards are limited: Only * (match anything) and $ (end of URL) are supported. There is no support for ?, character classes, or negation within the path itself.

Other crawlers — Bing, Yandex, older scrapers — may implement the spec differently, which is why testing against multiple user-agents matters.

Disallow vs. Noindex: A Critical Distinction

One of the most common and costly mistakes in technical SEO is conflating Disallow in robots.txt with the noindex meta tag. They do completely different things:

Disallow tells the crawler not to fetch the page at all. The crawler respects this and never visits the URL. But here is the trap: Google can still index a Disallowed URL if it discovers the URL through external links. The page will appear in search results with no title or description — just a bare URL — because Google never crawled it to get that information. This is not hypothetical; it happens regularly on sites that try to hide login pages or admin panels through robots.txt alone.

Noindex requires the crawler to actually fetch the page, read the meta robots tag, and then remove it from the index. If you Disallow a page and put noindex on it, the noindex will never be seen. The page stays disallowed but may still surface in results as a bare URL.

The correct approach: use Disallow to block crawl budget consumption on genuinely private or duplicate content. Use noindex for pages you want crawled (for signal passing) but not indexed. For truly sensitive pages, use server-level access controls — robots.txt is a courtesy protocol, not a security mechanism.

Crawl Budget and the Pages That Waste It

Crawl budget is the number of URLs Googlebot will crawl on your site within a given time window. For large sites — e-commerce catalogs with faceted navigation, news sites with tag archives, CMS installations with duplicate parameter-based URLs — crawl budget is a real constraint. Wasting it on low-value pages means important pages get crawled less frequently or not at all.

Common crawl budget wasters worth Disallowing:

• Faceted navigation URLs: /products/?color=red&size=M&sort=price — these can generate millions of unique URLs with near-identical content.
• Search result pages: /search?q=...
• Session ID parameters: /page?sessionid=abc123
• Printer-friendly versions: /print/article-name
• Staging or test directories left exposed on production
• Duplicate content under multiple paths (common in Magento, Shopify, and custom CMSs)

For parameter-based URLs, the pattern Disallow: /*? blocks all URLs with query strings — powerful but blunt. A better approach is targeting specific parameters: Disallow: /*?sort=. Test these wildcard patterns carefully; a misplaced * can silently block content you need indexed.

The Crawl-Delay Directive and Why It's Largely Ignored

Crawl-delay tells crawlers to wait N seconds between requests. The practical reality: Googlebot ignores this directive entirely. You control Googlebot's crawl rate through Google Search Console, not robots.txt. Bing and some other crawlers do honor Crawl-delay, so it's not useless — but don't rely on it to protect your server from Google's crawler. It will not work.

Sitemaps in robots.txt: The Right Way

The Sitemap: directive is technically an extension to the protocol, not part of the original spec, but all major search engines support it. Declaring your sitemap in robots.txt ensures crawlers can discover it even if you haven't submitted it through Search Console or Bing Webmaster Tools. Multiple Sitemap directives are allowed and useful for large sites with separate sitemaps for different content types (news, images, video, products).

Critical detail: the Sitemap directive belongs outside any User-agent block. Many tools incorrectly place it inside a block, associating it only with that bot. Place it at the file level or after all agent blocks for universal crawler discovery.

Blocking AI Training Bots in 2025

A significant use case that didn't exist five years ago is blocking AI training crawlers. OpenAI's GPTBot, Anthropic's Claude-Web (and anthropic-ai), Google's Google-Extended, Common Crawl's CCBot, and ByteDance's Bytespider are all distinct user-agents. Each requires its own User-agent block with a Disallow: / directive. You cannot block all of them with a single wildcard block if you still want standard search crawlers to index your site — the wildcard rule would block Googlebot too. The pattern is to block each AI bot individually and then have a final User-agent: * block that allows everything (or applies your standard rules).

Testing Before Deploying

Google Search Console has a robots.txt tester, but it only tests against Googlebot and requires you to already have the file live. Testing locally before deployment catches mistakes that could block your entire site from being indexed — one of the worst technical SEO disasters possible, and one that happens with surprising regularity when developers push a staging robots.txt to production.

The key things to verify: your most important pages (homepage, product pages, blog posts) return "allowed" against your rules. Your admin, private, and duplicate pages return "blocked." Your wildcard patterns match what you intend — test edge cases like URLs with query strings, URLs with trailing slashes vs. without, and any unusual path structures your CMS generates.

A complete, well-structured robots.txt is not glamorous SEO work. But a broken one can silently hemorrhage crawl budget, block key pages, or expose private URL patterns to anyone who reads the file. Getting it right is foundational.