What robots.txt controls — and what it doesn't
robots.txt is a crawling instruction, not an indexing instruction. A crawler that obeys the file will not request the URLs you disallow, but Google can still list a blocked URL in search results if other pages link to it. It shows up with no snippet and often no title, because Google never fetched the page — which looks worse than the page you were trying to hide.
So a Disallow line is the wrong tool for hiding a page. To keep a URL out of the index, let crawlers fetch it and return a "noindex" robots meta tag or an X-Robots-Tag header, or put it behind authentication. If the page is blocked in robots.txt, Google never sees the noindex — the two mechanisms cancel each other out.
The file is per host, per scheme and per port. https://example.com/robots.txt says nothing about https://blog.example.com/ or http://example.com/. Every subdomain needs its own file at its own root path, and a file served from anywhere other than the root is ignored.
And it is public. Writing "Disallow: /internal-pricing-2027/" tells the whole internet that the path exists. Anything that must stay private needs authentication, not a line in a file anyone can read.
How a crawler picks the group that applies to it
A group starts with one or more User-agent lines and runs until the next User-agent line appears after a rule. Product tokens are compared case-insensitively, so "User-agent: GoogleBot" and "user-agent: googlebot" are the same thing. RFC 9309 limits a token to the characters a–z, A–Z, "_" and "-" — anything else is a sign of a typo or a full browser user-agent string pasted in by mistake.
A crawler uses one group, not several. Googlebot-News follows a googlebot-news group if the file has one, otherwise the googlebot group, otherwise the "*" group. Rules from "*" are not added on top of a specific group — this surprises people who put shared rules in "*" and crawler-specific extras in a named group, and then find the shared rules ignored.
If several groups name exactly the same token, RFC 9309 says their rules must be combined into one group. If no group matches and there is no "*" group, everything is allowed — and a group with no rules also allows everything. /robots.txt itself is always fetchable, whatever the file says.
Allow vs Disallow: which rule wins
Only one rule decides a URL, and it is not the first one in the file. RFC 9309 puts it plainly: "The most specific match found MUST be used. The most specific match is the match that has the most octets." Order is irrelevant, so moving a line up or down changes nothing.
When an allow rule and a disallow rule match the same number of octets, the allow wins. Google phrases the same rule as "the least restrictive rule". Google's own example: with both "Allow: /folder" and "Disallow: /folder" in a group, /folder/page is allowed.
This is why the classic pattern works: "Disallow: /" plus "Allow: /public/" lets /public/page.html through, because the allow pattern is 8 octets and the disallow pattern is 1. It is also why a rule can be dead code — a short "Disallow: /" that is overridden by a longer allow on every URL you care about does nothing at all. The tester above shows the winning line, its octet count, and the rule it beat, so you can see the comparison instead of guessing at it.
Wildcards: what * and $ really match
"*" matches any sequence of characters including none, and "$" anchors the end of the URL. They are extensions to the original 1994 standard, but Google, Bing and Yandex all support them. Every path pattern is anchored at the start and must begin with "/"; "#" starts a comment anywhere on a line.
- /fish matches /fish, /fish.html, /fishheads, /fish/salmon.htm and /fish.php?id=anything — a pattern is a prefix, not a whole path segment.
- /fish does not match /Fish.asp or /catfish. Paths are case-sensitive and matching starts at the beginning of the path.
- /fish/ matches /fish/ and /fish/salmon.htm, but not /fish or /fish.html. The trailing slash is part of the pattern.
- /*.php matches /index.php, /folder/file.php and /file.php?x=1. /*.php$ matches only the first two, because "$" forbids anything after ".php".
The query string is part of what gets compared, so "Disallow: /*?" blocks every URL that has any query string — including the paginated and filtered pages you probably wanted crawled. Percent-encoding is normalised before comparison, which means /검색 and /%EA%B2%80%EC%83%89 are the same path, and %7E and ~ are the same character.
Where Google's robots.txt Tester went
Google retired the robots.txt Tester in Search Console in November 2023 and replaced it with the robots.txt report. The report shows the files Google fetched for your top 20 hosts, when it fetched them, the parse warnings it found, and a button to request a re-crawl of the file.
What it does not do is the one thing people used the old tool for: type an arbitrary URL, pick a user agent, and get "allowed" or "blocked". Ryte's free robots.txt tester, the other well-known option, has been shut down too — its page now says only that the tool is no longer available.
That is why this page implements the matching rules directly from RFC 9309 and Google's documentation instead of sending you to Search Console. Paste the file, paste the URLs, pick the crawlers, and you get the verdict without waiting for Google to re-fetch anything — including for a staging site that is not public yet, or a file you have only drafted.
Blocking AI crawlers: what Google-Extended actually controls
AI-related tokens fall into three kinds, and blocking each has a different consequence. Training crawlers — GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent — collect pages for model training. Retrieval crawlers — OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot — fetch pages so an AI answer can cite them. User-triggered fetchers — ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher — load one page because a person pasted that link.
Google-Extended and Applebot-Extended are not crawlers at all. They are control tokens: they decide whether content Google or Apple has already crawled may be used for AI training and generative answers. Disallowing Google-Extended does not remove you from Google Search and does not change your ranking. This is the single most common misunderstanding about the file, and the reason this page shows a separate "control token" group instead of mixing them in with the bots.
Blocking is not free. A December 2025 study by researchers at Rutgers and Wharton found that publishers who blocked AI crawlers saw total traffic fall about 23.1%, while their citation rate in AI answers did not fall in step. Decide with both numbers in view. "Block training, allow AI search" is the middle position most publishers land on, and there is a preset for exactly that above.
Finally, robots.txt is a request, not a fence. Bytespider and some Perplexity crawlers have been reported ignoring it. The verdicts on this page describe what a compliant crawler does. Anything that truly must not be fetched needs authentication or a server-side block by user agent and IP.
The robots.txt mistakes that cost the most
- Shipping the staging file. "User-agent: *" with "Disallow: /" copied from staging to production is the most expensive robots.txt bug there is; it can drop a whole site out of search within days.
- Blocking /wp-content/, /assets/ or /static/. Google renders pages before judging them; if it cannot fetch your CSS and JavaScript, it evaluates layout and mobile-friendliness on a broken version of the page.
- A relative Sitemap line. "Sitemap: /sitemap.xml" is invalid; the value must be a full absolute URL such as https://example.com/sitemap.xml.
- Forgetting subdomains and schemes. blog.example.com, shop.example.com and the http:// version each read their own file.
- Putting rules before the first User-agent line. Those lines belong to no group and are silently ignored.
Two more only show up under load. A file over 500 KiB is truncated — Google parses at most 500 kibibytes and ignores everything after that — and a 5xx response on /robots.txt is treated by RFC 9309 as a full disallow, so a flaky server can pause crawling of the entire site. Google softens this by using its cached copy for up to 30 days, but the first 12 hours of crawling still stop.
Crawl-delay and the directives that don't do what you think
Crawl-delay was never part of the standard. Google ignores it and says so in its documentation; Bing, Yandex and Seznam do honour it. If Googlebot is crawling too fast, the answer is Search Console's crawl rate settings and fixing slow responses, not a directive Google does not read.
"Noindex:" inside robots.txt stopped working on 1 September 2019, when Google removed support for it. "Host:" and "Clean-param:" are Yandex extensions. Unknown fields are not errors — a crawler simply skips lines it does not understand, which is exactly why a typo like "Disalow:" fails in complete silence. The checker on this page flags every one of these.
"Sitemap:" is the one directive that is global rather than per group. It can sit anywhere in the file, applies to every crawler regardless of user agent, and must hold an absolute URL. You can list several — one per line — which is the normal way to point at a sitemap index plus a news sitemap.