Robots.txt tester

Studio follows the Robots Exclusion Protocol standard (RFC 9309), which Google and Bing implement:

• A crawler uses the group whose User-agent matches its name. If there is none, it uses the User-agent: * group. A specific group replaces the * group; they are not combined.
• Within the group, the most specific (longest) matching rule wins. If an Allow and a Disallow are equally long, Allow wins.
• * matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files.
• If robots.txt returns 404, everything may be crawled. If it returns a server error (5xx), crawlers treat the whole site as blocked.

robots.txt controls which URLs compliant crawlers fetch. A blocked URL can still appear in search results if other sites link to it, just without a description. To keep a page out of results, allow it to be crawled and add a noindex robots meta tag or X-Robots-Tag header. robots.txt is also public and is not a security measure: never use it to hide private pages, because anyone can read it.

• A staging site's robots.txt copied to production: Disallow: / under User-agent: * blocks the whole site.
• Blocking CSS or JavaScript: search engines need them to render pages correctly.
• Noindex lines in robots.txt: not supported by Google since 2019.
• Typos: a missing colon or misspelled directive makes crawlers skip the line silently.

• Some crawlers add their own behaviour on top of the standard (for example Google's handling of its specialised crawlers). The result shows what the standard specifies.
• Only the first 500 KiB of the file is read, which is the minimum the standard requires crawlers to read.