Free · runs in your browser
Robots.txt generator
Choose what crawlers may read, add your sitemap and copy a valid robots.txt.
- No sign-up
- Instant, works offline once loaded
- Nothing you type leaves your device
Admin areas, cart and checkout, internal search results.
robots.txt
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
Upload it as robots.txt at the root of your site, then check a path with the robots.txt tester.
What robots.txt can and cannot do
- It asks well-behaved crawlers not to read certain paths. It is not a lock.
- It does not remove pages from Google. To keep a page out of search results, use a noindex tag and let Google crawl the page.
- Never block CSS and JavaScript folders: Google needs them to render your pages.
Test before you upload
After uploading, check important paths with the robots.txt tester to be sure you did not block anything you want in search.
Check the live page too
Studio's free checkers read your real pages and show what search engines see, with evidence for every finding.