Build a valid robots.txt in minutes: allow or block Googlebot, GPTBot, ClaudeBot, and other AI crawlers, then validate and download — free.
Unlock advanced features for professional-grade bot management and earn XP
Set custom crawl delays for different bots to control server load
Define multiple sitemaps for better organization and faster indexing
Create customized access rules for different search engines and AI bots
Use wildcards and advanced patterns for precise crawl control
Specify preferred domain for search engines (useful for www vs non-www)
Control bot request rates to prevent server overload
A robots.txt file tells web crawlers which parts of your site they may request. It lives at the root of your domain and is the first thing well-behaved bots check before crawling.
Use the generator above to assemble a syntactically correct file — set a default access policy, fine-tune path rules, declare your sitemap, and validate before you publish.
Start by allowing all crawlers, blocking all crawlers, or switching to custom rules for fine-grained control.
List the directories to disallow or allow, and optionally set a crawl-delay to ease load from aggressive bots.
Point crawlers at your sitemap.xml so they can discover every important page on your site.
Check the syntax with one click, then download robots.txt and upload it to your domain's root directory.
The most permissive baseline — an empty Disallow allows everything.
User-agent: * Disallow:
AI training tokens are separate from search crawlers, so this stays fully indexable in Google and Bing.
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
Keep admin and checkout paths out of crawlers while exposing your sitemap.
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml