Robots.txt generator
Build crawler rules and sitemap directives locally, with explicit controls for common search and AI crawlers.
What does robots.txt do?
robots.txt gives crawlers rules about which paths they may request. This generator supports common search and AI crawler names, repeatable Allow/Disallow rules, optional Crawl-delay values, and global Sitemap directives.
Blocking a crawler in robots.txt is a request, not an access-control mechanism. Do not use it as a security boundary for private content.
Why is Crawl-delay documented carefully?
Some crawlers respect Crawl-delay, but Google ignores the directive entirely. For Google, use its Search Console crawl-rate controls where available.
Frequently asked questions
Does Google support Crawl-delay?
No. Google ignores Crawl-delay in robots.txt and provides crawl-rate controls through Search Console.
Are Sitemap directives tied to a crawler block?
No. Sitemap directives are global, so this generator places them at the end of the file.
Can I control AI crawlers?
Yes. The user-agent selector includes GPTBot, CCBot, ClaudeBot, and other common crawler names.