Robots.txt Generator
Build a robots.txt file with user-agent rules, sitemap URL, and crawl delay.
Processed on your device — never uploaded
Rules
Generated robots.txt
User-agent: *
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed or disallowed from crawling.
Does robots.txt block pages from appearing in Google?
No, robots.txt only prevents crawling, not indexing. Google may still index a URL if other pages link to it. Use a "noindex" meta tag to prevent a page from appearing in search results.
Where should I put my robots.txt file?
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt). It won't work if placed in a subdirectory.
What does crawl delay do in robots.txt?
Crawl delay tells bots to wait a specified number of seconds between requests, which helps reduce server load. Note that Google ignores the crawl-delay directive.