Robots.txt Generator
Generate a robots.txt file with allow and disallow rules, crawl-delay and sitemap. Free, runs in your browser.
Frequently asked questions
- What is a robots.txt file?
- It is a small text file at your site root that tells crawlers which parts of your site they may or may not visit.
- Does Disallow hide pages from Google?
- No. Disallow prevents crawling, not indexing. To keep a page out of search results use a noindex meta tag instead.
- Is my input uploaded?
- No. The file is generated entirely in your browser.
- Where should the file be placed?
- At the root of your domain, for example https://example.com/robots.txt. Subdirectories are ignored by crawlers.
This tool runs entirely in your browser. No data is uploaded anywhere.