Free Robots.txt Generator
Build a valid robots.txt file in seconds. Choose a preset, set your allow/disallow rules, user-agent, crawl-delay, and sitemap URL, then copy the result. Runs entirely in your browser — nothing is uploaded.
Robots.txt Generator
Save the output as a file named robots.txt in your site's root
(e.g. https://example.com/robots.txt).
Generated robots.txt
Loading…
What is a robots.txt file?
A robots.txt file is a plain-text file placed in your website's root directory that tells search
engine crawlers which parts of your site they may or may not request. It uses simple
User-agent, Allow, and Disallow rules, and can point crawlers
to your XML sitemap. It controls crawling, not indexing — to keep a page out of search results, use
a noindex meta tag instead.
Frequently Asked Questions
In the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look there — a robots.txt in a subfolder is ignored.
No. Disallow stops crawling, but a blocked URL can still appear in search results without a snippet if other pages link to it. To remove a page from the index, allow crawling and add a noindex meta tag, or use the removals tool in Search Console.
Crawl-delay asks crawlers to wait a set number of seconds between requests. Google ignores it (manage crawl rate in Search Console instead), but Bing and some other bots honor it. Only set it if a crawler is overloading your server.
Yes. Adding a Sitemap line helps search engines discover all your URLs, and you can list more than one sitemap. It is one of the easiest wins for crawl coverage.
Related Free SEO Tools & Guides
- Meta tag generator — build SEO + Open Graph meta tags.
- SERP snippet preview — preview your Google result.
- Technical SEO checklist — robots, sitemaps, crawl, and indexing fixes.
- Free SEO audit tool — scan your site for issues.