Free Robots.txt Generator
Build a valid robots.txt file in seconds. Choose a preset, set your allow/disallow rules, user-agent, crawl-delay, and sitemap URL, then copy the result. Runs entirely in your browser — nothing is uploaded.
Robots.txt Generator
Save the output as a file named robots.txt in your site's root
(e.g. https://example.com/robots.txt).
Generated robots.txt
Loading…
What is a robots.txt file?
A robots.txt file is a plain-text file placed in your website's root directory that tells search
engine crawlers which parts of your site they may or may not request. It uses simple
User-agent, Allow, and Disallow rules, and can point crawlers
to your XML sitemap. It controls crawling, not indexing — to keep a page out of search results, use
a noindex meta tag instead.
Frequently asked questions
Where do I put the robots.txt file?
In the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers
only look there — a robots.txt in a subfolder is ignored.
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still appear in search results without a snippet
if other pages link to it. To remove a page from the index, allow crawling and add a
noindex meta tag, or use the removals tool in Search Console.
What is crawl-delay and should I use it?
Crawl-delay asks crawlers to wait a set number of seconds between requests. Google ignores it (manage crawl rate in Search Console instead), but Bing and some other bots honor it. Only set it if a crawler is overloading your server.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap: line helps search engines discover all your URLs. You can list
more than one sitemap. It is one of the easiest wins for crawl coverage.
Related Free SEO Tools & Guides
- Meta tag generator — build SEO + Open Graph meta tags.
- SERP snippet preview — preview your Google result.
- Technical SEO checklist — robots, sitemaps, crawl, and indexing fixes.
- Free SEO audit tool — scan your site for issues.