Crawl control
Robots.txt Tester
Robots.txt controls crawling, not guaranteed indexing. Test the exact URL and user agent so a broad rule does not quietly block useful pages, styles, scripts, or images.
Run the check
No signup. Be considerate when checking websites you do not own.
What this tool helps you see
- Fetch the site’s live robots.txt file
- Test an exact URL for a chosen user agent
- Keep the original file visible for human review
How to use the result
- Enter the page URL you want to test.
- Choose a crawler such as Googlebot.
- Review the verdict and the live rules together.
A checker can show evidence, but it cannot know your business intent. Confirm that every change still makes the page clearer and more useful for the person who searched.
Frequently asked questions
Does Disallow remove a page from Google?
Not reliably. It blocks crawling; a URL can still be known from links. Use an indexable response with noindex when removal is required.
Where should robots.txt live?
At the root of the hostname and protocol it controls, such as https://example.com/robots.txt.
Can robots.txt protect private information?
No. The file is public and rules are voluntary. Protect sensitive resources with authentication and authorization.