SEO & web
Robots.txt Tester
Test whether robots.txt rules allow or block a URL path for a selected crawler.
How to use this tool
Paste robots.txt rules, enter a path and identify the most specific matching Allow or Disallow directive.
Related tool: Robots.txt Generator →When to use the Robots.txt Tester
Use it to check whether a specific URL is allowed or blocked for a crawler before you deploy robots.txt changes.
Examples
- Confirm Googlebot can still reach
/blog/after adding new rules. - Check a parameter URL is blocked as intended.
Common mistakes
- Assuming a blocked page is removed from search; use noindex for that.
- Rule order and longest-match behaviour producing an unexpected result.