Tools / Robots.txt Generator
Free browser tool

Robots.txt Generator

Create a clean robots.txt file with allow, disallow and sitemap directives.

When to use the Robots.txt Generator

Use it to create a robots.txt that keeps crawlers out of admin or duplicate areas and points them to your sitemap.

Examples

  • Disallow /wp-admin/ but allow /wp-admin/admin-ajax.php.
  • Add a Sitemap: line with the full URL.

Common mistakes

  • Using robots.txt to hide private pages; it is public and not access control.
  • Blocking CSS and JavaScript that search engines need to render pages.