Crawl directives

Robots.txt Generator

Generate SEO-friendly robots.txt files with custom crawl and indexing directives, including individual AI bot controls.

Try it — sign in to run

Sitemap URL

Paths to block

Free forever plan · No credit card required · Takes 10 seconds

Generated file

User-agent: *

Disallow: /admin/

User-agent: GPTBot — Allow

Sitemap: https://yoursite.com/sitemap.xml

Custom disallow rules

Block any path from crawler access with simple rules.

Sitemap inclusion

Automatically references your sitemap so crawlers find it.

AI bot controls

Individually allow or block GPTBot, PerplexityBot, Claude-Web, and more.

One-click copy

Copy the finished robots.txt straight into your site root.

FAQ

What should a basic robots.txt file include?

A basic robots.txt should specify user-agent rules, any disallowed paths (like /admin/), and a reference to your sitemap.xml so crawlers can find it.

Is the robots.txt generator free?

Yes, sign in free to generate your file — no credit card required.

More tools

Control crawler access
across your whole site, in one place.

SEOVentra deploys robots.txt changes directly and validates them live against your actual site.