Robots.txt Generator

Generate robots.txt files for your website. Control which bots crawl what, set crawl delay, and link your sitemap. Free, private.

⚙️ Settings

Default Rule for All Bots: Crawl Delay (seconds): Sitemap URL: Disallowed Paths (one per line): Block Specific Bots:

GPTBot CCBot Claude Google AI ByteSpider Ahrefs Semrush

📄 Output

📝 What goes in robots.txt?

User-agent: Which crawler the rule applies to (* = all)
Allow/Disallow: Paths the crawler can or cannot access
Crawl-delay: Seconds to wait between requests
Sitemap: URL of your XML sitemap

All processing in-browser · No data sent to any server

About Robots.txt Generator

Generate robots.txt files for your website. Control which pages search engines can crawl. Set crawl delay, sitemap URLs, and per-bot rules. All in your browser — no data sent anywhere.

Visual Builder

Add allow/disallow rules without writing syntax. See the robots.txt build up as you add directives.

Per-Bot Rules

Set different rules for Googlebot, Bingbot, and generic crawlers. Block AI scrapers selectively.

Presets Included

Start from WordPress, ecommerce, or custom presets. Tweak as needed for your specific site.

Validation Built-in

Checks syntax before you download. No broken robots.txt files — validated output every time.

📋 Common Use Cases

Block Crawlers

Disallow search engines from indexing sensitive directories like admin, login, or private areas.

Allow Access

Explicitly allow crawlers to access the pages and resources you want indexed.

Crawl Management

Set crawl delay to reduce server load from aggressive search engine bots.

Sitemap Discovery

Add your XML sitemap URL so search engines find and crawl your pages.

🪜 How to Use This Tool

Set the user agent
Choose the crawler the rules apply to, or use * for all bots.

Add allow/disallow rules
Specify which paths crawlers may or may not access.

Configure crawl delay and sitemap
Set a crawl delay and add your sitemap URL if needed.

Generate and download
Copy or download your robots.txt file and upload it to your site root.

Frequently Asked Questions

Are these SEO tools free?▼

Yes. All SEO tools are completely free. No signup needed.

Do the meta tags work for Google?▼

Yes. The meta tag generator creates standard Open Graph, Twitter Card, and HTML meta tags recognized by Google, Facebook, Twitter, and other platforms.

Is the schema markup valid?▼

Yes. The schema generator produces valid JSON-LD following Schema.org specifications for Article, FAQ, Product, LocalBusiness, and other types.