Robots.txt Generator

Generate robots.txt with User-agent, Allow, Disallow, Crawl-delay and Sitemap rules, including common presets and automatic syntax and URL validation.

Presets

Crawl rules

#1

Generated robots.txt

About This Tool

The robots.txt generator assembles crawl rules using standard syntax. Create multiple rule groups for different User-agents, each with its own Allow and Disallow paths and Crawl-delay, then append Sitemap URLs at the end. Three presets cover allow all, block all and block admin paths, so you can start from a working baseline.

Validation runs before generation: groups missing User-agent are reported and skipped, and Sitemap URLs must be absolute http or https links, preventing a single bad line from breaking the whole file.

Key Features

Multiple rule groups

Configure independent rules per search engine.

Common presets

Apply allow, block and admin-blocking configurations instantly.

Syntax validation

Checks for missing User-agent and invalid Sitemap URLs.

Deploy-ready output

Standard text you can copy into robots.txt at the site root.

When to Use It

New site launch

Generate baseline rules that allow crawling and declare the sitemap.

Block sensitive paths

Disallow admin, API and temporary directories to reduce exposure.

Per-engine control

Set crawl delays or blocks for specific search engines only.

How to Use It

  1. 1Pick a preset or add a rule group manually.
  2. 2Fill in User-agent plus Allow and Disallow paths, one per line.
  3. 3Enter absolute Sitemap URLs.
  4. 4Generate and copy the result to the site root.

Frequently Asked Questions

Does robots.txt prevent indexing?

robots.txt is only a crawl convention and does not guarantee removal from indexes. Use a noindex meta tag or header to block indexing completely.

What wins when Allow and Disallow conflict?

Allow wins for equally long prefixes, but engine implementations vary, so avoid writing conflicting rules.

Other Tools

CodeSwordsman

100+ technical tools with browser-local processing — your data never leaves your device. Official Microsoft Store desktop apps: free, ad-free, offline.

Completely Free

© 2026 CodeSwordsman All Rights Reserved

Service Terms | Privacy Policy | Help | Version Updates | About Us

This website uses cookies to ensure you get the best experience on our website. Privacy Policy.