Error-Free Syntax
Generate a `robots.txt` file with the correct syntax every time, avoiding costly mistakes that could block important search engines.
✓Ready to ship
The AI `robots.txt` Generator
Generate a `robots.txt`.
Easily create a `robots.txt` file to control which AI bots can access your website. Select the bots you want to allow or block, and generate the rules instantly.
Step 01 · The generator
Configure the bots.
Pick which AI crawlers are in or out. Copy the file. Ship it.
Configure AI Bot Access
GPTBot
ChatGPT-User
anthropic-ai
ClaudeBot
PerplexityBot
Google-Extended
BingBot
Amazonbot
Applebot
FacebookBot
Bytespider
CCBot
Generated `robots.txt`
The case for it
Why use our `robots.txt` generator?
Take control of your website's relationship with AI. Our tool makes it simple to create clear, effective rules for AI crawlers.
Comprehensive Bot List
Our tool includes an up-to-date list of the most important AI crawlers, so you don't have to hunt down their user-agent strings.
✓Ready to ship
Instant Download & Copy
Get your generated `robots.txt` content immediately, ready to be copied or downloaded and uploaded to your server.
✓Ready to ship
The proof
Join thousands of smart webmasters.
50,000+
Files Generated
100%
Free & Unlimited
Zero
Errors in Syntax
The process
How it works.
Create your custom `robots.txt` file in three simple steps.
1
Select Bot Permissions
For each AI bot in our list, simply choose whether to 'Allow' or 'Block' its access.
2
Generate the File
The `robots.txt` content is generated in real-time based on your selections.
3
Copy or Download
Copy the generated text or download the `robots.txt` file directly, ready to be uploaded to your website's root directory.
The audience
Perfect for.
Bloggers
Easily specify which AI tools can use your articles for training data.
E-commerce Stores
Control how AI bots interact with your product pages and categories.
Startups & SaaS
Protect your proprietary marketing copy and feature descriptions from competitors using AI.
Anyone with a Website
Take a proactive step in managing your digital footprint in the age of AI.
The fine print
Frequently asked questions.
The `robots.txt` file must be placed in the root directory of your website. For example, `https://www.yourwebsite.com/robots.txt`.
`Disallow: /` tells a bot it should not crawl any pages on the site. `Allow: /` explicitly permits a bot to crawl all pages. If no rule is specified for a bot, it is implicitly allowed.
This basic generator blocks access to the entire site. For more complex rules, like blocking specific folders (e.g., `Disallow: /private/`), you can manually edit the generated file before uploading it.
The next move
Ready to create your `robots.txt`?
Generate your custom `robots.txt` file in seconds and take control of your site's AI bot access.
The reading room
The `robots.txt` file: your website's gatekeeper
The `robots.txt` file is a fundamental part of the Robots Exclusion Protocol (REP), a standard used by websites to communicate with web crawlers and other web robots. The file, which must be placed at the root of a domain, gives instructions about which parts of the website should not be processed or scanned by crawlers.
While originally designed for traditional search engines like Google, `robots.txt` has become the de facto standard for controlling access for a new wave of AI crawlers. These bots, operated by AI companies, collect vast amounts of text and data to train their models. By using a `robots.txt` file, you can signal your preferences about whether your content should be used for this purpose.
Key directives in `robots.txt`
Our generator uses two main directives to create rules for AI bots:
- User-agent: This directive specifies which crawler the rule applies to. For example, `User-agent: GPTBot` targets OpenAI's main training crawler. Each bot has a unique user-agent string.
- Disallow: This directive tells the specified user-agent not to crawl the paths that follow. Our generator uses `Disallow: /` to block a bot from accessing the entire website.
- Allow: While not part of the original standard, `Allow` is recognized by major crawlers like Google. It can be used to counteract a `Disallow` directive for a specific sub-path. Our generator uses `Allow: /` to explicitly permit access, which is the default behavior if no `Disallow` rule matches.
It's important to remember that `robots.txt` is a guideline, not an enforcement mechanism. Reputable companies will respect the rules you set, but malicious actors will likely ignore them.