Crawler control · free generator
Decide who may read your site, then write it down properly
robots.txt tells well-behaved crawlers where they may go. In 2026 that includes three kinds of AI crawler, and each one needs its own decision.
Training
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot
Collect pages that may train future models. Blocking them is a business choice and does not remove you from AI search on its own.
AI search
OAI-SearchBot, Claude-SearchBot, PerplexityBot
Build the indexes that AI answers cite. Block these and you lose the chance to be named.
User fetch
ChatGPT-User, Claude-User, Perplexity-User
Fetch a page because a person asked for it. They may not treat robots.txt the same way as crawlers.
Generate your robots.txt
Rules that stop expensive mistakes
- Never block CSS or JavaScript files: Google needs them to render the page.
- robots.txt does not remove a page from the index. Use a noindex meta tag (and let the page be crawled so the tag is seen).
- On WordPress, check Settings, Reading: "Discourage search engines" must be off on the live site.
- Test the file in Search Console's robots.txt report after every change.
- llms.txt is an optional plain-text summary of your key pages for AI tools. Keep it accurate. It is not a ranking signal and no engine promises to read it.
BreakdownStep by step
- Know the three kinds of crawler
Search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) and training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot).
- Decide on training separately
Blocking training crawlers is a business choice. It does not remove you from AI search, which uses different agents.
- Never block search crawlers by accident
A blanket "Disallow: /" or an over-eager firewall can hide you from Google and AI search at once.
- List your sitemap
Add a Sitemap line with the full URL.
- Consider llms.txt
A plain summary file for language models. Useful, but not a ranking factor and not a replacement for good pages.
Worked example
Two robots.txt files, two outcomes (illustrative)
File A blocks GPTBot and allows everything else: the site can still appear in ChatGPT search through OAI-SearchBot. File B blocks "*" except Googlebot: the site disappears from Bing, Copilot and most AI answers, and nobody notices for months.
TipsWhat works in practice
AvoidCommon mistakes
- Blocking all AI agents in one line and losing AI search visibility.
- Using robots.txt to hide private pages; use authentication instead.
- Forgetting the Sitemap line.
Quest · Gatekeeper badge
Gatekeeper
You block GPTBot. What happens to your ChatGPT search visibility?
Why: GPTBot is for training. ChatGPT search appearance is controlled by OAI-SearchBot.