SEO BrainFree · Built in Bahrain
Book a DemoLet’s Connect

Crawler control · free generator

Decide who may read your site, then write it down properly

robots.txt tells well-behaved crawlers where they may go. In 2026 that includes three kinds of AI crawler, and each one needs its own decision.

9 min readIntermediate80 XP quest

Training

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot

Collect pages that may train future models. Blocking them is a business choice and does not remove you from AI search on its own.

AI search

OAI-SearchBot, Claude-SearchBot, PerplexityBot

Build the indexes that AI answers cite. Block these and you lose the chance to be named.

User fetch

ChatGPT-User, Claude-User, Perplexity-User

Fetch a page because a person asked for it. They may not treat robots.txt the same way as crawlers.

Two honest limits. robots.txt is a request, not a lock: badly behaved bots ignore it, so real blocking happens at the firewall. And Google-Extended is a control token, not a separate crawler; it does not change normal Google ranking. [V] see sources

Generate your robots.txt

AI crawlers
AI search crawlers (be cited in answers)OAI-SearchBot, Claude-SearchBot, PerplexityBot
User-triggered fetchersChatGPT-User, Claude-User, Perplexity-User
Model training crawlersGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot

  

Rules that stop expensive mistakes

  • Never block CSS or JavaScript files: Google needs them to render the page.
  • robots.txt does not remove a page from the index. Use a noindex meta tag (and let the page be crawled so the tag is seen).
  • On WordPress, check Settings, Reading: "Discourage search engines" must be off on the live site.
  • Test the file in Search Console's robots.txt report after every change.
  • llms.txt is an optional plain-text summary of your key pages for AI tools. Keep it accurate. It is not a ranking signal and no engine promises to read it.

BreakdownStep by step

  1. Know the three kinds of crawler

    Search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) and training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot).

  2. Decide on training separately

    Blocking training crawlers is a business choice. It does not remove you from AI search, which uses different agents.

  3. Never block search crawlers by accident

    A blanket "Disallow: /" or an over-eager firewall can hide you from Google and AI search at once.

  4. List your sitemap

    Add a Sitemap line with the full URL.

  5. Consider llms.txt

    A plain summary file for language models. Useful, but not a ranking factor and not a replacement for good pages.

Worked example

Two robots.txt files, two outcomes (illustrative)

File A blocks GPTBot and allows everything else: the site can still appear in ChatGPT search through OAI-SearchBot. File B blocks "*" except Googlebot: the site disappears from Bing, Copilot and most AI answers, and nobody notices for months.

TipsWhat works in practice

Check your CDN or firewall bot settings too; they can block what robots.txt allows.
Test with the generator on this hub, then paste into your site.
Keep staging and internal folders disallowed.
Re-check robots.txt after every plugin or host change.

AvoidCommon mistakes

  • Blocking all AI agents in one line and losing AI search visibility.
  • Using robots.txt to hide private pages; use authentication instead.
  • Forgetting the Sitemap line.

Quest · Gatekeeper badge

Gatekeeper

3 tasks + 1 question · 80 XP

You block GPTBot. What happens to your ChatGPT search visibility?

Why: GPTBot is for training. ChatGPT search appearance is controlled by OAI-SearchBot.

Gatekeeper badge unlockedChapter complete. Your progress is saved in this browser.