AI Crawlers Explained: Who They Are and How to Let Them In

Meet the crawlers

• **GPTBot / OAI-SearchBot** — OpenAI (ChatGPT & its search).
• **Google-Extended** — Google's AI experiences.
• **PerplexityBot** — Perplexity.
• **ClaudeBot** — Anthropic's Claude.

The trade-off

Blocking these crawlers protects nothing valuable for most commerce sites and guarantees you are invisible in AI answers. Allowing them is usually the right call.

How to configure robots.txt

Explicitly allow the AI user-agents you want, keep a `Sitemap:` reference, and avoid a blanket `Disallow: /` for `User-agent: *` unless you truly mean it.

Verify

After changing robots.txt, re-run your AI Visibility Audit — crawler access is one of the fastest wins to move your score.

Next step

Use the AI Visibility Checker to confirm each engine can reach and mention you.