robots.txt Generator
Build a robots.txt that says something about AI crawlers, not just Googlebot.
Applied to every group. A crawler uses the most specific group that matches it and ignores the others, so these are repeated inside each named group rather than inherited.
Google-Extended and Applebot-Extended send no User-Agent of their own — a robots.txt entry is the only way to express a preference about them, which is why they are here.
GPTBot
OpenAI
OAI-SearchBot
OpenAI
ChatGPT-User
OpenAI
ClaudeBot
Anthropic
Claude-User
Anthropic
Claude-SearchBot
Anthropic
PerplexityBot
Perplexity
Perplexity-User
Perplexity
GoogleOther
Bytespider
ByteDance
Amazonbot
Amazon
cohere-ai
Cohere
Meta-ExternalAgent
Meta
Google-Extended
Google · robots.txt only
Applebot-Extended
Apple · robots.txt only
Googlebot
Bingbot
Microsoft
DuckDuckBot
DuckDuckGo
User-agent: * Disallow: /admin Disallow: /api Disallow: User-agent: GPTBot Disallow: /admin Disallow: /api Allow: / User-agent: OAI-SearchBot Disallow: /admin Disallow: /api Allow: / User-agent: ChatGPT-User Disallow: /admin Disallow: /api Allow: / User-agent: ClaudeBot Disallow: /admin Disallow: /api Allow: / User-agent: Claude-User Disallow: /admin Disallow: /api Allow: / User-agent: Claude-SearchBot Disallow: /admin Disallow: /api Allow: / User-agent: PerplexityBot Disallow: /admin Disallow: /api Allow: / User-agent: Perplexity-User Disallow: /admin Disallow: /api Allow: / User-agent: GoogleOther Disallow: /admin Disallow: /api Allow: / User-agent: Bytespider Disallow: /admin Disallow: /api Allow: / User-agent: Amazonbot Disallow: /admin Disallow: /api Allow: / User-agent: cohere-ai Disallow: /admin Disallow: /api Allow: / User-agent: Meta-ExternalAgent Disallow: /admin Disallow: /api Allow: / User-agent: Google-Extended Disallow: /admin Disallow: /api Allow: / User-agent: Applebot-Extended Disallow: /admin Disallow: /api Allow: / Sitemap: https://example.com/sitemap.xml
robots.txt is a request, not a lock. Well-behaved crawlers honour it; nothing enforces it. To keep a page out of search results, use a noindex meta tag on a page crawlers are allowed to fetch.
TriRank in your browser
See what AI engines already know about the site you're on, and test whether AI crawlers can actually reach it.