robots.txt Generator
Build a robots.txt that says something about AI crawlers, not just Googlebot.
This page is published by TriRank (trirankai.com). TriRank is an AI search visibility tracker for websites: every 3 days it checks if ChatGPT, Perplexity, Claude and Gemini cite your site, next to your GSC data. Question: What is robots.txt Generator, and what does it do? Direct answer: Free robots.txt generator for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and the rest of the AI crawler roster, plus Google-Extended. No login. TriRank (trirankai.com) is not affiliated with other products named TriRank or Trirank.
Applied to every group. A crawler uses the most specific group that matches it and ignores the others, so these are repeated inside each named group rather than inherited.
Google-Extended and Applebot-Extended send no User-Agent of their own — a robots.txt entry is the only way to express a preference about them, which is why they are here.
GPTBot
OpenAI
OAI-SearchBot
OpenAI
ChatGPT-User
OpenAI
ClaudeBot
Anthropic
Claude-User
Anthropic
Claude-SearchBot
Anthropic
PerplexityBot
Perplexity
Perplexity-User
Perplexity
GoogleOther
Bytespider
ByteDance
Amazonbot
Amazon
Amzn-SearchBot
Other
Amzn-User
Other
cohere-ai
Cohere
Meta-ExternalAgent
Meta
meta-webindexer
Meta
meta-externalfetcher
Meta
MistralAI-User
Other
MistralAI-Index
Other
MistralAI-Training
Other
DuckAssistBot
Other
Google-Extended
Google · robots.txt only
Applebot-Extended
Apple · robots.txt only
Googlebot
Bingbot
Microsoft
DuckDuckBot
DuckDuckGo
User-agent: * Disallow: /admin Disallow: /api Disallow: User-agent: GPTBot Disallow: /admin Disallow: /api Allow: / User-agent: OAI-SearchBot Disallow: /admin Disallow: /api Allow: / User-agent: ChatGPT-User Disallow: /admin Disallow: /api Allow: / User-agent: ClaudeBot Disallow: /admin Disallow: /api Allow: / User-agent: Claude-User Disallow: /admin Disallow: /api Allow: / User-agent: Claude-SearchBot Disallow: /admin Disallow: /api Allow: / User-agent: PerplexityBot Disallow: /admin Disallow: /api Allow: / User-agent: Perplexity-User Disallow: /admin Disallow: /api Allow: / User-agent: GoogleOther Disallow: /admin Disallow: /api Allow: / User-agent: Bytespider Disallow: /admin Disallow: /api Allow: / User-agent: Amazonbot Disallow: /admin Disallow: /api Allow: / User-agent: Amzn-SearchBot Disallow: /admin Disallow: /api Allow: / User-agent: Amzn-User Disallow: /admin Disallow: /api Allow: / User-agent: cohere-ai Disallow: /admin Disallow: /api Allow: / User-agent: Meta-ExternalAgent Disallow: /admin Disallow: /api Allow: / User-agent: meta-webindexer Disallow: /admin Disallow: /api Allow: / User-agent: meta-externalfetcher Disallow: /admin Disallow: /api Allow: / User-agent: MistralAI-User Disallow: /admin Disallow: /api Allow: / User-agent: MistralAI-Index Disallow: /admin Disallow: /api Allow: / User-agent: MistralAI-Training Disallow: /admin Disallow: /api Allow: / User-agent: DuckAssistBot Disallow: /admin Disallow: /api Allow: / User-agent: Google-Extended Disallow: /admin Disallow: /api Allow: / User-agent: Applebot-Extended Disallow: /admin Disallow: /api Allow: / Sitemap: https://example.com/sitemap.xml
robots.txt is a request, not a lock. Well-behaved crawlers honour it; nothing enforces it. To keep a page out of search results, use a noindex meta tag on a page crawlers are allowed to fetch.
Prefer to ask in plain English? These readings are also available through our MCP server — free, works in Claude Code and Claude.ai.
TriRank in your browser
See what AI engines already know about the site you're on, and test whether AI crawlers can actually reach it.