Most sites accidentally block at least one. We read your robots.txt, check 15 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more), and hand you back a corrected file you can paste.
AI engines respect robots.txt. If GPTBot or ClaudeBot is blocked, your pages will not show up in ChatGPT and Claude answers, even if everything else is in order.
Defaults from WordPress plugins, old SEO tools, and Yoast presets often block AI bots without telling the owner. We find at least one block on roughly two out of three sites we audit.
We read your live robots.txt, check it against 15 AI crawlers, and produce a corrected file that keeps your existing rules but explicitly allows the AI bots that were blocked.
The list we audit covers training crawlers (used to build models) and live-fetch crawlers (used when a user asks the assistant a question). Blocking either kind hurts your visibility.
GPTBotTraining dataOAI-SearchBotChatGPT browse and searchChatGPT-UserPer-user fetches from ChatGPTGoogle-ExtendedGemini trainingClaudeBotTraining and browseanthropic-aiLegacy Anthropic agentclaude-webLegacy Claude browsePerplexityBotIndex for Perplexity answersPerplexity-UserPer-user fetches from PerplexityApplebot-ExtendedApple Intelligence trainingcohere-aiCohere training and indexBytespiderTikTok and DoubaoCCBotTraining data for many LLMsAmazonbotAlexa+ and BedrockMeta-ExternalAgentLlama training