Version 9.4.6 | SEO Book Pro

The main logic behind this version is:
OpenAI uses OAI-SearchBot for ChatGPT search visibility, while GPTBot is the training crawler; ChatGPT-User is user-initiated and not the main automatic search crawler. Google’s AI features in Search are governed by normal Googlebot crawling, while Google-Extended is the separate control for Gemini/Vertex training and grounding. Perplexity uses PerplexityBot for search surfacing, and Anthropic distinguishes ClaudeBot, Claude-SearchBot, and Claude-User.

For the blocked competitors, Semrush documents SemrushBot and SiteAuditBot, Ahrefs documents AhrefsBot and AhrefsSiteAudit, Majestic documents MJ12bot, and Moz’s common crawler tokens are Rogerbot and DotBot.

One caution: robots.txt is advisory, not hard security. User-agents can be spoofed, and some user-triggered fetchers are not governed the same way as automatic crawlers. For strict enforcement, mirror these rules in Cloudflare/WAF/server bot rules and allowlist approved bot IP ranges where vendors publish them.