--- title: "12 AI Web Crawlers You Need in robots.txt: 2026 Reference" description: "Master 12 AI web crawlers in robots.txt: configure ChatGPT-User, PerplexityBot, Claude, and Applebot, block ByteSpider at WAF, and test with AI Audit." canonical: "https://webcarespro.com/blog/post/12-ai-crawlers-robots-txt-guide" author: "Mir Alamin" date: "September 28, 2026, 11:15 AM" last_updated: "2026-09-16" category: "Cloudflare" tags: ["AI Crawlers","robots.txt","Cloudflare WAF","Generative Search"] --- # 12 AI Web Crawlers You Need in robots.txt: 2026 Reference The explosive growth of generative search engines and autonomous AI agents has completely disrupted the traditional rules of web crawling and `robots.txt` configuration. For decades, technical webmasters maintained a simple mental model: allow `Googlebot` and `Bingbot`, disallow internal admin directories, and leave `User-agent: *` open to the broader web. In 2026, that naive approach introduces two severe business risks: either your production origin servers are crushed by unthrottled, aggressive LLM training scrapers consuming 60% to 80% of server CPU, or your edge firewalls blindly block all AI user-agents, completely erasing your domain from ChatGPT Search, Perplexity AI, Claude Deep Research, and Apple Intelligence citations. The critical solution lies in establishing a strict architectural distinction between **Conversational Search Retrieval Bots** (which fetch live content to answer human user prompts and deliver high-intent citation traffic) and **Foundational Model Training Spiders** (which rip millions of pages into pre-training corpora with zero attribution or referral visits). In this production engineering masterclass, I break down the 12 primary AI web crawlers active in 2026: dissecting their network signatures, reverse DNS verification hosts, parent organizations, and operational purposes. Furthermore, I provide a copy-paste, battle-tested `robots.txt` configuration, dual-tier Cloudflare WAF routing rules, and demonstrate how to verify your domain's crawler health automatically using the [WebCare Pro AI Readiness & Technical SEO Audit Tool](https://webcarespro.com/ai-audit). --- ## 1. Prerequisites & Modern Crawler Defense Stack To implement and verify granular crawler control across edge and web server tiers, verify that your production environment satisfies these baseline requirements: - **Edge Layer**: Cloudflare (Free, Pro, Business, or Enterprise) with access to DNS management, Custom WAF Rules, or Bot Management. - **Web Server Layer**: Nginx 1.24+ or 1.26+ on Ubuntu 24.04 LTS or RHEL 10 with the `http_limit_req_module` and `http_geo_module` enabled. - **Testing & Verification Tools**: - The free [WebCare Pro AI Readiness & Technical SEO Audit Tool](https://webcarespro.com/ai-audit) to simulate and verify 12+ AI crawler user-agents against your live edge headers. - Command-line `curl` and `host` for DNS pointer (`PTR`) validation. - **Related Foundational Architecture**: Review my guides on [AI Scraper Defense: Shield Origins Without Losing SEO](/blog/post/ai-scraper-defense-origin-shielding), [How to Audit Websites for AI Search Readiness](/blog/post/audit-website-ai-search-readiness), and [Stabilize Origin Servers for AI Search Traffic Surges](/blog/post/stabilize-servers-for-ai-traffic-spikes). ``` ================================================================================ DUAL-TIER AI CRAWLER TRAFFIC CLASSIFICATION & ROUTING ================================================================================ [ Incoming HTTP/HTTPS Traffic ] │ ▼ [ Tier 1: Cloudflare Global Edge Network (WAF & Bot Mgmt) ] ├─ Group A: Verified Real-Time AI Search Engines ──► [ ALLOW: Global Edge Cache ] │ (ChatGPT-User, OAI-SearchBot, PerplexityBot, Claude-Web, Applebot-Extended) ├─ Group B: Traditional Organic Search Crawlers ────► [ ALLOW: Pass-Through ] │ (Googlebot, Bingbot, YandexBot) ├─ Group C: Foundational Model Pre-Training Spiders ► [ DISALLOW / THROTTLE ] │ (GPTBot, ClaudeBot, Google-Extended, Amazonbot) └─ Group D: Rogue Unregulated Scraping Clusters ────► [ DROP: 403 Forbidden ] (Bytespider, CCBot, Diffbot, Scrapy, HeadlessChrome) │ ▼ (Legitimate Cache Misses Only) [ Tier 2: Nginx Origin Web Server (FastCGI & Microcaching) ] ├─ Token-Bucket Rate Limiting (limit_req_zone) ├─ Real IP Restoration (CF-Connecting-IP) └─ Dynamic Honeypot Trap (/wp-content/ai-trap.php) ``` --- ## 2. The Comprehensive Technical Breakdown: 12 Primary AI Crawlers To configure access directives intelligently, technical architects must understand the precise role, parent corporation, user-agent string, and network behavior of each major crawler. Below is the definitive technical directory of the 12 AI web crawlers operating in 2026: ### 1. `ChatGPT-User` (OpenAI Real-Time Search) - **Parent Organization**: OpenAI - **Primary Function**: Real-time user browsing. When a user asks ChatGPT a question requiring up-to-date web information (*"What is the latest release of Ubuntu?"*), ChatGPT dispatches this crawler to fetch live web pages. - **User-Agent String**: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)` - **Reverse DNS Domain**: `*.openai.com` - **Architectural Policy**: **MUST ALLOW**. Blocking `ChatGPT-User` completely removes your website from real-time ChatGPT conversational search citations. ### 2. `OAI-SearchBot` (OpenAI Search Indexer) - **Parent Organization**: OpenAI - **Primary Function**: Dedicated search indexing crawler for OpenAI's search engine (ChatGPT Search). It builds and refreshes index listings so results appear instantly without requiring on-the-fly scraping during conversations. - **User-Agent String**: `OAI-SearchBot/1.0 (+https://openai.com/searchbot)` - **Reverse DNS Domain**: `*.openai.com` - **Architectural Policy**: **MUST ALLOW**. Essential for Generative Engine Optimization (GEO) in OpenAI search products. ### 3. `GPTBot` (OpenAI Model Training Harvester) - **Parent Organization**: OpenAI - **Primary Function**: Foundational model pre-training. Collects massive web text data to train future generations of GPT models (such as GPT-5). Does **not** directly power real-time user citations. - **User-Agent String**: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)` - **Reverse DNS Domain**: `*.openai.com` - **Architectural Policy**: **OPTIONAL / CONDITIONAL**. You can disallow `GPTBot` without harming your presence in ChatGPT Search. ### 4. `PerplexityBot` (Perplexity AI Search & Citations) - **Parent Organization**: Perplexity AI - **Primary Function**: Real-time search indexing and factual answer grounding. Perplexity operates as an answer engine that explicitly links and cites primary sources in numbered footnotes. - **User-Agent String**: `PerplexityBot/1.0 (+https://perplexity.ai/perplexitybot)` - **Reverse DNS Domain**: `*.perplexity.ai` - **Architectural Policy**: **MUST ALLOW**. Perplexity drives among the highest-converting referral traffic among all AI answer engines. Configure a courteous `Crawl-delay: 1` to prevent origin bursts. ### 5. `Claude-Web` (Anthropic Real-Time Retrieval) - **Parent Organization**: Anthropic - **Primary Function**: User-initiated live web fetching. When a Claude user instructs the model to inspect a specific URL, analyze a public documentation page, or research a current topic, `Claude-Web` retrieves the document. - **User-Agent String**: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-Web/1.0; +https://www.anthropic.com/claude-web)` - **Reverse DNS Domain**: `*.anthropic.com` - **Architectural Policy**: **MUST ALLOW**. Blocking this agent prevents enterprise and professional Claude users from referencing your technical guides or product pages. ### 6. `ClaudeBot` (Anthropic Foundation Corpus Crawler) - **Parent Organization**: Anthropic - **Primary Function**: Large-scale web scraping to build training corpora for future Claude models. Operates in high-throughput batches across public domains. - **User-Agent String**: `ClaudeBot/1.0 (+https://www.anthropic.com/claudebot)` - **Reverse DNS Domain**: `*.anthropic.com` - **Architectural Policy**: **DISALLOW OR RATE LIMIT**. If your origin server experiences high CPU load from broad crawls, you can disallow `ClaudeBot` while leaving `Claude-Web` allowed. ### 7. `Google-Extended` (Google Gemini & Vertex AI Token) - **Parent Organization**: Google LLC - **Primary Function**: Autonomous control token. Google uses `Google-Extended` specifically to let website owners opt out of having their content used to train Gemini models and Vertex AI foundation models. - **User-Agent String**: Controlled via `User-agent: Google-Extended` directive in `robots.txt`. - **Architectural Policy**: **CONFIGURABLE**. Crucially, disallowing `Google-Extended` **does NOT impact Googlebot or your organic rankings in Google Search**. It only governs generative model training. ### 8. `Applebot-Extended` (Apple Intelligence Web Grounding) - **Parent Organization**: Apple Inc. - **Primary Function**: Powers generative AI features across iOS, macOS, and Siri in Apple Intelligence. Operates alongside traditional `Applebot` (which handles Spotlight search indexing). - **User-Agent String**: `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)` - **Reverse DNS Domain**: `*.apple.com` - **Architectural Policy**: **MUST ALLOW**. Essential for ensuring your brand and technical answers are suggested to hundreds of millions of Apple Intelligence users. ### 9. `Amazonbot` (Amazon Alexa & Rufus E-Commerce Agent) - **Parent Organization**: Amazon - **Primary Function**: Discovers and indexes web content to power Amazon Rufus (the in-app shopping assistant), Alexa voice answers, and AWS generative intelligence services. - **User-Agent String**: `Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)` - **Reverse DNS Domain**: `*.amazon.com` - **Architectural Policy**: **ALLOW (WITH RATE LIMITS)**. Vital for e-commerce stores, product manufacturers, and SaaS vendors seeking attribution in Amazon's conversational ecosystem. ### 10. `Meta-ExternalAgent` (Meta AI Assistant) - **Parent Organization**: Meta Platforms (Facebook, Instagram, WhatsApp) - **Primary Function**: Fetches real-time web context when users interact with Meta AI across WhatsApp, Messenger, and Instagram feeds. - **User-Agent String**: `Mozilla/5.0 (compatible; Meta-ExternalAgent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler)` - **Reverse DNS Domain**: `*.fbsv.net` / `*.tfbnw.net` - **Architectural Policy**: **MUST ALLOW**. Drives conversational citations within the world's largest social messaging platforms. ### 11. `cohere-ai` (Cohere Enterprise Search & RAG) - **Parent Organization**: Cohere - **Primary Function**: Harvests enterprise web data to power enterprise Retrieval-Augmented Generation (RAG) models, semantic search embeddings, and business intelligence tooling. - **User-Agent String**: `Mozilla/5.0 (compatible; cohere-ai/1.0; +https://cohere.com/bot)` - **Reverse DNS Domain**: `*.cohere.ai` - **Architectural Policy**: **ALLOW**. Broadly utilized in B2B enterprise semantic search tooling. ### 12. `Bytespider` (ByteDance Scraper — The Rogue Offender) - **Parent Organization**: ByteDance (TikTok / Douyin) - **Primary Function**: Large-scale scraping for ByteDance LLMs and recommendation engines. - **Operational Reality**: **Extremely Aggressive**. In production server audits, `Bytespider` is frequently responsible for 40% to 60% of all bot requests hitting origin servers. It notoriously ignores `crawl-delay` directives and repeatedly hammers dynamic search and checkout endpoints. - **User-Agent String**: `Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)` - **Architectural Policy**: **STRICTLY DISALLOW & BLOCK AT EDGE WAF**. Do not allow `Bytespider` to touch your origin server; drop its requests at the Cloudflare edge. --- ## 3. The Production robots.txt Master Configuration A high-performance `robots.txt` must achieve three simultaneous goals: 1. Protect sensitive administrative paths and private e-commerce funnels. 2. Grant frictionless, prioritized access to verified AI search and attribution crawlers. 3. Explicitly forbid unthrottled foundational training scrapers and abusive harvesters. Save the following production configuration in your webroot (`public/robots.txt`): ```text # ============================================================================== # WebCare Pro Production robots.txt - AI Search & Agentic Specification # Target Domain: https://webcarespro.com # Specification: RFC 9309 Compliant + AI Content Signals # ============================================================================== # ------------------------------------------------------------------------------ # 1. DEFAULT GLOBAL POLICY (Traditional Engines & Baseline Protections) # ------------------------------------------------------------------------------ User-agent: * Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /checkout/ Disallow: /cart/ Disallow: /account/ Disallow: /search/ Disallow: /?s=* Disallow: /api/internal/ Allow: /wp-admin/admin-ajax.php # ------------------------------------------------------------------------------ # 2. VERIFIED REAL-TIME AI SEARCH & ANSWER ENGINES (GEO DISCOVERY) # ------------------------------------------------------------------------------ # OpenAI Real-Time Conversational Search User-agent: ChatGPT-User Allow: / # OpenAI SearchBot (ChatGPT Search Indexer) User-agent: OAI-SearchBot Allow: / # Perplexity AI Citation Engine User-agent: PerplexityBot Allow: / Crawl-delay: 1 # Anthropic Claude User-Initiated Web Retrieval User-agent: Claude-Web Allow: / # Apple Intelligence & Siri Conversational Grounding User-agent: Applebot-Extended Allow: / # Amazon Alexa & Rufus Shopping Assistant User-agent: Amazonbot Allow: / # Meta AI (WhatsApp / Messenger / Instagram Retrieval) User-agent: Meta-ExternalAgent Allow: / # Cohere Enterprise Semantic Search & RAG User-agent: cohere-ai Allow: / # ------------------------------------------------------------------------------ # 3. SELECTIVE TRAINING & DATASET HARVESTING RESTRICTIONS # ------------------------------------------------------------------------------ # OpenAI Model Pre-Training Spider (Disallowed to shield origin compute) User-agent: GPTBot Disallow: / # Anthropic Model Pre-Training Spider User-agent: ClaudeBot Disallow: / # Google Gemini & Vertex AI Foundation Model Training Token User-agent: Google-Extended Disallow: / # ------------------------------------------------------------------------------ # 4. AGGRESSIVE & UNREGULATED DATA SCRAPERS (MANDATORY BLOCK) # ------------------------------------------------------------------------------ User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Diffbot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Scrapy Disallow: / # ------------------------------------------------------------------------------ # 5. MACHINE CONTEXT PROTOCOL & SITEMAP DECLARATIONS # ------------------------------------------------------------------------------ # Content-Signal header for emerging autonomous agent protocols Content-Signal: search=yes, ai-train=no, ai-input=yes # Machine-Readable Context Directory # /llms.txt: Curated Markdown feed for LLM context ingestion # /llms-full.txt: Complete technical knowledge base Sitemap: https://webcarespro.com/sitemap.xml ``` --- ## 4. Edge Enforcement: Cloudflare WAF Rules for AI Crawlers Because `robots.txt` is an advisory protocol based on voluntary compliance, rogue scrapers like `Bytespider`, commercial data brokers, and headless Python scripts will frequently ignore your disallow directives and hammer your web server anyway. To guarantee server stability, enforce crawler boundaries at the **Cloudflare Edge Layer** before incoming traffic ever touches your origin CPU or database. ### Step 1: Create Edge Priority Skip Rule for AI Search Bots Navigate to **Cloudflare Dashboard** -> **Security** -> **WAF** -> **Custom Rules**. Create a high-priority rule titled `01_Permit_Verified_AI_Search`: ```text (http.user_agent contains "ChatGPT-User") or (http.user_agent contains "OAI-SearchBot") or (cf.client.bot and http.user_agent contains "PerplexityBot") or (http.user_agent contains "Claude-Web") or (cf.client.bot and http.user_agent contains "Applebot") or (http.user_agent contains "Meta-ExternalAgent") ``` - **Action**: **Skip** (Select: Bypass all remaining Custom Rules, Rate Limiting, and Super Bot Fight Mode). - **Result**: Legitimate AI search bots receive instant sub-30ms cached responses without being intercepted by Turnstile verification challenges. ### Step 2: Create Edge Block Rule for Abusive Training Spiders Create a second custom rule titled `02_Block_Aggressive_Scrapers`: ```text (http.user_agent contains "Bytespider") or (http.user_agent contains "Diffbot") or (http.user_agent contains "CCBot") or (http.user_agent contains "ImagesiftBot") or (http.user_agent contains "Scrapy") ``` - **Action**: **Block** (Returns HTTP 403 Forbidden at the Cloudflare edge point of presence in < 3ms). - **Result**: Zero PHP worker sockets, zero RAM allocation, and zero bandwidth consumption on your origin server. --- ## 5. Origin Hardening: Nginx User-Agent Rate Limiting If your origin IP address is exposed, or if you operate outside of a Cloudflare proxy, you must configure rate-limiting zones in Nginx to safeguard PHP-FPM pools from crawler exhaustion. Open `/etc/nginx/nginx.conf` and inject the following mapping and rate-limiting zones inside the `http {}` block: ```nginx # /etc/nginx/nginx.conf # Map incoming User-Agents to security risk levels map $http_user_agent $bot_threat_level { default 0; # Level 1: Rogue scrapers that must be dropped instantly ~*(Bytespider|Diffbot|CCBot|ImagesiftBot|Scrapy|HeadlessChrome) 1; # Level 2: Foundational training crawlers (Permit with heavy rate-limits) ~*(ClaudeBot|GPTBot|cohere-ai) 2; # Level 3: Verified AI search engines (Permit with generous burst capacity) ~*(PerplexityBot|ChatGPT-User|OAI-SearchBot|Claude-Web) 3; } # Define dedicated shared-memory rate limiting zones limit_req_zone $binary_remote_addr zone=ai_search_zone:10m rate=12r/s; limit_req_zone $binary_remote_addr zone=ai_training_zone:10m rate=2r/s; limit_req_zone $binary_remote_addr zone=standard_crawlers:10m rate=5r/s; ``` Now, apply these protections within your server block (`/etc/nginx/sites-available/production.conf`): ```nginx # /etc/nginx/sites-available/production.conf server { listen 443 ssl http2; server_name webcarespro.com; # 1. Instantly terminate rogue scrapers with zero response body if ($bot_threat_level = 1) { return 444; # Special Nginx non-standard code: close TCP socket immediately } # 2. Rate limit foundation training spiders to protect PHP workers if ($bot_threat_level = 2) { set $rate_limit_zone "ai_training_zone"; } # 3. Dynamic Application Gateway location / { try_files $uri $uri/ /index.php?$args; # Apply appropriate rate limits to dynamic requests limit_req zone=standard_crawlers burst=10 nodelay; } # 4. Shield Dynamic Search from Synthetic Crawler Saturation location ~* ^/(?:search|sitemap|?s=) { limit_req zone=standard_crawlers burst=3 nodelay; try_files $uri $uri/ /index.php?$args; } # 5. FastCGI Caching & Stale-While-Revalidate Origin Shield location ~ \.php$ { include fastcgi_params; fastcgi_pass unix:/run/php/php8.3-fpm.sock; fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name; # Serve microcache while regenerating in background fastcgi_cache PRODUCTION_CACHE; fastcgi_cache_valid 200 301 302 10m; fastcgi_cache_use_stale error timeout updating http_500 http_503; fastcgi_cache_lock on; } } ``` Verify your configuration and reload Nginx: ```bash sudo nginx -t && sudo systemctl reload nginx ``` --- ## 6. How to Verify Authentic AI Crawlers via Reverse DNS Malicious scrapers and botnet operators routinely spoof user-agent headers by impersonating `PerplexityBot` or `ChatGPT-User`. To determine whether an incoming crawler is genuine or an imposter, execute a two-step reverse DNS lookup: ```bash # Step 1: Perform reverse DNS lookup on the incoming client IP host 20.171.207.214 # Output: # 214.207.171.20.in-addr.arpa domain name pointer chatgpt-user-20-171-207-214.openai.com. # Step 2: Forward DNS lookup to confirm IP matches the reverse host host chatgpt-user-20-171-207-214.openai.com # Output: # chatgpt-user-20-171-207-214.openai.com has address 20.171.207.214 ``` If the domain pointer does not terminate in the official parent organization's domain (`*.openai.com`, `*.perplexity.ai`, `*.anthropic.com`, `*.apple.com`), the request is a spoofed scraper and should be dropped immediately. --- ## 7. Automated Testing with the WebCare Pro AI Readiness Audit Tool Manually crafting `curl` requests with spoofed user-agent headers to test all 12 crawlers against your `robots.txt` and edge firewall rules is cumbersome and error-prone. To solve this, use the **[WebCare Pro AI Readiness & Technical SEO Audit Tool](https://webcarespro.com/ai-audit)**. The tool includes a dedicated **AI Crawler Simulation Suite** that automatically executes real-world HTTP handshakes against your website across all major AI search agents: 1. **Verifies robots.txt Directives**: Validates whether your file correctly permits `ChatGPT-User`, `PerplexityBot`, `Claude-Web`, and `Applebot-Extended`. 2. **Detects Edge Firewall Misconfigurations**: Checks if Cloudflare Super Bot Fight Mode or aggressive WAF rules are inadvertently returning HTTP 403 Forbidden or Turnstile challenges to legitimate search bots. 3. **Inspects Content-Signal Headers**: Confirms whether your server properly broadcasts machine rights directives (`search=yes, ai-train=no`). 4. **Delivers an Instant PDF Executive Audit**: Generates an exhaustive report with exact remediation directives for your systems administration team. Test your website's crawler readiness for free at: **[https://webcarespro.com/ai-audit](https://webcarespro.com/ai-audit)**. --- ## Production Architectural Specifications & Reference Standards The reference matrix below outlines the 12 primary AI web crawlers, their operational characteristics, and recommended production handling: | Crawler User-Agent | Parent Company | Operational Purpose | Primary Reverse DNS Domain | Recommended Action | Risk if Blocked | | :--- | :--- | :--- | :--- | :--- | :--- | | **`ChatGPT-User`** | OpenAI | Live user queries in ChatGPT | `*.openai.com` | **Allow (Priority)** | Loss of real-time ChatGPT citations | | **`OAI-SearchBot`** | OpenAI | ChatGPT Search engine indexing | `*.openai.com` | **Allow (Priority)** | Disappearance from SearchGPT index | | **`GPTBot`** | OpenAI | Foundational model pre-training | `*.openai.com` | **Disallow / Limit** | Minimal (Zero impact on citations) | | **`PerplexityBot`** | Perplexity AI | Conversational search & citations | `*.perplexity.ai` | **Allow (Crawl-delay 1)** | Complete removal from Perplexity answers | | **`Claude-Web`** | Anthropic | Real-time web retrieval in Claude | `*.anthropic.com` | **Allow (Priority)** | Claude cannot analyze your URLs | | **`ClaudeBot`** | Anthropic | Foundational Anthropic pre-training | `*.anthropic.com` | **Disallow / Limit** | Minimal (Shields origin CPU) | | **`Google-Extended`** | Google | Controls Gemini & Vertex training | Managed via robots.txt | **Disallow / Opt-out** | Zero impact on Googlebot or Google Search | | **`Applebot-Extended`**| Apple Inc. | Apple Intelligence & Siri answers | `*.apple.com` | **Allow (Priority)** | Invisibility on iOS & macOS AI features | | **`Amazonbot`** | Amazon | Alexa voice & Rufus AI shopping | `*.amazon.com` | **Allow** | Omission from Alexa & Rufus recommendations | | **`Meta-ExternalAgent`**| Meta | Meta AI in WhatsApp & Instagram | `*.fbsv.net` | **Allow** | Omission from Meta AI search answers | | **`cohere-ai`** | Cohere | Enterprise RAG & semantic search | `*.cohere.ai` | **Allow** | Exclusion from B2B search platforms | | **`Bytespider`** | ByteDance | Aggressive training scraper | `*.bytedance.com` | **Block at Edge WAF** | Zero risk; prevents origin server collapse | ### Authoritative Technical References & RFC Standards | Standard / Specification | Governing Body / Publisher | Canonical Reference URL | | :--- | :--- | :--- | | **RFC 9309 (Robots Exclusion Protocol)** | [Internet Engineering Task Force (IETF)](https://datatracker.ietf.org/doc/html/rfc9309) | Formal specification of the `robots.txt` syntax, path matching, and disallow rules | | **OpenAI Crawler Documentation** | [OpenAI Developers](https://platform.openai.com/docs/bots) | Official guide to GPTBot, OAI-SearchBot, and ChatGPT-User IP ranges and behavior | | **Anthropic Crawler Reference** | [Anthropic Technical Documentation](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Official guide to Claude-Web and ClaudeBot user-agents | | **Perplexity AI Bot Specification** | [Perplexity Documentation](https://docs.perplexity.ai/) | Directives and IP verification for PerplexityBot | | **Applebot Documentation** | [Apple Support](https://support.apple.com/en-us/119829) | Official guide to Applebot and Applebot-Extended specifications | | **Cloudflare Bot Management** | [Cloudflare Docs](https://developers.cloudflare.com/bots/) | Heuristic detection, verified bot directories, and edge WAF rules | --- ## Recommended Next Steps & Related Architecture Guides To complete your edge security and server reliability architecture for the AI era, review these related engineering masterclasses: - **[How to Audit Websites for AI Search Readiness: 2026 Guide](/blog/post/audit-website-ai-search-readiness)** — The comprehensive 5-phase diagnostic blueprint for evaluating TTFB, Schema graphs, and WebMCP. - **[Building an AI Agent Ready Website: Architecture & Hiring Guide](/blog/post/ai-agent-ready-website-architecture-guide)** — How WebCare Pro ranked #1 globally on GeoTest.ai with sub-50ms TTFB and WebMCP endpoints. - **[AI Scraper Defense: Shield Origins Without Losing SEO](/blog/post/ai-scraper-defense-origin-shielding)** — Complete guide to configuring Cloudflare edge firewalls and token-bucket Nginx rate limits. - **[Cloudflare Edge Security & WAF Masterclass](/blog/post/cloudflare-edge-waf-ddos-security-masterclass)** — Deep dive into Layer 7 firewall rules, Turnstile challenge tokens, and bot fighting modes. - **[Stabilize Origin Servers for AI Search Traffic Surges](/blog/post/stabilize-servers-for-ai-traffic-spikes)** — Engineer high-concurrency microcaching and Redis architectures to survive viral AI traffic bursts. --- ## Frequently Asked Questions (FAQ) ### Q1: Will disallowing GPTBot or ClaudeBot prevent my website from being cited in ChatGPT or Claude? No. Foundational training crawlers (`GPTBot`, `ClaudeBot`) are distinct from conversational search and retrieval agents (`ChatGPT-User`, `OAI-SearchBot`, `Claude-Web`). Disallowing `GPTBot` simply instructs OpenAI not to use your content to train future foundational model weights. As long as `ChatGPT-User` and `OAI-SearchBot` remain explicitly allowed in your `robots.txt`, ChatGPT will continue to crawl, cite, and link to your website during active user queries. ### Q2: Why does ByteSpider ignore my robots.txt crawl-delay directive? `robots.txt` is an advisory standard based on voluntary compliance. Many unregulated scrapers and rogue crawlers—most notably `Bytespider`—deliberately ignore `crawl-delay` directives to scrape content at maximum speed. Furthermore, `crawl-delay` is not part of the official IETF standard (RFC 9309). To stop aggressive spiders, you must enforce deterministic IP or user-agent drop rules at the edge CDN layer (such as Cloudflare WAF) or at the web server layer (via Nginx return 444 directives). ### Q3: How do I test whether my edge WAF is accidentally blocking AI search engines? The fastest and most reliable method is running your domain through the [WebCare Pro AI Readiness Audit Tool](https://webcarespro.com/ai-audit). The audit engine simulates requests using real-world user-agent strings for 12+ AI crawlers and analyzes the HTTP status codes, headers, and challenge bodies returned by your server. Alternatively, you can test manually from the command line using `curl -I -A "PerplexityBot" https://example.com/` to verify that your edge returns `200 OK` rather than a `403 Forbidden` or Cloudflare Turnstile challenge. ### Q4: Does Google-Extended in robots.txt impact my organic Google Search rankings? No. Google has officially documented that `Google-Extended` only controls whether your website's content can be used to train Google's Gemini models and Vertex AI generative APIs. It has zero effect on `Googlebot` or your site's visibility, indexing, or ranking in standard Google Search results. --- *© 2026 WebCare Pro. Authored by Mir Alamin.* ## Sitemap See the full [sitemap](/sitemap.md) for all pages. - **Canonical URL:** https://webcarespro.com/blog/post/12-ai-crawlers-robots-txt-guide - **Markdown Mirror:** https://webcarespro.com/blog/post/12-ai-crawlers-robots-txt-guide.md - **Blog Sitemap:** https://webcarespro.com/blog/sitemap.xml - **Main Website Sitemap:** https://webcarespro.com/sitemap.xml - **Markdown Sitemap:** https://webcarespro.com/sitemap.md - **LLMs Context Feed:** https://webcarespro.com/llms.txt - **Full LLMs Index:** https://webcarespro.com/llms-full.txt - **AI Agent Skills:** https://webcarespro.com/AGENTS.md - **WebMCP Tool Catalog:** https://webcarespro.com/.well-known/webmcp.json