How do I fix the robots-txt audit test in WebCare Pro Auditor?

Create or open your robots.txt file in the root directory (or app/robots.ts in Next.js). Declare User-agent: OAI-SearchBot and User-agent: PerplexityBot with Allow: /. Explicitly disallow aggressive training bots if you wish to protect proprietary content without losing search visibility. Ensure the Sitemap: directive points to your canonical XML sitemap. Verify with WebCare Pro GEO & AI Auditor.

Back to All 44 Audit Guides Run Live Audit on WebCare Pro

AI Bots StandardsTest ID: robots-txtVerified Specification

Robots.txt Configuration for AI Crawlers & Search Engines

How to properly configure robots.txt to grant search citation access to ChatGPT, Perplexity, and Claude while blocking non-consensual model scrapers.

Why this matters for your SEO & AI Visibility:

If your robots.txt inadvertently disallows OAI-SearchBot or PerplexityBot, AI search engines cannot index, cite, or recommend your website in live conversational answers.

Step-by-Step Remediation Guide

5 Actionable Steps

1Create or open your robots.txt file in the root directory (or app/robots.ts in Next.js).
2Declare User-agent: OAI-SearchBot and User-agent: PerplexityBot with Allow: /.
3Explicitly disallow aggressive training bots if you wish to protect proprietary content without losing search visibility.
4Ensure the Sitemap: directive points to your canonical XML sitemap.
5Verify with WebCare Pro GEO & AI Auditor.

Production Implementation Code

plaintext

Copy and deploy this production snippet into your application to satisfy the audit test.

robots-txt configuration snippet

# robots.txt - Optimized for Generative Engine Optimization (GEO)
User-agent: *
Allow: /

# Allow AI Search & Citation Engines
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-Web
Allow: /

# Disallow Unconsented LLM Model Training Crawlers
User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Technical Architecture & In-Depth Details

Why AI Bot Directives in robots.txt Are Critical in 2026

Traditional SEO focused strictly on Googlebot and Bingbot. In 2026, over 35% of web product searches occur inside AI answer engines like OpenAI Search, Perplexity AI, and Claude.

These platforms operate two distinct classes of crawlers:

Citation & Search Bots (OAI-SearchBot, PerplexityBot, Claude-Web): These browse the web in real-time when a user asks a question to cite your business and link to your landing pages.
Model Training Bots (GPTBot, CCBot, Bytespider): These scrape raw text to train future foundational weights.

Blocking all AI bots indiscriminately destroys your presence in AI-driven search results. The recommended configuration above separates citation crawlers from model training crawlers.