ANI — Quick Answer
AI Narration and Indexing (ANI) is the discipline of making your website readable,
parseable, and credibly attributed to AI crawlers — including GPTBot, PerplexityBot,
and ClaudeBot — so that your content gets indexed by the AI systems that drive
modern answer engines.
Your content might be perfectly written and your SEO solid, but if an AI crawler
cannot access your site, parse your HTML cleanly, or identify who wrote the content,
none of your AEO or GEO work will show up in AI search results.
Open your robots.txt file via Rank Math → General Settings → Edit robots.txt.
Add these lines at the bottom to explicitly allow all major AI crawlers:
robots.txt — add to bottom of file
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: OAI-SearchBot
Allow: /
NOTE
After saving, visit yoursite.com/robots.txt in your browser and confirm these
lines appear. Some security plugins add broad Disallow rules that override your
Allow rules — check for these and remove them if found.
ANI implementation checklist
☐ yoursite.com/robots.txt checked — AI crawlers listed with Allow: /
☐ H1 then H2 then H3 hierarchy with no skipped levels on all pages
☐ Exactly one H1 tag per page (verified via View Page Source)
☐ No Word or Google Docs paste artifacts in the HTML
☐ Simple Author Box showing on all article pages
☐ Published and last-modified dates visible on articles
☐ llms.txt file live and accessible at yoursite.com/llms.txt
Want the complete walkthrough?
The Complete Optimization Playbook covers all six disciplines with step-by-step
instructions, every checklist, a 30-day plan, and exact plugin settings.
Get the full guide →
Frequently asked questions about ANI
Everything you need to know about AI Narration and Indexing.
What is ANI — AI Narration and Indexing?
ANI stands for AI Narration and Indexing. It is the discipline of making your website readable, parseable, and credibly attributed to AI crawlers — including GPTBot (ChatGPT), PerplexityBot, and ClaudeBot — so that your content is properly indexed by the AI systems that power modern answer engines. Without ANI, your AEO and GEO work may never be seen by the systems you are optimizing for.
What are the main AI crawlers I need to allow?
The main AI crawlers are: GPTBot (used by ChatGPT and OpenAI), PerplexityBot (Perplexity AI), ClaudeBot (Anthropic), Google-Extended (Google AI training and overviews), and OAI-SearchBot (OpenAI search). All five should be explicitly allowed in your robots.txt file. Some security plugins accidentally block these, so it is worth verifying your robots.txt is correct.
How do I allow AI crawlers in my robots.txt?
In WordPress with Rank Math, go to General Settings and click Edit robots.txt. Add these lines at the bottom: User-agent: GPTBot then Allow: /, then the same for PerplexityBot, ClaudeBot, Google-Extended, and OAI-SearchBot. After saving, visit yoursite.com/robots.txt in your browser and confirm the lines appear. Verify no other plugin is overriding them.
What is an llms.txt file?
An llms.txt file is a plain text document placed at yoursite.com/llms.txt that provides AI language models with a curated index of your most important pages. It follows a simple markdown-like format with your site name, a one-sentence description, and a list of key pages with their URLs and brief descriptions. Some AI crawlers actively read this file to prioritize and understand your content.
Why does HTML structure matter for AI indexing?
AI crawlers parse your HTML to understand the structure and hierarchy of your content. Clean semantic HTML — using H1, H2, H3 in the correct order, using nav, main, article, and section tags — gives AI parsers clear signals about what content is a heading, what is body text, what is navigation, and what is the main article. Poor HTML structure causes AI systems to misread or skip your content.
How do I know if AI crawlers are visiting my site?
Log into your hosting account and find the Raw Access Logs section under your domain settings. Download the most recent log file and open it in a text editor. Search for GPTBot, PerplexityBot, and ClaudeBot. If you find entries for any of these, AI systems are actively indexing your content. You can also install a plugin like WP Statistics to track bot visits in your WordPress dashboard.
Why is author attribution important for AI indexing?
AI systems are more likely to cite content that has clear, credible authorship. When your content is attributed to a named author with visible credentials and a bio page, AI systems can identify the content as coming from a qualified source. Content without clear authorship is treated with less trust. Add author attribution using the Simple Author Box plugin and Person schema on your author page.
What is a canonical tag and why do I need one?
A canonical tag is a line of HTML code that tells search engines and AI crawlers which version of a URL is the definitive one. Without it, if your page is accessible at multiple URLs (with and without trailing slash, with www and without, etc.), crawlers may index multiple versions and dilute your authority signals. Rank Math adds canonical tags automatically to every page — verify they exist by viewing your page source and searching for rel=canonical.
Does my site language setting affect AI indexing?
Yes. The lang attribute on your HTML element tells AI crawlers and search engines what language your content is in. This affects how your content is indexed, parsed, and matched to queries. Your HTML tag should start with html lang=”en” for English content. Check this in your theme settings or view your page source and look for the opening HTML tag.
What common mistakes block AI crawlers from indexing a site?
The most common mistakes are: a robots.txt file that blocks all crawlers with Disallow: / (often added by security plugins or staging site settings), missing or incorrect canonical tags, JavaScript-heavy pages that AI crawlers cannot render, no-index meta tags left from development, and pasted content from Word or Google Docs that fills the HTML with messy span tags and inline styles that confuse parsers.
The Complete Optimization Playbook
Ready to implement all six disciplines on your site?
Step-by-step WordPress instructions, six checklists, a 30-day action plan,
and exact plugin settings — everything you need to rank in Google, get cited
by AI, and build a real audience.
Get the Complete Guide →
