Black Friday30% off your first year on yearly plans with30% off yearly · . Ends 1 Dec.See pricing

Docs

How to fix

How to Fix Blocked AI Search Crawlers

How to Fix Blocked AI Search Crawlers

How to Fix Blocked AI Search Crawlers

Last Updated on

Read Time

2 min read

In short

What it does

This site check fails when your robots.txt blocks OAI-SearchBot, PerplexityBot or Claude-SearchBot.

Where

Framer Site settings > Hosting > Files

Cost

Free

Goes live

After you publish and recrawl.

Before you start

You uploaded your own robots.txt (Framer Pro)

Why it matters

AI search tools only show sites their crawler can read. Blocking them keeps you out of ChatGPT search, Perplexity and Claude search answers.

What RankFrame checks

RankFrame reads your robots.txt for the AI search crawlers OAI-SearchBot, PerplexityBot and Claude-SearchBot. Blocking AI training crawlers like GPTBot is your choice and is never shown as a problem.

How to fix it

1

Open Site checks

Open Audit > Pages, open any page and pick the Site checks tab.

2

Remove the rules

In Framer, open Site settings > Hosting > Files and remove the Disallow rules for the crawlers listed.

3

Publish and recrawl

Publish, then click Recrawl at the bottom of the plugin sidebar.

Tip:

You can block training crawlers and still allow search crawlers. They use different names.

Related guides

How to fix: Site check: Googlebot can read your site

Google Indexing Overview

Frequently asked questions

Which crawlers does RankFrame check?

OAI-SearchBot (ChatGPT search), PerplexityBot (Perplexity) and Claude-SearchBot (Claude search).

Can I still block AI training?
Does this affect my score?
Was this page helpful?

Still stuck?

Try the troubleshooting guides, or send support the page link and a screenshot.