How to fix
Docs
How to fix
How to Fix Blocked AI Search Crawlers
Last Updated on
Read Time
2 min read
In short
What it does
This site check fails when your robots.txt blocks OAI-SearchBot, PerplexityBot or Claude-SearchBot.
Where
Framer Site settings > Hosting > Files
Cost
Free
Goes live
After you publish and recrawl.
Before you start
You uploaded your own robots.txt (Framer Pro)
Why it matters
AI search tools only show sites their crawler can read. Blocking them keeps you out of ChatGPT search, Perplexity and Claude search answers.
What RankFrame checks
RankFrame reads your robots.txt for the AI search crawlers OAI-SearchBot, PerplexityBot and Claude-SearchBot. Blocking AI training crawlers like GPTBot is your choice and is never shown as a problem.
How to fix it
1
Open Site checks
Open Audit > Pages, open any page and pick the Site checks tab.
2
Remove the rules
In Framer, open Site settings > Hosting > Files and remove the Disallow rules for the crawlers listed.
3
Publish and recrawl
Publish, then click Recrawl at the bottom of the plugin sidebar.

Tip:
You can block training crawlers and still allow search crawlers. They use different names.
Related guides
Frequently asked questions
Which crawlers does RankFrame check?
OAI-SearchBot (ChatGPT search), PerplexityBot (Perplexity) and Claude-SearchBot (Claude search).
Can I still block AI training?
Does this affect my score?
Related
Still stuck?
Try the troubleshooting guides, or send support the page link and a screenshot.