Loading
Can ChatGPT, Claude and Perplexity read and cite your site?
Paste a URL to see which AI crawlers your robots.txt lets in, whether your text is readable without JavaScript, and whether you have an llms.txt, a sitemap and structured data. You get a score, a fix for each problem and an llms.txt written for your site.
What we check
AI crawler access
- Page answers crawlers
- The page answers 200, not a bot wall or an error.
- AI search crawlers
- robots.txt lets in the crawlers ChatGPT, Claude and Perplexity use to find and quote pages.
- AI training crawlers
- GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, so you can choose.
Files for crawlers
- robots.txt
- Loads, and reads as rules rather than an error page.
- llms.txt
- A markdown summary of your site at /llms.txt, with links.
- Sitemap
- Listed in robots.txt or at /sitemap.xml, with URLs in it.
Readable content
- Readable without JavaScript
- Your text is in the HTML, since AI crawlers don't run scripts.
- Clear title and description
- They say what the site does, in words an assistant can reuse.
- Indexing and snippets
- No noindex, nosnippet or max-snippet:0 holding your text back.
- Questions and answers
- Direct answers to real questions, which assistants like to quote.
Structured data
- JSON-LD
- Structured data that tells machines what the page is.
- Organization or product schema
- Who is behind the site and what it offers.
Questions
- What is GEO?
- Generative engine optimization: making your site easy for AI assistants like ChatGPT, Claude and Perplexity to find, read and cite. It overlaps with SEO, but AI crawlers have their own names in robots.txt, and the major ones don't run JavaScript.
- Should I block AI training crawlers?
- That's your call. Blocking GPTBot or ClaudeBot keeps your pages out of training data, but models then know less about your product. Search crawlers such as OAI-SearchBot and Claude-SearchBot are different: block them and assistants can't find or cite your pages when people ask.
- What is llms.txt?
- A markdown file at /llms.txt that sums up your site and links to its key pages, proposed at llmstxt.org. Not every assistant reads it yet, but it takes minutes to add. We build one from your page that you can edit and upload.
- Does robots.txt control all AI access?
- It's a request, not a lock. The major AI companies say their crawlers follow it, but fetchers acting for a user may not, and a firewall or bot protection can turn away crawlers that robots.txt allows. We check both the file and whether your page answers our reader.
Get your product in front of search engines and AI
Launch on ProductsFarm and your product gets a permanent page in our sitemap and llms.txt, with structured data that search engines and AI assistants read and cite. Free listings take a minute.