See which AI crawlers read your website
Find out which bots from OpenAI, Anthropic, Perplexity, Google and others visit your site, which pages they read, which pages answer them with an error, and which visits only pretend to be a crawler.
AI assistants read your site, and you can’t see it
Most AI assistants get your pages through their own crawlers: GPTBot and OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Google’s bots for Gemini and AI Overviews. Google Analytics doesn’t show these visits: crawlers rarely run its tag, and GA4 leaves known bots out anyway. So you can’t tell whether they come, what they read, or whether they hit an error on the way.
How it works
- Connect your site: Install the WordPress plugin, or add a few lines for Cloudflare, Vercel, Netlify or Node.js, or let Vector or Fluent Bit read your Nginx or Apache logs. It takes a few minutes and one API key.
- Only crawler visits are sent: Your site reports a visit only when the user agent belongs to a known crawler, after the page has been served. Visits from people are never sent.
- Read the Crawlers tab: Visits per day and per AI platform, top pages, top crawlers, error pages, fake crawlers, and a visit log you can export to CSV.
What the Crawlers tab shows you
- Visits by AI platform: Crawler visits grouped by the product they feed, such as ChatGPT, Google, Copilot or Perplexity, compared with the previous period.
- Top pages and crawlers: The 10 pages crawlers read most and the 10 most active bots, with a daily chart by crawler.
- Pages crawlers could not read: Every page that answered a crawler with an error (status 400 or more), with the bot and the date of its last visit.
- Fake crawlers detected: Visits that used a crawler’s name but did not come from its operator’s addresses. They are kept out of every other number.
How Crawler Analytics works
Connect your site in a few minutes
Pick the integration that fits your site. Each one only reports visits from known crawlers, once the page has been served, and never changes what your visitors get. If a report fails, your site carries on as usual.
- WordPress plugin, Cloudflare Worker (free plan included), Vercel or Next.js, Netlify, Node.js or Express
- Nginx or Apache: Vector or Fluent Bit reads your access logs, nothing to change on the site
- Webflow through Cloudflare, and an API for any other server
See what crawlers read, and what fails
Top pages show where crawlers spend their visits. The error list shows the pages they could not read, with the status code, the crawler and the last visit, so you know which pages to fix first.
- Top 10 pages and top 10 crawlers over 7, 30 or 90 days
- Filters by page, status code and crawler
- Every page answered with a status of 400 or more, 404s and 500s included
Real crawler or fake one
Any scraper can call itself GPTBot. Vizible checks each visit against the IP lists that OpenAI, Anthropic, Google, Microsoft, Apple, Amazon, Perplexity and others publish. Fakes are flagged and left out of your numbers.
- Verified: the address is on the operator’s official list
- Fake: the list is current and the address is not on it
- Not verifiable: the operator publishes no list (Grok, Meta, ByteDance…), and the visit still counts
Every visit, ready to export
The visit log lists each visit, newest first: the time, the crawler, the status code, the method, the page, the referring domain and the verification. Export it to CSV with the filters you set.
- Up to 10,000 visits per CSV export
- Details kept 90 days, daily totals kept 400 days
- Googlebot and Bingbot counted as daily totals, without page detail
Why it matters
- Check that AI assistants can read you: If GPTBot, ClaudeBot or PerplexityBot never show up, something may be blocking them: your robots.txt, a firewall rule or your CDN. Now you can see it.
- Know which pages they read: See whether crawlers reach your product, pricing and blog pages, or spend their visits on old URLs you forgot about.
- Fix the errors that block them: A crawler that gets a 404 or a 500 leaves with nothing to read. Fix the error pages crawlers hit, starting with the ones they visit most.
- Spot fake crawlers: See how many visits only pretend to be AI crawlers, and which names they borrow, before you decide how open your site should be to bots.
FAQ
What is Crawler Analytics?
It’s the Crawlers tab in Vizible AI. It shows which AI crawlers and search engine bots visit your website, which pages they read, which pages answer them with an error, and whether each visit really came from the company that runs the bot.
Which crawlers does Vizible recognise?
At least 40, named from each operator’s own documentation where there is one: GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User, Google’s AI fetchers and Googlebot, Bingbot, Applebot, Amazonbot, meta-externalagent, Bytespider, CCBot, DuckAssistBot, MistralAI-User, Grok’s crawlers and more.
How do I connect my site?
In the Crawlers tab, create an API key, then follow the guide for your site: the WordPress plugin, a Cloudflare Worker (any Cloudflare plan, the free one included), Vercel or Next.js, Netlify, Node.js or Express, or Vector and Fluent Bit for Nginx and Apache logs. Any other server can send its visits through the API. Webflow works through Cloudflare. Wix and Shopify don’t allow it today, and Squarespace only at your own risk.
How do you tell a real crawler from a fake one?
Each visit’s IP address is compared with the lists the operators publish: OpenAI, Anthropic, Google, Microsoft, Apple, Amazon, Perplexity, DuckDuckGo, Common Crawl and others. The lists are refreshed every night. An address on the list is Verified. An address missing from a current list is Fake, and the visit is left out of your numbers. Crawlers whose operator publishes no list, Grok’s for example, show as not verifiable and still count, and so do visits that reach you through a proxy hiding the real address.
Do you store IP addresses?
No. The IP address is only used in memory to check the crawler, then dropped. Vizible keeps the crawler’s name, the page without its query string, the status code, the method, the referring domain and the time. Visits from people are never sent at all.
Does a crawler visit mean I will be cited?
No. A visit means the bot could fetch the page, not that an assistant will quote it. Perplexity, for example, often answers from its own index without visiting your page again. Use Crawler Analytics to make sure crawlers can read your pages, and Vizible’s AI visibility tracking to see whether you are actually mentioned.
Why don’t I see Gemini, AI Overviews or Copilot as their own crawler?
Google feeds Gemini, AI Overviews and AI Mode with its own crawlers, so their visits count under Google. When Gemini opens a page live, the request arrives with the single word “Google” as its user agent: Vizible lists it as Gemini (live), inside the Google card. Bingbot counts under Copilot, since Copilot answers from Bing.
Are there limits?
One website per account, every subdomain included. Up to 50,000 visits a day are kept in detail; past that, visits are still counted in the daily totals, which are always complete. Details are kept 90 days and daily totals 400 days. Googlebot and Bingbot are counted as daily totals only, without page detail.
Which plans include it?
Every plan, and the 7-day free trial too.