Cloudflare: See Which AI Crawlers Read Your Site — and What GA4 Never Counts

Everyone wants to know whether AI is reading their site. Google Search Console added an AI Overviews report this year. Bing Webmaster Tools added Copilot citations. Both are useful, and both have the same problem: they live in a dashboard, with no API, so you can’t ask a question about them, join them to your other data, or put them on a schedule.

There is one place that sees every AI system touch your site, and it does have an API: your CDN. Cloudflare is now a connected data source in AI Data Stream, on Pro and Business plans. If your site sits behind Cloudflare, one read-only token gives your AI the view from the edge.

Two Questions Nobody Else Can Answer

The citation reports tell you whether an AI mentioned you. Cloudflare answers the question before that one, and the question after it.

Which AI systems are reading your site, and what are they reading? Every crawler announces itself in its user agent, and Cloudflare sees all of them: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot and Claude-SearchBot from Anthropic, PerplexityBot, Google’s Vertex crawler, Common Crawl’s CCBot, ByteDance’s Bytespider, Meta, Apple, Amazon and the rest. We group them the way Cloudflare does: AI Crawlers collecting training data, AI Search bots indexing for AI search products, and AI Assistants fetching a page live because someone just asked about you. That last category is the closest thing to “people asking an AI about your business” that any data source can currently offer.

Is any of it sending people back? When a ChatGPT or Perplexity answer links to you and someone clicks, Cloudflare sees the referrer. The AI referrals tool counts those visits by assistant and by landing page. One honest caveat, which we’ll come back to: Cloudflare only exposes referrer data on its paid plans.

What Your AI Can Do With It

The connection adds four data tools your AI can use in chat:

Which AI crawlers hit the site this month, and how much bandwidth did they use?

What pages is GPTBot reading? Are they the pages we rank for?

Which URLs are AI crawlers requesting that return 404?

Is ChatGPT sending us any actual visitors, and to which pages?

How does our edge page-view count compare with GA4 sessions for the last 14 days?

That last question is the one we didn’t expect to be the most interesting. Cloudflare counts every request at the edge: bots, ad-blocker users, visitors who declined the consent banner, and pages where the analytics tag simply failed to fire. Google Analytics counts only browsers that ran the tag. The gap between the two is real traffic you have never seen, and the new Edge vs Analytics Gap skill decomposes it into AI crawlers, other bots, and untracked humans, so you finally have a number for how much GA4 under-counts.

The AI Crawler Visibility skill does the rest in one run: who is reading the site, what they read, wasted crawls on dead URLs, bandwidth cost, and whether anyone is sending visitors back. Both skills are in the catalog and can be scheduled like any other.

On the Dashboard

The Cloudflare widget shows daily requests and page views, unique visitors, cache hit rate and blocked threats for your dashboard’s date range, alongside the AI crawlers seen at the edge, your top countries, and the split of response codes. It is the first widget in AI Data Stream that shows traffic from the server side rather than from a tag, which makes it a useful sanity check on every other widget.

Setup: One Permission

There is no OAuth flow and nothing to apply for. In the Cloudflare dashboard, create an API token with a single permission, Zone → Analytics → Read, limited to the zone you want to connect. Paste the token and the zone ID into the connection form, click Test Connection, and AI Data Stream confirms the token works and tells you how much history the zone keeps.

The token can read analytics and nothing else. The Cloudflare analytics API has no write operations at all, so there is no way for this connection to change a DNS record, a firewall rule or a cache setting, even by accident.

The Fine Print

Three things worth knowing before you connect:

• Retention depends on your Cloudflare plan. Cloudflare keeps per-request data for a limited window that is short on Free zones and longer on paid ones. Every tool checks what your zone actually retains, clamps its date range to it, and says so in the response. The daily traffic totals are kept far longer than the per-request data behind the crawler tools.
• AI referrals need a paid Cloudflare plan. Cloudflare only exposes the referrer host on paid zones. On a Free zone the crawler tools work in full and the referrals tool explains what is missing instead of failing.
• Crawlers are identified by user agent. Without Cloudflare’s Bot Management add-on, that is the only signal available, and user agents can be spoofed. We keep the list of known AI crawlers current, and the AI is told to treat unusual volume from a single bot with that caveat.

Get Started

Cloudflare is available on Pro and Business plans. Add it from your property’s Connections page. If you have questions, check the full documentation or contact us.