Skip to main content
The rest of MentionLab reads what AI engines answer. Agent analytics reads what they do on your site: it takes your site’s request logs from Cloudflare and shows the AI bots that read your pages, and the people that AI apps send to you.
Beta. This feature is not on by default. To turn it on for your organization, contact [email protected].
Agent analytics needs a site behind Cloudflare. If the side menu has no Agent analytics group, the feature is not on for your organization.

The pages

Key terms

Every bot has one of three categories, with the same color on every page: MentionLab knows about 90 bots and 22 AI apps. A bot outside this list, such as an SEO tool or an uptime monitor, is not counted.

Set up

You connect one log source per Cloudflare zone. A source sends the zone’s logs to MentionLab in one of two modes:
Cloudflare offers Logpush as self-service on its Free, Pro and Business plans. On an Enterprise plan it depends on your contract. See Cloudflare’s Logpush documentation.
1

Add a log source

Open Settings → Project → Log sources and click Add a log source.
  • Name: a label for you, such as “Main website”.
  • Hostnames: the hosts of the zone, one per line, up to 20. Each one covers its subdomains: brand.com covers shop.brand.com.
  • Mode: Logpush or Worker.
MentionLab then shows an ingest URL and an ingest token.
The ingest token shows once. With the Worker, copy both values now. With Logpush you do not need them: the next step creates the job with a token of its own.
2

Connect Cloudflare

MentionLab creates the Logpush job with a Cloudflare API token that you make for this one use:
  1. In Cloudflare, open My Profile → API Tokens, then Create Token → Create Custom Token.
  2. Add a zone policy: Include, Specific zone, then the zone of your hostnames. Set a TTL that ends soon, such as tomorrow.
  3. In that policy, tick two permissions: Zone → Read under DNS & Zones, and Logs → Read and Edit under Analytics & Logs.
  4. Create the token, paste it in MentionLab and click Connect.
MentionLab uses the token once to create the job, and does not keep it.
3

Check that the logs arrive

Back on Log sources, the row of your source shows a Last event time and a count under Rows kept, 7 days once the first bot reads your site. A Logpush source that has no job yet shows Not connected under its mode.
The data starts when you connect the source. MentionLab cannot read the logs from before that moment.

Log sources

Settings → Project → Log sources lists your sources, with their health. Most log lines are dropped, and that is normal: a Logpush job sends every request of your zone, and Agent analytics keeps only the AI bots and the AI visits. The menu of a source holds Edit source, Connect Cloudflare, Check access now, Rotate token and Delete source.

Rows this month

The card above the list shows the rows your organization stored this calendar month (UTC), against its limit. A hit and a visit each count as one row, and every source of the organization shares the limit.
Past the limit, new rows are dropped until the 1st of the next month. The organization’s admins get an email at 80% and at 100% of the limit.

Overview

Agent analytics → Traffic → Overview sums up the date range. Below the figures:
  • Hits over time, split by Category, Operator, Bot or Status.
  • Operators: the companies behind the bots, each with its hits and its number of bots.
  • Top fetched pages: the pages AI agents fetched most for people.

Bots

Agent analytics → Traffic → Bots has one row per bot that hit your site. The filters above the table narrow every figure: Hosts, Operators, Bots, Categories, Verification and Status. The Overview and the Pages page have the same ones. Click a row to open the bot’s drawer: its hits over time, its response times (average, median and 95th percentile), its mix of HTTP statuses, the 25 pages it hit most, and its spoofing addresses.
The robots.txt column reads one host. Pick exactly one host in the Hosts filter to fill it.

Verified and spoofed

Anyone can send a request that claims to be GPTBot. MentionLab checks each hit: By default, every figure counts the verified hits only. To add the hits that nothing could check, pick Not checked in the Verification filter. To see the fakes, pick Spoofed, or open the bot’s drawer.
Most bots have no published way to be verified, and a coding agent runs on a person’s own computer. Their hits read “Not checked”, so these bots show only when you pick Not checked. A bot that then reads 0% verified is not a spoofed bot.

Pages

Agent analytics → Traffic → Pages puts two things side by side for each page: how bots read it in your logs, and how AI answers cite it in your tracked prompts. Switch between Pages and Folders above the table. Beside the bot filters, this page has Folders, a path search (Starts with or Contains) and Engines. Engines narrows the citations to the engines you pick, and the hits to the operators of those engines. Click a row to open the page’s drawer: its hits over time per operator, its mix of HTTP statuses, and its citations per engine.
One page is one row, whatever the form of its address. www.brand.com/Blog/ and brand.com/blog?utm_source=x count as the same page.

AI visitors

Agent analytics → Traffic → AI visitors counts people, not bots. A visit is one landing on a page of your site from a link in an AI app’s answer. MentionLab finds it from the referrer of the request, or from the utm_source=chatgpt.com tag that ChatGPT adds to its links. The page shows the total of visits, the AI apps that sent the most, the visits over time per AI app, and a table that you can read by AI app, Landing page, Country or Device. Its filters are Hosts, AI apps, Countries and Devices.
An AI visit is a lower bound. An AI app that opens links without a referrer, or a person who copies the address, is not counted.

Access

Agent analytics → Access shows what your site tells AI bots. Pick a host at the top.

Files

MentionLab reads three files on each host once a day: robots.txt, llms.txt and llms-full.txt. Check now asks for a new read, and the result shows within a minute. A host can be checked once a minute at most. When your llms.txt is found, a card shows what it says: its title, its summary, and its sections with the number of links in each.

Rules per bot

One row per bot, with the rule your robots.txt gives it for /, and Blocked on most-crawled pages: of the host’s pages with the most bot hits in the date range, up to 100, how many your robots.txt blocks for this bot. A bot that is allowed on / can still be blocked on the folders that matter.

Changes

Every state the three files took, newest first. For robots.txt, a change lists the bots whose rule moved, such as GPTBot from Allowed to Blocked.
If a bot protection blocks our check of robots.txt, the rules on this page come from the last copy MentionLab could read. Our check identifies itself as MentionLabBot/1.0 (+https://mentionlab.io/bot).

Good to know

These pages read your site’s logs, which hold no prompt, no engine and no market. The global filters other than the date do not apply. The Pages page has its own Engines pick for the citations.
The Pages page and a bot’s drawer cover 92 days at most. AI visitors covers 366 days at most. A chart shows hours only for a range of 7 days or less. Pick a shorter range if a view says so.
A blocked hit is a hit answered 401, 402, 403 or 429. The logs hold the answer Cloudflare sent, so the figure cannot tell a block by a Cloudflare rule from a refusal by your own site.
Cited answers, Citation rate and State come from the answers MentionLab collected for your prompts, not from all of AI. With no answer in the date range, they show -: nothing was measured, so the page is neither cited nor not cited. Files for bots, such as robots.txt and llms.txt, always show -.
Requests for images, fonts, video, CSS and JavaScript are left out, with both modes. The figures are about pages.
A Logpush job sends every request of the zone except static files, and MentionLab decides which ones are AI traffic. The Worker decides on your zone, from the user agent and the referrer, and sends only those lines. Both modes cover every bot and every AI app MentionLab knows.
Workers Free runs 100,000 requests a day. Past that, with the route set to Fail open, your site keeps working, but no log line reaches MentionLab until midnight UTC. A site with more traffic needs Logpush or a paid Workers plan.
MentionLab does not keep your Cloudflare API token. The ingest token shows once: if you lose it, use Rotate token. The old token stops at once. With the Worker, set the new one as MENTIONLAB_INGEST_TOKEN. With Logpush, use Connect Cloudflare again within 24 hours, or Cloudflare disables the job.
A paused source takes no logs, and the daily check of its files stops. Check access now still works. When you delete a source, the logs already stored stay, and MentionLab refuses new ones. For a Logpush source, also delete its job in the Cloudflare dashboard.
The log lines of your normal visitors are dropped when they arrive: only their count is kept, by reason. For a bot hit, MentionLab stores the request with its IP address and its user agent, to identify the bot and to verify it. For an AI visit, it stores no IP address and no user agent.
A CDN other than Cloudflare, the import of older logs, an export of these figures, and alerts.