> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mentionlab.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent analytics

> See which AI bots read your site, which pages they read, and which visits AI apps send you

The rest of MentionLab reads what AI engines **answer**. Agent analytics reads what they **do on
your site**: it takes your site's request logs from Cloudflare and shows the AI bots that read your
pages, and the people that AI apps send to you.

> [!NOTE]
>
> **Beta.** This feature is not on by default. To turn it on for your organization, contact
> [sales@mentionlab.io](mailto:sales@mentionlab.io).

Agent analytics needs a site behind **Cloudflare**. If the side menu has no **Agent analytics**
group, the feature is not on for your organization.

## The pages

| Where | What it shows |
| - | - |
| **Agent analytics → Traffic → Overview** | The headline figures, the hits over time, the companies behind the bots, and the pages AI agents fetch most |
| **Agent analytics → Traffic → Bots** | One row per bot: how often it reads your site, how well, and what your robots.txt tells it |
| **Agent analytics → Traffic → Pages** | One row per page or folder: the bot hits beside the citations in your tracked prompts |
| **Agent analytics → Traffic → AI visitors** | The people who came from a link in an AI answer: per AI app, landing page, country and device |
| **Agent analytics → Access** | Your robots.txt and llms.txt files as a bot finds them, the rule for each bot, and every change |
| **Settings → Project → Log sources** | Where you connect your Cloudflare zone and check that the logs arrive |

## Key terms

| Term | Meaning |
| - | - |
| **Hit** | One request of a known bot to your site |
| **Crawl** | A hit of a crawler: it collects pages ahead of time |
| **Fetch** | A hit of an AI agent: it reads a page now, for a person who asked an AI app |
| **AI visit** | A person who lands on your site from a link in an AI app's answer |
| **Operator** | The company that runs a bot, such as OpenAI for GPTBot and ChatGPT-User |

Every bot has one of three categories, with the same color on every page:

| Category | What the bot does | Examples |
| - | - | - |
| **AI agent** | Fetches a page now, for a person who asked an AI app | ChatGPT-User, Claude-User, Perplexity-User |
| **AI crawler** | Collects pages ahead of time for an AI company | GPTBot, ClaudeBot, PerplexityBot |
| **Search crawler** | Indexes pages for a search engine | Googlebot, Bingbot, Applebot |

MentionLab knows about 90 bots and 22 AI apps. A bot outside this list, such as an SEO tool or an
uptime monitor, is not counted.

***

## Set up

You connect one **log source** per Cloudflare zone. A source sends the zone's logs to MentionLab in
one of two modes:

| Mode | How it works | Use it when |
| - | - | - |
| **Logpush** | Cloudflare sends the zone's logs. MentionLab creates the job for you | Your zone can use Logpush. This is the mode to prefer |
| **Worker** | A small Worker on your zone sends the log lines | Your zone cannot use Logpush |

> [!NOTE]
>
> Cloudflare offers Logpush as self-service on its Free, Pro and Business plans. On an Enterprise
> plan it depends on your contract. See
> [Cloudflare's Logpush documentation](https://developers.cloudflare.com/logs/logpush/).

**1. Add a log source**

Open **Settings → Project → Log sources** and click **Add a log source**.

* **Name:** a label for you, such as "Main website".
* **Hostnames:** the hosts of the zone, one per line, up to 20. Each one covers its subdomains:
  `brand.com` covers `shop.brand.com`.
* **Mode:** Logpush or Worker.

MentionLab then shows an **ingest URL** and an **ingest token**.

> [!WARNING]
>
> The ingest token shows once. With the Worker, copy both values now. With Logpush you do not
> need them: the next step creates the job with a token of its own.

**2. Connect Cloudflare**

> [tab] Logpush
>
> MentionLab creates the Logpush job with a Cloudflare API token that you make for this one use:
>
> 1. In Cloudflare, open **My Profile → API Tokens**, then **Create Token → Create Custom Token**.
> 2. Add a zone policy: **Include**, **Specific zone**, then the zone of your hostnames. Set a TTL
>    that ends soon, such as tomorrow.
> 3. In that policy, tick two permissions: **Zone → Read** under **DNS & Zones**, and
>    **Logs → Read** and **Edit** under **Analytics & Logs**.
> 4. Create the token, paste it in MentionLab and click **Connect**.
>
> MentionLab uses the token once to create the job, and does not keep it.

> [tab] Worker
>
> 1. In Cloudflare, open **Workers & Pages → Create → Worker** and paste the Worker script as its
>    code. The setup window has a **Copy script** button. You can also deploy it with wrangler.
> 2. In the Worker's **Settings → Variables and Secrets**, add `MENTIONLAB_INGEST_URL` as a
>    variable and `MENTIONLAB_INGEST_TOKEN` as a secret, with the two values of the first step.
> 3. On the zone, add a route that sends your site's traffic to the Worker, such as
>    `brand.com/*` (**Settings → Domains & Routes**). Set its failure mode to
>    **Fail open (proceed)**.
>
> The Worker passes every request to your site unchanged. It sends a log line to MentionLab only
> when the request looks like an AI bot or a visit from an AI app, and only after your visitor
> has the answer. If the Worker fails, Cloudflare sends the request to your site as if no Worker
> ran.

**3. Check that the logs arrive**

Back on **Log sources**, the row of your source shows a **Last event** time and a count under
**Rows kept, 7 days** once the first bot reads your site. A Logpush source that has no job yet
shows **Not connected** under its mode.

> [!NOTE]
>
> The data starts when you connect the source. MentionLab cannot read the logs from before that
> moment.

***

## Log sources

**Settings → Project → Log sources** lists your sources, with their health.

| Column | Meaning |
| - | - |
| **Mode** | Logpush or Worker |
| **State** | **Enabled** or **Paused**. A paused source takes no logs |
| **Last event** | When the last log line arrived |
| **Rows kept, 7 days** | The bot hits and AI visits stored in the last 7 days |
| **Rows dropped, 7 days** | The log lines not stored, with the reason for each |

Most log lines are dropped, and that is normal: a Logpush job sends every request of your zone, and
Agent analytics keeps only the AI bots and the AI visits.

| Reason | What was dropped |
| - | - |
| **people** | Requests of your normal visitors |
| **other bots** | Bots outside the list, such as SEO tools and monitors |
| **assets** | Static files, such as images, fonts, CSS and JavaScript |
| **host not listed** | Requests for a host that is not in the source's hostnames |
| **unreadable or too old** | A line that could not be read, or one older than 30 days |
| **the Worker's own requests** | Requests your own Workers made |
| **over quota** | Lines that arrived after the month's limit was reached |
| **paused** | Lines that arrived while the source was paused |

The menu of a source holds **Edit source**, **Connect Cloudflare**, **Check access now**,
**Rotate token** and **Delete source**.

### Rows this month

The card above the list shows the rows your organization stored this calendar month (UTC), against
its limit. A hit and a visit each count as one row, and every source of the organization shares the
limit.

> [!WARNING]
>
> Past the limit, new rows are dropped until the 1st of the next month. The organization's admins
> get an email at 80% and at 100% of the limit.

***

## Overview

**Agent analytics → Traffic → Overview** sums up the date range.

| Figure | Meaning |
| - | - |
| **Bot hits** | The hits of known bots, with the change against the comparison period |
| **Crawls and fetches** | The hits split between crawlers and AI agents. Crawls plus fetches equal the hits |
| **AI visits** | The people AI apps sent you |
| **Pages read** | Distinct pages that answered a bot with success (a 2xx status) |
| **Error rate** | The share of hits answered with a 4xx or 5xx status |
| **Blocked hits** | The hits answered 401, 402, 403 or 429 |
| **Avg. response time** | The mean time to first byte of the hits |

Below the figures:

* **Hits over time**, split by **Category**, **Operator**, **Bot** or **Status**.
* **Operators:** the companies behind the bots, each with its hits and its number of bots.
* **Top fetched pages:** the pages AI agents fetched most for people.

## Bots

**Agent analytics → Traffic → Bots** has one row per bot that hit your site.

| Column | Meaning |
| - | - |
| **Hits** | The bot's hits in the date range |
| **Pages** | Distinct pages the bot hit. This is an estimate, within about 1% |
| **Verified** | The share of hits whose identity was proved, by Cloudflare or by an address of the operator. It reads 100% until you add other hits with the **Verification** filter |
| **Error rate** | The share of hits answered with a 4xx or 5xx status |
| **Blocked hits** | The hits answered 401, 402, 403 or 429 |
| **Last seen** | The bot's last hit |
| **robots.txt** | **Allowed** or **Blocked**: what your robots.txt tells this bot for `/` |

The filters above the table narrow every figure: **Hosts**, **Operators**, **Bots**, **Categories**,
**Verification** and **Status**. The Overview and the Pages page have the same ones.

Click a row to open the bot's drawer: its hits over time, its response times (average, median and
95th percentile), its mix of HTTP statuses, the 25 pages it hit most, and its spoofing addresses.

> [!NOTE]
>
> The **robots.txt** column reads one host. Pick exactly one host in the **Hosts** filter to fill
> it.

### Verified and spoofed

Anyone can send a request that claims to be GPTBot. MentionLab checks each hit:

| Verification | Meaning |
| - | - |
| **Verified by Cloudflare** | Cloudflare confirmed the bot |
| **Verified by address** | The request came from an address of the bot's operator: one it publishes, its own network, or one its DNS names |
| **Not checked** | Nothing can prove it or disprove it: the operator publishes no address list and no DNS name, or the agent runs on a person's own computer |
| **Spoofed** | A check says the bot's name is false: the address is outside its operator's published list or network, its DNS name is not the operator's, or, in the last 24 hours, the same address failed a check or claimed the bots of several operators |

By default, every figure counts the verified hits only. To add the hits that nothing could check,
pick **Not checked** in the **Verification** filter. To see the fakes, pick **Spoofed**, or open
the bot's drawer.

> [!TIP]
>
> Most bots have no published way to be verified, and a coding agent runs on a person's own
> computer. Their hits read "Not checked", so these bots show only when you pick **Not checked**.
> A bot that then reads **0% verified** is not a spoofed bot.

## Pages

**Agent analytics → Traffic → Pages** puts two things side by side for each page: how bots read it in your
logs, and how AI answers cite it in your tracked prompts. Switch between **Pages** and **Folders**
above the table.

| Column | Meaning |
| - | - |
| **In your logs** | The page's crawls and fetches |
| **Last hit** | When a bot last hit the page, and the HTTP status the page answered |
| **Cited answers** | The answers to your tracked prompts that cite the page |
| **Citation rate** | Cited answers over all the answers in the date range |
| **AI visits** | The people AI apps sent to this page |
| **State** | How AI uses the page, in one label |

| State | Meaning |
| - | - |
| **Read and cited** | Bots read the page, and tracked answers cite it |
| **Fetched, not cited** | AI agents fetched it for people, and no tracked answer cites it. Consider tracking new prompts |
| **Read, not cited** | Only crawlers read it, and no tracked answer cites it |

Beside the bot filters, this page has **Folders**, a path search (**Starts with** or **Contains**)
and **Engines**. **Engines** narrows the citations to the engines you pick, and the hits to the
operators of those engines.

Click a row to open the page's drawer: its hits over time per operator, its mix of HTTP statuses,
and its citations per engine.

> [!NOTE]
>
> One page is one row, whatever the form of its address. `www.brand.com/Blog/` and
> `brand.com/blog?utm_source=x` count as the same page.

## AI visitors

**Agent analytics → Traffic → AI visitors** counts people, not bots. A visit is one landing on a page of your
site from a link in an AI app's answer. MentionLab finds it from the referrer of the request, or
from the `utm_source=chatgpt.com` tag that ChatGPT adds to its links.

The page shows the total of visits, the AI apps that sent the most, the visits over time per AI app,
and a table that you can read by **AI app**, **Landing page**, **Country** or **Device**. Its
filters are **Hosts**, **AI apps**, **Countries** and **Devices**.

> [!NOTE]
>
> An AI visit is a lower bound. An AI app that opens links without a referrer, or a person who
> copies the address, is not counted.

## Access

**Agent analytics → Access** shows what your site tells AI bots. Pick a host at the top.

### Files

MentionLab reads three files on each host once a day: `robots.txt`, `llms.txt` and `llms-full.txt`.
**Check now** asks for a new read, and the result shows within a minute. A host can be checked once
a minute at most.

| Outcome | Meaning |
| - | - |
| **Found** | The file was read |
| **No file** | The host has no such file. For robots.txt this means bots may crawl everything |
| **Unreachable** | A server error, a timeout or a network error. For robots.txt this means bots must treat the whole site as blocked |
| **Our check was blocked** | A bot protection answered in place of the file, so MentionLab cannot tell what it says |
| **Returns a page, not a file** | The address answers with a web page, often a catch-all home page |
| **Not checked yet** | The first check has not run |

When your `llms.txt` is found, a card shows what it says: its title, its summary, and its sections
with the number of links in each.

### Rules per bot

One row per bot, with the rule your robots.txt gives it for `/`, and **Blocked on most-crawled
pages**: of the host's pages with the most bot hits in the date range, up to 100, how many your
robots.txt blocks for this bot. A bot that is allowed on `/` can still be blocked on the folders
that matter.

### Changes

Every state the three files took, newest first. For robots.txt, a change lists the bots whose rule
moved, such as GPTBot from **Allowed** to **Blocked**.

> [!WARNING]
>
> If a bot protection blocks our check of robots.txt, the rules on this page come from the last
> copy MentionLab could read. Our check identifies itself as
> `MentionLabBot/1.0 (+https://mentionlab.io/bot)`.

***

## Good to know

> [accordion] Only the date filter applies
>
> These pages read your site's logs, which hold no prompt, no engine and no market. The global
> filters other than the date do not apply. The Pages page has its own **Engines** pick for the
> citations.

> [accordion] Some views limit the date range
>
> The Pages page and a bot's drawer cover 92 days at most. AI visitors covers 366 days at most.
> A chart shows hours only for a range of 7 days or less. Pick a shorter range if a view says so.

> [accordion] Blocked hits do not say who blocked
>
> A blocked hit is a hit answered 401, 402, 403 or 429. The logs hold the answer Cloudflare sent,
> so the figure cannot tell a block by a Cloudflare rule from a refusal by your own site.

> [accordion] Citations cover your tracked prompts only
>
> **Cited answers**, **Citation rate** and **State** come from the answers MentionLab collected
> for your prompts, not from all of AI. With no answer in the date range, they show `-`: nothing
> was measured, so the page is neither cited nor not cited. Files for bots, such as robots.txt and
> llms.txt, always show `-`.

> [accordion] Static files are not counted
>
> Requests for images, fonts, video, CSS and JavaScript are left out, with both modes. The figures
> are about pages.

> [accordion] Logpush and Worker do not send the same lines
>
> A Logpush job sends every request of the zone except static files, and MentionLab decides which
> ones are AI traffic. The Worker decides on your zone, from the user agent and the referrer, and
> sends only those lines. Both modes cover every bot and every AI app MentionLab knows.

> [accordion] The Worker on Cloudflare's Workers Free plan
>
> Workers Free runs 100,000 requests a day. Past that, with the route set to **Fail open**, your
> site keeps working, but no log line reaches MentionLab until midnight UTC. A site with more
> traffic needs Logpush or a paid Workers plan.

> [accordion] Tokens
>
> MentionLab does not keep your Cloudflare API token. The ingest token shows once: if you lose it,
> use **Rotate token**. The old token stops at once. With the Worker, set the new one as
> `MENTIONLAB_INGEST_TOKEN`. With Logpush, use **Connect Cloudflare** again within 24 hours, or
> Cloudflare disables the job.

> [accordion] Pausing and deleting a source
>
> A paused source takes no logs, and the daily check of its files stops. **Check access now** still
> works. When you delete a source, the logs already stored stay, and MentionLab refuses new ones.
> For a Logpush source, also delete its job in the Cloudflare dashboard.

> [accordion] What MentionLab stores
>
> The log lines of your normal visitors are dropped when they arrive: only their count is kept, by
> reason. For a bot hit, MentionLab stores the request with its IP address and its user agent, to
> identify the bot and to verify it. For an AI visit, it stores no IP address and no user agent.

> [accordion] Not available yet
>
> A CDN other than Cloudflare, the import of older logs, an export of these figures, and alerts.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.