Beta. This feature is not on by default. To turn it on for your organization, contact
[email protected].
The pages
Key terms
Every bot has one of three categories, with the same color on every page:
MentionLab knows about 90 bots and 22 AI apps. A bot outside this list, such as an SEO tool or an
uptime monitor, is not counted.
Set up
You connect one log source per Cloudflare zone. A source sends the zone’s logs to MentionLab in one of two modes:Cloudflare offers Logpush as self-service on its Free, Pro and Business plans. On an Enterprise
plan it depends on your contract. See
Cloudflare’s Logpush documentation.
1
Add a log source
Open Settings → Project → Log sources and click Add a log source.
- Name: a label for you, such as “Main website”.
- Hostnames: the hosts of the zone, one per line, up to 20. Each one covers its subdomains:
brand.comcoversshop.brand.com. - Mode: Logpush or Worker.
2
Connect Cloudflare
- Logpush
- Worker
MentionLab creates the Logpush job with a Cloudflare API token that you make for this one use:
- In Cloudflare, open My Profile → API Tokens, then Create Token → Create Custom Token.
- Add a zone policy: Include, Specific zone, then the zone of your hostnames. Set a TTL that ends soon, such as tomorrow.
- In that policy, tick two permissions: Zone → Read under DNS & Zones, and Logs → Read and Edit under Analytics & Logs.
- Create the token, paste it in MentionLab and click Connect.
3
Check that the logs arrive
Back on Log sources, the row of your source shows a Last event time and a count under
Rows kept, 7 days once the first bot reads your site. A Logpush source that has no job yet
shows Not connected under its mode.
The data starts when you connect the source. MentionLab cannot read the logs from before that
moment.
Log sources
Settings → Project → Log sources lists your sources, with their health.
Most log lines are dropped, and that is normal: a Logpush job sends every request of your zone, and
Agent analytics keeps only the AI bots and the AI visits.
The menu of a source holds Edit source, Connect Cloudflare, Check access now,
Rotate token and Delete source.
Rows this month
The card above the list shows the rows your organization stored this calendar month (UTC), against its limit. A hit and a visit each count as one row, and every source of the organization shares the limit.Overview
Agent analytics → Traffic → Overview sums up the date range.
Below the figures:
- Hits over time, split by Category, Operator, Bot or Status.
- Operators: the companies behind the bots, each with its hits and its number of bots.
- Top fetched pages: the pages AI agents fetched most for people.
Bots
Agent analytics → Traffic → Bots has one row per bot that hit your site.
The filters above the table narrow every figure: Hosts, Operators, Bots, Categories,
Verification and Status. The Overview and the Pages page have the same ones.
Click a row to open the bot’s drawer: its hits over time, its response times (average, median and
95th percentile), its mix of HTTP statuses, the 25 pages it hit most, and its spoofing addresses.
The robots.txt column reads one host. Pick exactly one host in the Hosts filter to fill
it.
Verified and spoofed
Anyone can send a request that claims to be GPTBot. MentionLab checks each hit:
By default, every figure counts the verified hits only. To add the hits that nothing could check,
pick Not checked in the Verification filter. To see the fakes, pick Spoofed, or open
the bot’s drawer.
Pages
Agent analytics → Traffic → Pages puts two things side by side for each page: how bots read it in your logs, and how AI answers cite it in your tracked prompts. Switch between Pages and Folders above the table.
Beside the bot filters, this page has Folders, a path search (Starts with or Contains)
and Engines. Engines narrows the citations to the engines you pick, and the hits to the
operators of those engines.
Click a row to open the page’s drawer: its hits over time per operator, its mix of HTTP statuses,
and its citations per engine.
One page is one row, whatever the form of its address.
www.brand.com/Blog/ and
brand.com/blog?utm_source=x count as the same page.AI visitors
Agent analytics → Traffic → AI visitors counts people, not bots. A visit is one landing on a page of your site from a link in an AI app’s answer. MentionLab finds it from the referrer of the request, or from theutm_source=chatgpt.com tag that ChatGPT adds to its links.
The page shows the total of visits, the AI apps that sent the most, the visits over time per AI app,
and a table that you can read by AI app, Landing page, Country or Device. Its
filters are Hosts, AI apps, Countries and Devices.
An AI visit is a lower bound. An AI app that opens links without a referrer, or a person who
copies the address, is not counted.
Access
Agent analytics → Access shows what your site tells AI bots. Pick a host at the top.Files
MentionLab reads three files on each host once a day:robots.txt, llms.txt and llms-full.txt.
Check now asks for a new read, and the result shows within a minute. A host can be checked once
a minute at most.
When your
llms.txt is found, a card shows what it says: its title, its summary, and its sections
with the number of links in each.
Rules per bot
One row per bot, with the rule your robots.txt gives it for/, and Blocked on most-crawled
pages: of the host’s pages with the most bot hits in the date range, up to 100, how many your
robots.txt blocks for this bot. A bot that is allowed on / can still be blocked on the folders
that matter.
Changes
Every state the three files took, newest first. For robots.txt, a change lists the bots whose rule moved, such as GPTBot from Allowed to Blocked.Good to know
Only the date filter applies
Only the date filter applies
These pages read your site’s logs, which hold no prompt, no engine and no market. The global
filters other than the date do not apply. The Pages page has its own Engines pick for the
citations.
Some views limit the date range
Some views limit the date range
The Pages page and a bot’s drawer cover 92 days at most. AI visitors covers 366 days at most.
A chart shows hours only for a range of 7 days or less. Pick a shorter range if a view says so.
Blocked hits do not say who blocked
Blocked hits do not say who blocked
A blocked hit is a hit answered 401, 402, 403 or 429. The logs hold the answer Cloudflare sent,
so the figure cannot tell a block by a Cloudflare rule from a refusal by your own site.
Citations cover your tracked prompts only
Citations cover your tracked prompts only
Cited answers, Citation rate and State come from the answers MentionLab collected
for your prompts, not from all of AI. With no answer in the date range, they show
-: nothing
was measured, so the page is neither cited nor not cited. Files for bots, such as robots.txt and
llms.txt, always show -.Static files are not counted
Static files are not counted
Requests for images, fonts, video, CSS and JavaScript are left out, with both modes. The figures
are about pages.
Logpush and Worker do not send the same lines
Logpush and Worker do not send the same lines
A Logpush job sends every request of the zone except static files, and MentionLab decides which
ones are AI traffic. The Worker decides on your zone, from the user agent and the referrer, and
sends only those lines. Both modes cover every bot and every AI app MentionLab knows.
The Worker on Cloudflare's Workers Free plan
The Worker on Cloudflare's Workers Free plan
Workers Free runs 100,000 requests a day. Past that, with the route set to Fail open, your
site keeps working, but no log line reaches MentionLab until midnight UTC. A site with more
traffic needs Logpush or a paid Workers plan.
Tokens
Tokens
MentionLab does not keep your Cloudflare API token. The ingest token shows once: if you lose it,
use Rotate token. The old token stops at once. With the Worker, set the new one as
MENTIONLAB_INGEST_TOKEN. With Logpush, use Connect Cloudflare again within 24 hours, or
Cloudflare disables the job.Pausing and deleting a source
Pausing and deleting a source
A paused source takes no logs, and the daily check of its files stops. Check access now still
works. When you delete a source, the logs already stored stay, and MentionLab refuses new ones.
For a Logpush source, also delete its job in the Cloudflare dashboard.
What MentionLab stores
What MentionLab stores
The log lines of your normal visitors are dropped when they arrive: only their count is kept, by
reason. For a bot hit, MentionLab stores the request with its IP address and its user agent, to
identify the bot and to verify it. For an AI visit, it stores no IP address and no user agent.
Not available yet
Not available yet
A CDN other than Cloudflare, the import of older logs, an export of these figures, and alerts.