> ## Documentation Index
> Fetch the complete documentation index at: https://docs.citino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler activity

> Which AI crawlers fetched which of your pages, when, and with what result, from your own site’s logs.

Crawler activity answers: which AI crawlers actually came to my site, and what did they fetch? Where [Crawlability](https://docs.citino.ai/guides/screens/crawlability.md) reads what your `robots.txt` allows, this screen shows what happened, from your site’s own request logs.

It shows which AI crawlers fetched your pages. It cannot show that a fetch led to a citation: crawler visits sit beside the answers that read or cited a page, and are never combined with them.

## Setting it up

Until a way in is connected, the screen is a setup panel. Pick one, or more than one:

| Way in | What it installs | What it needs |
| - | - | - |
| Cloudflare | A small Worker on your zone that sends Citino one line for each AI-crawler request, after the page is served. Every request to the zone runs it and counts against your Workers quota. | A Cloudflare API token with three permissions: Account › Workers Scripts › Edit, Zone › Zone › Read and Zone › Workers Routes › Edit. |
| Vercel | A log drain from every project in the Vercel team you pick. Vercel sends every request; Citino keeps AI crawlers on your domain and drops the rest on arrival. | A Vercel access token. Log drains need a Vercel Pro or Enterprise plan. |
| WordPress plugin | A plugin that sends one line for each AI-crawler request after the page is served, never blocking it. | Upload the zip under Plugins → Add New → Upload Plugin, then activate it. Downloading it again replaces its key. |
| Upload a log file | Reads a `.csv` or a combined-format `.log` in your browser and sends only its AI-crawler lines. The file itself is not uploaded. | A log file. Importing a file twice counts its visits twice. |
| Send from anything else | An address and a key for a CDN, proxy or script of your own, which sends the requests as JSON. | Somewhere that can send an HTTPS request with a bearer key. The key is shown once. |

Only visits to your own domain are kept, and only from known AI crawlers. Owners, admins and editors can connect a way in.

## What is on it, once set up

* **Filters**: the window (7, 30 or 90 days), one bot or every bot, and **Verified only**.
* **Four figures**: **Bot visits**, **Active bots**, **Failure rate** (responses in the 4xx and 5xx ranges) and **Top page**.
* **Visits over time**, by vendor.
* **Bots**: each crawler’s vendor and purpose (Training, Search or User fetch), its **verified visits**, its **claimed, unverified** visits in a column of their own, and when it was last seen. Verified means the address is the vendor’s; claimed means the user-agent says so and nothing confirms it. Some crawlers can’t be verified at all, and say so.
* **Pages**: each path, its visits, the mix of response codes, when it was last fetched, and then the retrievals and citations from the same days’ answers. Those sit beside the visits and are never divided by them: a fetch is not a citation. Pages beyond the table’s cap are folded into **Other pages**.
* **Sources**: each connected way in, when it last sent something, and this month’s visits against your plan’s cap. A way in that has sent nothing for 48 hours is flagged, so you can check it is still installed. **Disconnect** removes it.
* **Add another way in** opens the setup panel again.

Your own pages on [Sources](https://docs.citino.ai/guides/screens/sources.md) also show **Fetches** and **Last fetched** once crawler logs are connected.

## Deleting the data

**Delete crawler data** removes every crawler visit stored for the project, after a confirm. The ways in stay connected. It is for an owner or admin.

## States

* **Not set up**: the setup panel.
* **Waiting for the first request**: connected, nothing received yet. Open a page on your site, or send a test; a crawler’s visit shows within a minute of arriving.
* **Monthly cap reached**: visits after the day the cap was reached are not counted this month. See [Plans and limits](https://docs.citino.ai/guides/account/plans-and-limits.md).
* A way in that failed to install shows the vendor’s own reason.

## Related

> [card] [Crawlability](https://docs.citino.ai/guides/screens/crawlability.md)
>
> What your robots.txt allows.

> [card] [Integrations](https://docs.citino.ai/guides/account/integrations.md)
>
> Every connection a project can have.

> [card] [Sources](https://docs.citino.ai/guides/screens/sources.md)
>
> Which of your pages the engines read.

> [card] [What we do not measure](https://docs.citino.ai/guides/methodology/what-we-do-not-measure.md)
>
> Why a fetch is not a citation.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.