Skip to main content
Crawler activity answers: which AI crawlers actually came to my site, and what did they fetch? Where Crawlability reads what your robots.txt allows, this screen shows what happened, from your site’s own request logs. It shows which AI crawlers fetched your pages. It cannot show that a fetch led to a citation: crawler visits sit beside the answers that read or cited a page, and are never combined with them.

​
Setting it up

Until a way in is connected, the screen is a setup panel. Pick one, or more than one: Only visits to your own domain are kept, and only from known AI crawlers. Owners, admins and editors can connect a way in.

​
What is on it, once set up

  • Filters: the window (7, 30 or 90 days), one bot or every bot, and Verified only.
  • Four figures: Bot visits, Active bots, Failure rate (responses in the 4xx and 5xx ranges) and Top page.
  • Visits over time, by vendor.
  • Bots: each crawler’s vendor and purpose (Training, Search or User fetch), its verified visits, its claimed, unverified visits in a column of their own, and when it was last seen. Verified means the address is the vendor’s; claimed means the user-agent says so and nothing confirms it. Some crawlers can’t be verified at all, and say so.
  • Pages: each path, its visits, the mix of response codes, when it was last fetched, and then the retrievals and citations from the same days’ answers. Those sit beside the visits and are never divided by them: a fetch is not a citation. Pages beyond the table’s cap are folded into Other pages.
  • Sources: each connected way in, when it last sent something, and this month’s visits against your plan’s cap. A way in that has sent nothing for 48 hours is flagged, so you can check it is still installed. Disconnect removes it.
  • Add another way in opens the setup panel again.
Your own pages on Sources also show Fetches and Last fetched once crawler logs are connected.

​
Deleting the data

Delete crawler data removes every crawler visit stored for the project, after a confirm. The ways in stay connected. It is for an owner or admin.

​
States

  • Not set up: the setup panel.
  • Waiting for the first request: connected, nothing received yet. Open a page on your site, or send a test; a crawler’s visit shows within a minute of arriving.
  • Monthly cap reached: visits after the day the cap was reached are not counted this month. See Plans and limits.
  • A way in that failed to install shows the vendor’s own reason.

Crawlability

What your robots.txt allows.

Integrations

Every connection a project can have.

Sources

Which of your pages the engines read.

What we do not measure

Why a fetch is not a citation.