Crawlability

Crawlability answers: can the engines’ crawlers read your site at all? It reads the robots.txt of your site, and of the competitor sites you add, every day, and shows what each AI crawler is allowed to do. Nothing but the file is fetched, and the screen claims only what the file says.

​

Before you start

Crawlability needs your site’s domain. Add it, and optionally a domain for each competitor you track, under Domains in Project settings. Saving queues a check.

​

The toolbar

Run again reads every site’s robots.txt again; the result arrives in a minute or two. Owners, admins and editors can run it.

​

What is on it, top to bottom

​

Domains

Your site first, then your competitors’ sites. Each shows its worst status. Pick one to see it below, with when it was last checked and when its robots.txt last changed.

​

What robots.txt allows

For your own site, a sentence says whether its robots.txt blocks any crawler that feeds an engine you track. Each blocked crawler gets a card: the engines it feeds, how often those engines cited your site in the window, the line in robots.txt that blocks it, a suggested replacement, and, for a search or user-fetch crawler, Create work item, which adds a technical fix to Gaps with the rule, its line and the replacement.

A block at your CDN or firewall doesn’t show in robots.txt.

For a competitor’s site, the panel compares it with yours, and its rows read “Competitor’s site · nothing to do”.

​

Crawlers

Every crawler, in three groups:

• Feeds engines you track: the ones that matter most.
• Other search and assistant crawlers.
• Training crawlers, collapsed. Blocking training crawlers doesn’t stop engines reading your pages when they answer.

Each row: the crawler, its vendor, what it is for (Training, Search or User fetch), the engines it feeds, its status, and why. Filter by Vendor and Status. The why names the rule: the crawler’s own rule and its line, the rules for every crawler (*) that it follows, or no rule applying. A block on Googlebot is marked severe: it removes the site from Google search, not only from AI Overviews.

​

Test a path

Enter a path, or a full address on that site, and see which crawlers can read it and which rule decides. It is checked against the stored file; nothing is fetched.

​

The file

The robots.txt as it was read, with line numbers and the blocking lines marked, the address it was served from if a redirect moved it, and when it last changed.

​

Domain states

• Checking now: the file is being read for the first time; results in a minute.
• No robots.txt: the site answers “not found”, so every crawler may read the site.
• Didn’t load: the file failed to load. Crawlers treat that as everything blocked, so while it fails, a crawler that respects robots.txt reads nothing on the site. The day it last loaded is shown, with Run again.
• Can’t be checked: the domain resolves to a private or reserved address, or isn’t a public host, so nothing was fetched. Correct the spelling in Project settings and it is checked again.
• No domain yet: “Add your site’s domain in Project settings to check it”, with Open Project settings.

​

Related