> ## Documentation Index
> Fetch the complete documentation index at: https://docs.citino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawlability

> Which AI crawlers your site’s robots.txt lets in, which engines each one feeds, and what to change when one is blocked.

Crawlability answers: can the engines’ crawlers read your site at all? It reads the `robots.txt` of your site, and of the competitor sites you add, every day, and shows what each AI crawler is allowed to do. Nothing but the file is fetched, and the screen claims only what the file says.

## Before you start

Crawlability needs your site’s domain. Add it, and optionally a domain for each competitor you track, under **Domains** in [Project settings](https://docs.citino.ai/guides/screens/project-settings.md). Saving queues a check.

## The toolbar

**Run again** reads every site’s `robots.txt` again; the result arrives in a minute or two. Owners, admins and editors can run it.

## What is on it, top to bottom

### Domains

Your site first, then your competitors’ sites. Each shows its worst status. Pick one to see it below, with when it was last checked and when its `robots.txt` last changed.

### What robots.txt allows

For your own site, a sentence says whether its `robots.txt` blocks any crawler that feeds an engine you track. Each blocked crawler gets a card: the engines it feeds, how often those engines cited your site in the window, the line in `robots.txt` that blocks it, a suggested replacement, and, for a search or user-fetch crawler, **Create work item**, which adds a technical fix to [Gaps](https://docs.citino.ai/guides/screens/gaps.md) with the rule, its line and the replacement.

> A block at your CDN or firewall doesn't show in robots.txt.

For a competitor’s site, the panel compares it with yours, and its rows read "Competitor’s site · nothing to do".

### Crawlers

Every crawler, in three groups:

* **Feeds engines you track**: the ones that matter most.
* **Other search and assistant crawlers**.
* **Training crawlers**, collapsed. Blocking training crawlers doesn't stop engines reading your pages when they answer.

Each row: the crawler, its vendor, what it is for (**Training**, **Search** or **User fetch**), the engines it feeds, its status, and why. Filter by **Vendor** and **Status**.

| Status | What it means |
| - | - |
| Allowed | The crawler may read the site. |
| Partial | Some paths are closed to it. A Partial that closes only a private path or non-page files (images, code, text) reads as open; one that closes documents says so. |
| Blocked | The crawler may not read the site. |
| Too complex | The rules could not be evaluated. |

The **why** names the rule: the crawler’s own rule and its line, the rules for every crawler (`*`) that it follows, or no rule applying. A block on Googlebot is marked severe: it removes the site from Google search, not only from AI Overviews.

### Test a path

Enter a path, or a full address on that site, and see which crawlers can read it and which rule decides. It is checked against the stored file; nothing is fetched.

### The file

The `robots.txt` as it was read, with line numbers and the blocking lines marked, the address it was served from if a redirect moved it, and when it last changed.

## Domain states

* **Checking now**: the file is being read for the first time; results in a minute.
* **No robots.txt**: the site answers "not found", so every crawler may read the site.
* **Didn’t load**: the file failed to load. Crawlers treat that as everything blocked, so while it fails, a crawler that respects `robots.txt` reads nothing on the site. The day it last loaded is shown, with **Run again**.
* **Can’t be checked**: the domain resolves to a private or reserved address, or isn’t a public host, so nothing was fetched. Correct the spelling in Project settings and it is checked again.
* **No domain yet**: "Add your site's domain in Project settings to check it", with **Open Project settings**.

## Related

> [card] [Crawler activity](https://docs.citino.ai/guides/screens/crawler-activity.md)
>
> Which crawlers actually fetched your pages.

> [card] [Project settings](https://docs.citino.ai/guides/screens/project-settings.md)
>
> Where the domains are set.

> [card] [Gaps](https://docs.citino.ai/guides/screens/gaps.md)
>
> Where a fix becomes a work item.

> [card] [Sources](https://docs.citino.ai/guides/screens/sources.md)
>
> How often the engines read your site.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.