> ## Documentation Index
> Fetch the complete documentation index at: https://docs.37audits.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Robots Crawl Delay
> Checks whether robots.txt declares a Crawl-delay directive that tells well-behaved crawlers how fast they may request pages.
**SEO Auditor** — fetches the site's `robots.txt` and checks that it declares a `Crawl-delay` directive with a numeric value, so crawlers that honor it space their requests.
Keeps aggressive crawlers from requesting pages faster than the site can serve them to real visitors.
Reads only `/robots.txt` on the audited host; the audited page itself is never requested.
`Crawl-delay` is an informal `robots.txt` directive that asks a crawler to wait a number of seconds between two requests to the site. Bing, Yandex and many smaller crawlers honor it; Google ignores it and takes its crawl rate from Search Console instead. RFC 9309 (the Robots Exclusion Protocol) does not define the directive, so this auditor has no specification pin. It reports whether the directive is there and whether its value is a number crawlers can use.
## How it works
1. **Build the robots URL** — takes the scheme, host and port of the audited URL and appends `/robots.txt`. An audited URL without a scheme or host ends the audit with `500`.
2. **Fetch** — requests `robots.txt` as 37AuditsBot, following redirects. This is a probe of another path, not the audited page, so a 403 is a finding (`400`), not a blocked bot: this auditor never emits `590`. A network error or an HTTP 5xx ends the audit with `500`.
3. **Parse** — when the answer is HTTP 200, scans the body line by line for `Crawl-delay:` (any letter case, surrounding whitespace allowed). The first matching line wins. Its value is valid when it is a non-negative integer or decimal such as `5` or `0.5`.
4. **Rules** — any status other than 200 (and below 500) fails (`400`); a 200 body with no `Crawl-delay` warns (`300`); a `Crawl-delay` whose value is empty or not a number warns (`301`).
5. **Aggregate** — FAIL wins over WARNING; when every rule passes the audit is `SUCCESS` (`200`) and the message quotes the declared value.
## What it audits
Every row maps to one `checkRule*` method and one `CheckCode`, in the order the auditor runs them. CheckCode suffixes use status ranges: `200` success, `300–399` warning, `400–499` fail, `500–599` error.
| CheckCode | Status | When it fires | Recommendation | Why it matters |
| - | - | - | - | - |
| `37A-RobotsCrawlDelayAuditor-400` | `FAIL` | `robots.txt` answered an HTTP status other than 200 and below 500 (404, 403, 401, a 3xx left after redirects…). | Publish a robots.txt file at `{robotsUrl}` that answers HTTP 200 (it currently answers HTTP `{httpStatus}`) and declare a Crawl-delay directive in it. | Without a reachable robots.txt, crawlers get no crawl-rate or exclusion guidance, so they may hit the site harder and index pages that should stay out of search results. |
| `37A-RobotsCrawlDelayAuditor-300` | `WARNING` | `robots.txt` was fetched (HTTP 200) but contains no `Crawl-delay` directive. | Add a "Crawl-delay: \" line (for example "Crawl-delay: 10") to `{robotsUrl}` under the User-agent groups whose crawl rate you want to limit. | Without a Crawl-delay, crawlers that honor it request pages as fast as they like, which can slow the site for real visitors and inflate hosting costs. |
| `37A-RobotsCrawlDelayAuditor-301` | `WARNING` | A `Crawl-delay` directive is present but its value is empty or not a non-negative number. | Replace "Crawl-delay: `{crawlDelay}`" in `{robotsUrl}` with a number of seconds, for example "Crawl-delay: 10". | Crawlers ignore a Crawl-delay they cannot read as a number of seconds, so the site gets none of the rate limiting the directive was meant to provide. |
| `37A-RobotsCrawlDelayAuditor-200` | `SUCCESS` | All rules passed: `robots.txt` declares a `Crawl-delay` with a numeric value. Message quotes the value and the robots.txt URL. | — | — |
| `37A-RobotsCrawlDelayAuditor-500` | `ERROR` | `robots.txt` could not be fetched: a network error, an audited URL without a scheme or host, or an HTTP 5xx answer. | Make sure `{robotsUrl}` is reachable over HTTP(S) and answers without a server error, then run the audit again. Last error: `{error}`. | — |
## Output documentation
| Status | Description | Test logic |
| - | - | - |
| **SUCCESS** | robots.txt declares a usable Crawl-delay | HTTP 200 and the first `Crawl-delay` line has a non-negative numeric value (`200`). |
| **WARNING** | Crawlers get no usable Crawl-delay | No `Crawl-delay` directive (`300`); a `Crawl-delay` whose value is empty or not a number (`301`). |
| **FAIL** | robots.txt is not available | `robots.txt` answered a status other than 200 and below 500, such as 404 or 403 (`400`). |
| **ERROR** | The auditor could not fetch robots.txt | Network error, invalid audited URL or HTTP 5xx on `robots.txt` (`500`). |
Every check carries `robotsUrl`, `httpStatus`, `crawlDelayFound`, `crawlDelay` (the raw value, or `null`) and `crawlDelaySeconds` (the parsed number, or `null`) in its `data`, and its `resource` is the `robots.txt` URL.
## Risks and considerations
### Which crawlers honor it
* `Crawl-delay` is not part of RFC 9309. Bing, Yandex, DuckDuckBot and many SEO and AI crawlers honor it; Googlebot ignores it, so it is not a substitute for rate limiting at the server or CDN.
* A very large value (for example `30`) can keep Bing from crawling a large site fully; values between 1 and 10 seconds are typical.
### Which line counts
* The first `Crawl-delay` line in the file wins, whichever `User-agent` group it sits in. A file that sets different delays per crawler is reported from its first one.
* Only `/robots.txt` on the audited scheme, host and port is read. A `robots.txt` on another subdomain or served only over the other scheme is not seen.
### Behavior changes from the previous version
* An HTTP 5xx answer on `robots.txt` used to be reported as `FAIL`; it is now `ERROR` (`500`), because a server error says nothing about whether the file declares a Crawl-delay.
* A malformed line such as `Crawl-delay` without a colon or `Crawl-delay: fast` used to crash the audit; it is now a `WARNING` (`301`).
* A `robots.txt` that answers 404, 403 or another status below 500 still fails (`400`), now with the observed status in the message.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.