Skip to main content
SEO Auditor — fetches the site’s robots.txt and checks that it declares a Crawl-delay directive with a numeric value, so crawlers that honor it space their requests.

Protection

Keeps aggressive crawlers from requesting pages faster than the site can serve them to real visitors.

Scope

Reads only /robots.txt on the audited host; the audited page itself is never requested.
Crawl-delay is an informal robots.txt directive that asks a crawler to wait a number of seconds between two requests to the site. Bing, Yandex and many smaller crawlers honor it; Google ignores it and takes its crawl rate from Search Console instead. RFC 9309 (the Robots Exclusion Protocol) does not define the directive, so this auditor has no specification pin. It reports whether the directive is there and whether its value is a number crawlers can use.

​
How it works

  1. Build the robots URL — takes the scheme, host and port of the audited URL and appends /robots.txt. An audited URL without a scheme or host ends the audit with 500.
  2. Fetch — requests robots.txt as 37AuditsBot, following redirects. This is a probe of another path, not the audited page, so a 403 is a finding (400), not a blocked bot: this auditor never emits 590. A network error or an HTTP 5xx ends the audit with 500.
  3. Parse — when the answer is HTTP 200, scans the body line by line for Crawl-delay: (any letter case, surrounding whitespace allowed). The first matching line wins. Its value is valid when it is a non-negative integer or decimal such as 5 or 0.5.
  4. Rules — any status other than 200 (and below 500) fails (400); a 200 body with no Crawl-delay warns (300); a Crawl-delay whose value is empty or not a number warns (301).
  5. Aggregate — FAIL wins over WARNING; when every rule passes the audit is SUCCESS (200) and the message quotes the declared value.

​
What it audits

Every row maps to one checkRule* method and one CheckCode, in the order the auditor runs them. CheckCode suffixes use status ranges: 200 success, 300–399 warning, 400–499 fail, 500–599 error.

​
Output documentation

Every check carries robotsUrl, httpStatus, crawlDelayFound, crawlDelay (the raw value, or null) and crawlDelaySeconds (the parsed number, or null) in its data, and its resource is the robots.txt URL.

​
Risks and considerations

​
Which crawlers honor it

  • Crawl-delay is not part of RFC 9309. Bing, Yandex, DuckDuckBot and many SEO and AI crawlers honor it; Googlebot ignores it, so it is not a substitute for rate limiting at the server or CDN.
  • A very large value (for example 30) can keep Bing from crawling a large site fully; values between 1 and 10 seconds are typical.

​
Which line counts

  • The first Crawl-delay line in the file wins, whichever User-agent group it sits in. A file that sets different delays per crawler is reported from its first one.
  • Only /robots.txt on the audited scheme, host and port is read. A robots.txt on another subdomain or served only over the other scheme is not seen.

​
Behavior changes from the previous version

  • An HTTP 5xx answer on robots.txt used to be reported as FAIL; it is now ERROR (500), because a server error says nothing about whether the file declares a Crawl-delay.
  • A malformed line such as Crawl-delay without a colon or Crawl-delay: fast used to crash the audit; it is now a WARNING (301).
  • A robots.txt that answers 404, 403 or another status below 500 still fails (400), now with the observed status in the message.