> ## Documentation Index > Fetch the complete documentation index at: https://docs.37audits.com/llms.txt > Use this file to discover all available pages before exploring further. # Robots > Checks that robots.txt is readable, means what it says, keeps search engines and AI assistants in, and declares a numeric Crawl-delay. > [!NOTE] > > **SEO Auditor** — fetches the site's `robots.txt` and checks how crawlers read it: whether it answers, whether every line is a directive they understand, which search engines, AI assistants, AI training crawlers and SEO tools it shuts out, whether it disallows the audited page, and whether it declares a usable `Crawl-delay`. > **Protection** > > Catches the robots.txt mistakes that quietly take a site out of search engines and AI answers, such as a staging `Disallow: /` that reached production. > **Scope** > > Reads only `/robots.txt` on the audited host; the audited page itself is never requested. `robots.txt` tells crawlers which paths they may fetch. Its grammar is the Robots Exclusion Protocol, [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) (September 2022): `User-agent` lines open a group, `Allow` and `Disallow` rules in the group apply to the crawlers it names, the longest matching rule wins, and a crawler no group names follows the `User-agent: *` group. `Sitemap` and `Crawl-delay` are widely honored extensions. This auditor matches groups and rules the way RFC 9309 crawlers do (through [crawler-commons](https://github.com/crawler-commons/crawler-commons)) and reports what each crawler it knows ends up allowed to fetch. ## How it works 1. **Build the robots URL** — takes the scheme, host and port of the audited URL and appends `/robots.txt`. An audited URL without a scheme or host ends the audit with `500`. 2. **Fetch** — requests `robots.txt` as 37AuditsBot, following redirects. This is a probe of another path, not the audited page, so a 403 is a finding (`400`), not a blocked bot: this auditor never emits `590`. HTTP 5xx fails (`401`), because RFC 9309 tells crawlers to treat it as a full Disallow. A network error ends the audit with `500`. 3. **Scan lines** — when the answer is HTTP 200, removes a byte-order mark and `#` comments and reads each line as `directive: value`. That feeds the syntax rules (`302`–`308`), the sensitive-paths list (`103`) and `Crawl-delay` (`300`, `301`). 4. **Match crawlers** — reads the `User-agent: *` group, then every crawler of the [bundled catalog](#crawler-catalog) that a `User-agent` line names. A crawler counts as blocked when `/` is disallowed for it. When the `*` group blocks every crawler (`402`), only crawlers with a blocking group of their own are reported again, so one staging leftover does not turn into dozens of checks. 5. **Audited page** — when `/` is open to `User-agent: *` and Googlebot but the audited URL is not, fails with `404`. 6. **Aggregate** — FAIL wins over WARNING. `INFORMATION` checks never change the status. When no rule warns or fails the audit is `SUCCESS` (`200`) and the message quotes the declared `Crawl-delay`. ## What it audits Every row maps to one `checkRule*` method and one `CheckCode`. CheckCode suffixes use status ranges: `100–199` information, `200` success, `300–399` warning, `400–499` fail, `500–599` error. | CheckCode | Status | When it fires | Recommendation | Why it matters | | - | - | - | - | - | | `37A-RobotsAuditor-100` | `INFORMATION` | Always: names RFC 9309, the specification this auditor implements. | — | — | | `37A-RobotsAuditor-101` | `INFORMATION` | One check listing the AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) that `robots.txt` keeps out of the whole site. | — | Blocking them is a legitimate choice: their owners will not train models on the site. | | `37A-RobotsAuditor-102` | `INFORMATION` | One check listing the SEO and marketing crawlers (AhrefsBot, SemrushBot, MJ12bot…) kept out of the whole site. | — | Those tools will not report the site's backlinks or rankings. | | `37A-RobotsAuditor-103` | `INFORMATION` | `Disallow` paths that contain `backup`, `.git`, `.svn`, `.env`, `config`, `secret`, `admin` or `.sql` (WordPress's `wp-admin` is skipped). | — | `robots.txt` is public, so these lines tell anyone where to look; only access control protects such paths. | | `37A-RobotsAuditor-104` | `INFORMATION` | One check listing the regional or niche search engines (Yandex, Baidu, Naver, Seznam, Petal, Applebot, DuckDuckBot…) kept out of the whole site. | — | Many sites block them on purpose; the site will not appear in those engines. | | `37A-RobotsAuditor-200` | `SUCCESS` | No rule warned or failed and `robots.txt` declares a numeric `Crawl-delay`. The message quotes the value. | — | — | | `37A-RobotsAuditor-300` | `WARNING` | `robots.txt` was fetched (HTTP 200) but contains no `Crawl-delay` directive. | Add a "Crawl-delay: \" line (for example "Crawl-delay: 10") to `{robotsUrl}` under the User-agent groups whose crawl rate you want to limit. | Without a Crawl-delay, crawlers that honor it request pages as fast as they like, which can slow the site for real visitors and inflate hosting costs. | | `37A-RobotsAuditor-301` | `WARNING` | A `Crawl-delay` directive is present but its value is empty or not a non-negative number. | Replace "Crawl-delay: `{crawlDelay}`" in `{robotsUrl}` with a number of seconds, for example "Crawl-delay: 10". | Crawlers ignore a Crawl-delay they cannot read as a number of seconds, so the site gets none of the rate limiting the directive was meant to provide. | | `37A-RobotsAuditor-302` | `WARNING` | The `Content-Type` is not `text/plain`, or the body is an HTML document (a soft 404 or a catch-all route). | Serve `{robotsUrl}` as a plain-text file with "Content-Type: text/plain" that contains only robots.txt directives. | A robots.txt served as an HTML page holds no directives crawlers can read, so the site gets none of the rules it meant to publish. | | `37A-RobotsAuditor-303` | `WARNING` | The body is larger than 500 KiB. | Shrink `{robotsUrl}` below 500 KiB: merge rules with wildcards and remove one-off paths. | RFC 9309 only requires crawlers to read the first 500 KiB and Google ignores the rest, so rules past that point never apply. | | `37A-RobotsAuditor-304` | `WARNING` | One check per line (at most 20) that has no colon or uses an unknown or misspelled directive. `Host`, `Clean-param`, `Request-rate` and `Visit-time` are tolerated. | Fix line `{lineNumber}` to use one of User-agent, Allow, Disallow, Sitemap or Crawl-delay followed by a colon, or remove it. | Crawlers silently skip such a line, so the rule its author meant never applies. | | `37A-RobotsAuditor-305` | `WARNING` | One check per `Allow` or `Disallow` line that comes before the first `User-agent` line. | Move line `{lineNumber}` under a "User-agent: \*" line (or the group of the crawler it targets). | RFC 9309 parsers ignore rules that belong to no group, so those paths stay open to every crawler. | | `37A-RobotsAuditor-306` | `WARNING` | One check per `Noindex`, `Nofollow` or `Noarchive` line. | Remove the line and send a `` tag or an `X-Robots-Tag` header on those pages instead, keeping them crawlable. | Google stopped honoring indexing directives in robots.txt in September 2019 and other crawlers never did, so those pages can still appear in search results. | | `37A-RobotsAuditor-307` | `WARNING` | One check per `Allow` or `Disallow` value that is not empty and does not start with `/` or `*` (a bare word or a full URL). | Rewrite line `{lineNumber}` with a path that starts with "/" (for example "Disallow: /admin"). | Rules match the URL path, which always starts with "/", so such a pattern never matches anything. | | `37A-RobotsAuditor-308` | `WARNING` | One check per `Sitemap` line whose value is not an absolute `http(s)` URL. Whether the sitemap answers is checked by the [Sitemap auditor](/auditors/sitemap-auditor). | Write line `{lineNumber}` as a full URL, for example "Sitemap: [https://example.com/sitemap.xml](https://example.com/sitemap.xml)". | The sitemaps protocol requires a fully qualified URL in robots.txt, so crawlers may skip a relative one and miss the pages it lists. | | `37A-RobotsAuditor-309` | `WARNING` | One check per AI search or assistant crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot…) kept out of the whole site. | If the site should appear in that company's AI answers, remove "Disallow: /" for its User-agent. | AI assistants only cite and link pages their crawlers may fetch, so a blocked crawler takes the site out of their answers. | | `37A-RobotsAuditor-400` | `FAIL` | `robots.txt` answered an HTTP status other than 200 and below 500 (404, 403, 401, a 3xx left after redirects…). | Publish a robots.txt file at `{robotsUrl}` that answers HTTP 200 and declare a Crawl-delay directive in it. | Without a reachable robots.txt, crawlers get no crawl-rate or exclusion guidance, so they may hit the site harder and index pages that should stay out of search results. | | `37A-RobotsAuditor-401` | `FAIL` | `robots.txt` answered HTTP 5xx. | Make `{robotsUrl}` answer HTTP 200 (or 404 when there are no rules); check the server or CDN rule that serves it. | RFC 9309 tells crawlers to treat a robots.txt that answers 5xx as a full Disallow, so search engines stop crawling the whole site until it answers again. | | `37A-RobotsAuditor-402` | `FAIL` | The `User-agent: *` group disallows `/`, unless Googlebot and Bingbot each have a group of their own that lets them in (a deliberate allow list). | Remove "Disallow: /" from the "User-agent: \*" group; disallow only the paths that must stay out of search. | It keeps every compliant crawler out, so the site drops out of search engines and AI answers. It is often left over from a staging site. | | `37A-RobotsAuditor-403` | `FAIL` | One check each when Googlebot or Bingbot is kept out of the whole site. | Remove "Disallow: /" for that User-agent. | Googlebot and Bingbot feed most search traffic (Bing also feeds DuckDuckGo, Yahoo and ChatGPT search), so blocking either removes the site from those results. | | `37A-RobotsAuditor-404` | `FAIL` | `/` is open but the audited URL is disallowed for `User-agent: *` or Googlebot. The check's resource is the audited URL. | Remove or narrow the matching Disallow rule, or add an "Allow" rule for the page's path, if the page should appear in search. | A disallowed page is not crawled, so search engines never see its content and keep a stale or empty entry for it. | | `37A-RobotsAuditor-500` | `ERROR` | `robots.txt` could not be fetched: a network error or an audited URL without a scheme or host. | Make sure `{robotsUrl}` is reachable over HTTP(S), then run the audit again. | — | ## Output documentation | Status | Description | Test logic | | - | - | - | | **SUCCESS** | robots.txt is healthy and declares a usable Crawl-delay | HTTP 200, no WARNING or FAIL rule fired, and the first `Crawl-delay` line has a non-negative numeric value (`200`). | | **WARNING** | Crawlers misread part of the file, or an AI assistant is shut out | Crawl-delay missing or invalid (`300`, `301`); wrong content type, oversize file or a line crawlers skip (`302`–`308`); an AI search crawler blocked (`309`). | | **FAIL** | Crawlers cannot read the file, or search engines are shut out | `robots.txt` missing (`400`) or answering 5xx (`401`); every crawler blocked (`402`); Googlebot or Bingbot blocked (`403`); the audited page disallowed (`404`). | | **ERROR** | The auditor could not fetch robots.txt | Network error or invalid audited URL (`500`). | Every check carries `robotsUrl`, `httpStatus`, `contentType`, `sizeBytes`, `crawlDelayFound`, `crawlDelay` (the raw value, or `null`) and `crawlDelaySeconds` (the parsed number, or `null`) in its `data`. Line rules add `lineNumber` and `line`, crawler rules add `bots` (the product tokens), `103` adds `paths`, and `404` adds `auditedUrl` and `agents`. The resource is the `robots.txt` URL, except for `404` (the audited URL) and `100` (the RFC). ## Crawler catalog The crawlers are bundled with the auditor and identified by the product token each owner documents for `robots.txt`. | Group | Crawlers | Blocked means | | - | - | - | | Major search engines | Googlebot, Bingbot | `FAIL` (`403`) | | Other search engines | Applebot, DuckDuckBot, YandexBot, Baiduspider, Yeti, SeznamBot, PetalBot, Slurp, Qwantbot, MojeekBot | `INFORMATION` (`104`) | | AI search and assistants | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, YouBot | `WARNING` (`309`) | | AI training | GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, cohere-training-data-crawler, Diffbot, ImagesiftBot, Omgilibot, AI2Bot | `INFORMATION` (`101`) | | SEO tools | AhrefsBot, SemrushBot, MJ12bot, DotBot, rogerbot, BLEXBot, DataForSeoBot, serpstatbot, Barkrowler | `INFORMATION` (`102`) | Scrapers, email harvesters and vulnerability scanners are not in the catalog: they ignore `robots.txt`, so naming them in the file does not keep them out. ## Risks and considerations ### What "blocked" means * A crawler is reported only when `/` is disallowed for it. A group that disallows some paths (`Disallow: /blog/`) is a normal exclusion and is not reported, except for the audited page (`404`). * `Disallow:` with an empty value allows everything, as RFC 9309 says. * `Allow` rules and wildcards (`*`, `$`) count: `Disallow: /` with `Allow: /$` still blocks every page except the home page, so auditing any other page fails with `404` even though `/` is open. ### Crawl-delay * `Crawl-delay` is not part of RFC 9309. Bing, Yandex and many SEO and AI crawlers honor it; Googlebot ignores it, so it is not a substitute for rate limiting at the server or CDN. * The first `Crawl-delay` line in the file wins, whichever `User-agent` group it sits in. ### Scope * Only `/robots.txt` on the audited scheme, host and port is read. A `robots.txt` on another subdomain or served only over the other scheme is not seen. * `robots.txt` asks; it does not enforce. A crawler that ignores the file is not stopped by it, which is why the sensitive-paths list (`103`) is information, not protection. ### Behavior changes from Robots Crawl Delay This auditor replaced the [Robots Crawl Delay](/auditors/robots-crawl-delay-auditor) auditor, deprecated in October 2026. Results it recorded keep their `37A-RobotsCrawlDelayAuditor-*` codes, explained on its page. * An HTTP 5xx answer on `robots.txt` used to be an `ERROR` (`500`); it is now a `FAIL` (`401`), because RFC 9309 crawlers treat it as a full Disallow. * Every check now starts with the RFC 9309 specification pin (`100`). * Rules `101`–`104`, `302`–`309` and `402`–`404` are new. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.