Robots

SEO Auditor — fetches the site’s robots.txt and checks how crawlers read it: whether it answers, whether every line is a directive they understand, which search engines, AI assistants, AI training crawlers and SEO tools it shuts out, whether it disallows the audited page, and whether it declares a usable Crawl-delay.

Protection

Catches the robots.txt mistakes that quietly take a site out of search engines and AI answers, such as a staging Disallow: / that reached production.

Scope

Reads only /robots.txt on the audited host; the audited page itself is never requested.

robots.txt tells crawlers which paths they may fetch. Its grammar is the Robots Exclusion Protocol, RFC 9309 (September 2022): User-agent lines open a group, Allow and Disallow rules in the group apply to the crawlers it names, the longest matching rule wins, and a crawler no group names follows the User-agent: * group. Sitemap and Crawl-delay are widely honored extensions. This auditor matches groups and rules the way RFC 9309 crawlers do (through crawler-commons) and reports what each crawler it knows ends up allowed to fetch.

How it works

Build the robots URL — takes the scheme, host and port of the audited URL and appends /robots.txt. An audited URL without a scheme or host ends the audit with 500.
Fetch — requests robots.txt as 37AuditsBot, following redirects. This is a probe of another path, not the audited page, so a 403 is a finding (400), not a blocked bot: this auditor never emits 590. HTTP 5xx fails (401), because RFC 9309 tells crawlers to treat it as a full Disallow. A network error ends the audit with 500.
Scan lines — when the answer is HTTP 200, removes a byte-order mark and # comments and reads each line as directive: value. That feeds the syntax rules (302–308), the sensitive-paths list (103) and Crawl-delay (300, 301).
Match crawlers — reads the User-agent: * group, then every crawler of the bundled catalog that a User-agent line names. A crawler counts as blocked when / is disallowed for it. When the * group blocks every crawler (402), only crawlers with a blocking group of their own are reported again, so one staging leftover does not turn into dozens of checks.
Audited page — when / is open to User-agent: * and Googlebot but the audited URL is not, fails with 404.
Aggregate — FAIL wins over WARNING. INFORMATION checks never change the status. When no rule warns or fails the audit is SUCCESS (200) and the message quotes the declared Crawl-delay.

What it audits

Every row maps to one checkRule* method and one CheckCode. CheckCode suffixes use status ranges: 100–199 information, 200 success, 300–399 warning, 400–499 fail, 500–599 error.

CheckCode	Status	When it fires	Recommendation	Why it matters
`37A-RobotsAuditor-100`	`INFORMATION`	Always: names RFC 9309, the specification this auditor implements.	—	—
`37A-RobotsAuditor-101`	`INFORMATION`	One check listing the AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) that `robots.txt` keeps out of the whole site.	—	Blocking them is a legitimate choice: their owners will not train models on the site.
`37A-RobotsAuditor-102`	`INFORMATION`	One check listing the SEO and marketing crawlers (AhrefsBot, SemrushBot, MJ12bot…) kept out of the whole site.	—	Those tools will not report the site’s backlinks or rankings.
`37A-RobotsAuditor-103`	`INFORMATION`	`Disallow` paths that contain `backup`, `.git`, `.svn`, `.env`, `config`, `secret`, `admin` or `.sql` (WordPress’s `wp-admin` is skipped).	—	`robots.txt` is public, so these lines tell anyone where to look; only access control protects such paths.
`37A-RobotsAuditor-104`	`INFORMATION`	One check listing the regional or niche search engines (Yandex, Baidu, Naver, Seznam, Petal, Applebot, DuckDuckBot…) kept out of the whole site.	—	Many sites block them on purpose; the site will not appear in those engines.
`37A-RobotsAuditor-200`	`SUCCESS`	No rule warned or failed and `robots.txt` declares a numeric `Crawl-delay`. The message quotes the value.	—	—
`37A-RobotsAuditor-300`	`WARNING`	`robots.txt` was fetched (HTTP 200) but contains no `Crawl-delay` directive.	Add a “Crawl-delay: <seconds>” line (for example “Crawl-delay: 10”) to `{robotsUrl}` under the User-agent groups whose crawl rate you want to limit.	Without a Crawl-delay, crawlers that honor it request pages as fast as they like, which can slow the site for real visitors and inflate hosting costs.
`37A-RobotsAuditor-301`	`WARNING`	A `Crawl-delay` directive is present but its value is empty or not a non-negative number.	Replace “Crawl-delay: `{crawlDelay}`” in `{robotsUrl}` with a number of seconds, for example “Crawl-delay: 10”.	Crawlers ignore a Crawl-delay they cannot read as a number of seconds, so the site gets none of the rate limiting the directive was meant to provide.
`37A-RobotsAuditor-302`	`WARNING`	The `Content-Type` is not `text/plain`, or the body is an HTML document (a soft 404 or a catch-all route).	Serve `{robotsUrl}` as a plain-text file with “Content-Type: text/plain” that contains only robots.txt directives.	A robots.txt served as an HTML page holds no directives crawlers can read, so the site gets none of the rules it meant to publish.
`37A-RobotsAuditor-303`	`WARNING`	The body is larger than 500 KiB.	Shrink `{robotsUrl}` below 500 KiB: merge rules with wildcards and remove one-off paths.	RFC 9309 only requires crawlers to read the first 500 KiB and Google ignores the rest, so rules past that point never apply.
`37A-RobotsAuditor-304`	`WARNING`	One check per line (at most 20) that has no colon or uses an unknown or misspelled directive. `Host`, `Clean-param`, `Request-rate` and `Visit-time` are tolerated.	Fix line `{lineNumber}` to use one of User-agent, Allow, Disallow, Sitemap or Crawl-delay followed by a colon, or remove it.	Crawlers silently skip such a line, so the rule its author meant never applies.
`37A-RobotsAuditor-305`	`WARNING`	One check per `Allow` or `Disallow` line that comes before the first `User-agent` line.	Move line `{lineNumber}` under a “User-agent: *” line (or the group of the crawler it targets).	RFC 9309 parsers ignore rules that belong to no group, so those paths stay open to every crawler.
`37A-RobotsAuditor-306`	`WARNING`	One check per `Noindex`, `Nofollow` or `Noarchive` line.	Remove the line and send a `<meta name="robots">` tag or an `X-Robots-Tag` header on those pages instead, keeping them crawlable.	Google stopped honoring indexing directives in robots.txt in September 2019 and other crawlers never did, so those pages can still appear in search results.
`37A-RobotsAuditor-307`	`WARNING`	One check per `Allow` or `Disallow` value that is not empty and does not start with `/` or `*` (a bare word or a full URL).	Rewrite line `{lineNumber}` with a path that starts with ”/” (for example “Disallow: /admin”).	Rules match the URL path, which always starts with ”/”, so such a pattern never matches anything.
`37A-RobotsAuditor-308`	`WARNING`	One check per `Sitemap` line whose value is not an absolute `http(s)` URL. Whether the sitemap answers is checked by the Sitemap auditor.	Write line `{lineNumber}` as a full URL, for example “Sitemap: https://example.com/sitemap.xml”.	The sitemaps protocol requires a fully qualified URL in robots.txt, so crawlers may skip a relative one and miss the pages it lists.
`37A-RobotsAuditor-309`	`WARNING`	One check per AI search or assistant crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot…) kept out of the whole site.	If the site should appear in that company’s AI answers, remove “Disallow: /” for its User-agent.	AI assistants only cite and link pages their crawlers may fetch, so a blocked crawler takes the site out of their answers.
`37A-RobotsAuditor-400`	`FAIL`	`robots.txt` answered an HTTP status other than 200 and below 500 (404, 403, 401, a 3xx left after redirects…).	Publish a robots.txt file at `{robotsUrl}` that answers HTTP 200 and declare a Crawl-delay directive in it.	Without a reachable robots.txt, crawlers get no crawl-rate or exclusion guidance, so they may hit the site harder and index pages that should stay out of search results.
`37A-RobotsAuditor-401`	`FAIL`	`robots.txt` answered HTTP 5xx.	Make `{robotsUrl}` answer HTTP 200 (or 404 when there are no rules); check the server or CDN rule that serves it.	RFC 9309 tells crawlers to treat a robots.txt that answers 5xx as a full Disallow, so search engines stop crawling the whole site until it answers again.
`37A-RobotsAuditor-402`	`FAIL`	The `User-agent: *` group disallows `/`, unless Googlebot and Bingbot each have a group of their own that lets them in (a deliberate allow list).	Remove “Disallow: /” from the “User-agent: *” group; disallow only the paths that must stay out of search.	It keeps every compliant crawler out, so the site drops out of search engines and AI answers. It is often left over from a staging site.
`37A-RobotsAuditor-403`	`FAIL`	One check each when Googlebot or Bingbot is kept out of the whole site.	Remove “Disallow: /” for that User-agent.	Googlebot and Bingbot feed most search traffic (Bing also feeds DuckDuckGo, Yahoo and ChatGPT search), so blocking either removes the site from those results.
`37A-RobotsAuditor-404`	`FAIL`	`/` is open but the audited URL is disallowed for `User-agent: *` or Googlebot. The check’s resource is the audited URL.	Remove or narrow the matching Disallow rule, or add an “Allow” rule for the page’s path, if the page should appear in search.	A disallowed page is not crawled, so search engines never see its content and keep a stale or empty entry for it.
`37A-RobotsAuditor-500`	`ERROR`	`robots.txt` could not be fetched: a network error or an audited URL without a scheme or host.	Make sure `{robotsUrl}` is reachable over HTTP(S), then run the audit again.	—

Output documentation

Status	Description	Test logic
SUCCESS	robots.txt is healthy and declares a usable Crawl-delay	HTTP 200, no WARNING or FAIL rule fired, and the first `Crawl-delay` line has a non-negative numeric value (`200`).
WARNING	Crawlers misread part of the file, or an AI assistant is shut out	Crawl-delay missing or invalid (`300`, `301`); wrong content type, oversize file or a line crawlers skip (`302`–`308`); an AI search crawler blocked (`309`).
FAIL	Crawlers cannot read the file, or search engines are shut out	`robots.txt` missing (`400`) or answering 5xx (`401`); every crawler blocked (`402`); Googlebot or Bingbot blocked (`403`); the audited page disallowed (`404`).
ERROR	The auditor could not fetch robots.txt	Network error or invalid audited URL (`500`).

Every check carries robotsUrl, httpStatus, contentType, sizeBytes, crawlDelayFound, crawlDelay (the raw value, or null) and crawlDelaySeconds (the parsed number, or null) in its data. Line rules add lineNumber and line, crawler rules add bots (the product tokens), 103 adds paths, and 404 adds auditedUrl and agents. The resource is the robots.txt URL, except for 404 (the audited URL) and 100 (the RFC).

Crawler catalog

The crawlers are bundled with the auditor and identified by the product token each owner documents for robots.txt.

Group	Crawlers	Blocked means
Major search engines	Googlebot, Bingbot	`FAIL` (`403`)
Other search engines	Applebot, DuckDuckBot, YandexBot, Baiduspider, Yeti, SeznamBot, PetalBot, Slurp, Qwantbot, MojeekBot	`INFORMATION` (`104`)
AI search and assistants	OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, YouBot	`WARNING` (`309`)
AI training	GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, cohere-training-data-crawler, Diffbot, ImagesiftBot, Omgilibot, AI2Bot	`INFORMATION` (`101`)
SEO tools	AhrefsBot, SemrushBot, MJ12bot, DotBot, rogerbot, BLEXBot, DataForSeoBot, serpstatbot, Barkrowler	`INFORMATION` (`102`)

Scrapers, email harvesters and vulnerability scanners are not in the catalog: they ignore robots.txt, so naming them in the file does not keep them out.

Risks and considerations

What “blocked” means

A crawler is reported only when / is disallowed for it. A group that disallows some paths (Disallow: /blog/) is a normal exclusion and is not reported, except for the audited page (404).
Disallow: with an empty value allows everything, as RFC 9309 says.
Allow rules and wildcards (*, $) count: Disallow: / with Allow: /$ still blocks every page except the home page, so auditing any other page fails with 404 even though / is open.

Crawl-delay

Crawl-delay is not part of RFC 9309. Bing, Yandex and many SEO and AI crawlers honor it; Googlebot ignores it, so it is not a substitute for rate limiting at the server or CDN.
The first Crawl-delay line in the file wins, whichever User-agent group it sits in.

Scope

Only /robots.txt on the audited scheme, host and port is read. A robots.txt on another subdomain or served only over the other scheme is not seen.
robots.txt asks; it does not enforce. A crawler that ignores the file is not stopped by it, which is why the sensitive-paths list (103) is information, not protection.

Behavior changes from Robots Crawl Delay

This auditor replaced the Robots Crawl Delay auditor, deprecated in October 2026. Results it recorded keep their 37A-RobotsCrawlDelayAuditor-* codes, explained on its page.

An HTTP 5xx answer on robots.txt used to be an ERROR (500); it is now a FAIL (401), because RFC 9309 crawlers treat it as a full Disallow.
Every check now starts with the RFC 9309 specification pin (100).
Rules 101–104, 302–309 and 402–404 are new.

Auditors

MCP Server

REST API

Protection

Scope

How it works

What it audits

Output documentation

Crawler catalog

Risks and considerations

What “blocked” means

Crawl-delay

Scope

Behavior changes from Robots Crawl Delay

Protection

Scope

​How it works

​What it audits

​Output documentation

​Crawler catalog

​Risks and considerations

​What “blocked” means

​Crawl-delay

​Scope

​Behavior changes from Robots Crawl Delay

How it works

What it audits

Output documentation

Crawler catalog

Risks and considerations

What “blocked” means

Crawl-delay

Scope

Behavior changes from Robots Crawl Delay