Web Bot Auth Auditor

Security Auditor — verifies the signed key directory at /.well-known/http-message-signatures-directory that origins such as Cloudflare use to recognise your bot.

Bot verification

Cloudflare Verified Bots and any RFC 9421 verifier look up your public keys in this directory before trusting your crawler.

Signed directory

The directory response itself must be signed by each key it publishes, with a short, unexpired validity window.

This auditor is for sites that operate a bot and sign its requests with Web Bot Auth. It audits the key directory the site serves, not the requests the bot sends: only the directory is observable from the outside.

How it works

The auditor ports the checks of Cloudflare’s http-signature-directory validator:

Locate — builds <scheme>://<host>/.well-known/http-message-signatures-directory from the audited URL (its path is ignored) and fetches it with Accept: application/http-message-signatures-directory+json, following redirects.
Blocked or missing — HTTP 403, 429 or a Cloudflare challenge ends the run as a blocked bot (590). Any other non-2xx status fails (400): a bot owner without a reachable directory cannot be verified.
Keys — parses the body as a JWKS. Each key must be an OKP key on curve Ed25519 with a 32-byte base64url x. The RFC 7638 thumbprint of each key is the keyid verifiers expect.
Signatures — reads Signature-Input and Signature. For every key there must be an entry with tag="http-message-signatures-directory", keyid equal to the thumbprint, created, expires and the ("@authority";req) component. The entry must not be expired, and the Ed25519 signature must verify over the RFC 9421 signature base built from the final response URL.
Hygiene — warns when the directory is served over http, with the wrong Content-Type, or with a Cache-Control lifetime longer than the signature’s remaining validity (a cached copy would be served with an expired signature).
Aggregate — ERROR wins, then FAIL, then WARNING; if every rule passes, the audit is SUCCESS.
Specification pin — every run emits 37A-WebBotAuthAuditor-100 (INFORMATION) naming draft-meunier-http-message-signatures-directory-03 and RFC 9421. That check does not change overall status.

What it audits

Every row maps to one checkRule* method and one CheckCode. 37A-WebBotAuthAuditor-100 is emitted on every run, including fetch errors. Key rules (403–409) run once per published key, so one audit can carry several checks with the same code. 401 skips every later FAIL rule; 402 skips the key rules; a key that fails 403 or 404 skips 406–409; 405 skips 406–409 for all keys; 406 skips 407–409 for that key; 407 skips 408 and 409. CheckCode suffixes use status ranges: 100 information, 200 success, 300–399 warning, 400–499 fail, 500–599 error.

CheckCode	Status	When it fires	Recommendation	Why it matters
`37A-WebBotAuthAuditor-100`	`INFORMATION`	Every run. Records that this auditor implements draft-meunier-http-message-signatures-directory-03 and RFC 9421.	— (recommendation is `null`; nothing for the site to change)	—
`37A-WebBotAuthAuditor-300`	`WARNING`	The directory was fetched over `http`.	Serve `{directoryUrl}` over HTTPS and point Signature-Agent at the https origin.	Over plain HTTP the directory can be swapped in transit, and Cloudflare only accepts https Signature-Agent origins, so the bot cannot be verified.
`37A-WebBotAuthAuditor-400`	`FAIL`	The directory answered a status other than 2xx (after the blocked check).	Publish the JWKS at `{directoryUrl}` so it answers HTTP 200 with media type application/http-message-signatures-directory+json.	Without a reachable key directory, origins such as Cloudflare cannot verify your bot’s signatures, so it is treated as an unknown crawler and may be blocked.
`37A-WebBotAuthAuditor-301`	`WARNING`	`Content-Type` is absent or its media type is not `application/http-message-signatures-directory+json`.	Set Content-Type: application/http-message-signatures-directory+json on `{directoryUrl}`.	Verifiers that check the media type skip the directory, so keys may be ignored even though the signature is right.
`37A-WebBotAuthAuditor-401`	`FAIL`	The body is not a JSON object with a `keys` array.	Return a JSON object with a “keys” array of JWKs at `{directoryUrl}`, for example `{"keys":[{"kty":"OKP","crv":"Ed25519","x":"…"}]}`.	A directory that cannot be parsed hands verifiers no keys, so every signed request from your bot fails verification.
`37A-WebBotAuthAuditor-402`	`FAIL`	The `keys` array is empty.	Add the Ed25519 public key your bot signs with to the “keys” array at `{directoryUrl}`.	With no keys published there is nothing to verify your bot’s signatures against, so verification always fails.
`37A-WebBotAuthAuditor-403`	`FAIL`	A key is not `kty=OKP` on curve `Ed25519`.	Replace key `{index}` (`{kid or thumbprint}`) with an Ed25519 OKP key (`{"kty":"OKP","crv":"Ed25519","x":"…"}`).	Cloudflare Web Bot Auth only verifies Ed25519 signatures, so requests signed with this key are rejected.
`37A-WebBotAuthAuditor-404`	`FAIL`	An Ed25519 key’s `x` is missing, not base64url, or does not decode to 32 bytes.	Export the public key of key `{index}` again and publish its raw 32 bytes base64url-encoded (no padding) in “x”.	A malformed public key cannot be imported by verifiers, so your bot’s signatures can never be checked.
`37A-WebBotAuthAuditor-405`	`FAIL`	The response has no `Signature-Input` or no `Signature` header.	Sign the directory response: add Signature-Input: sig1=(“@authority”;req);alg=“ed25519”;keyid=“`{thumbprint}`”;created=…;expires=…;tag=“http-message-signatures-directory” and the matching Signature: sig1=:…: header.	Verifiers only trust keys whose directory response is signed by them, so an unsigned directory leaves your bot unverified.
`37A-WebBotAuthAuditor-406`	`FAIL`	No `Signature-Input` entry has `tag="http-message-signatures-directory"`, `keyid` equal to the key’s thumbprint, `created`, `expires` and an `@authority` component.	Add a Signature-Input entry for key `{thumbprint}` with tag “http-message-signatures-directory”, keyid `{thumbprint}`, created, expires and the “@authority”;req component, plus its Signature entry.	Verifiers ignore keys that do not have their own valid directory signature, so this key cannot vouch for your bot.
`37A-WebBotAuthAuditor-407`	`FAIL`	The matching entry covers `@authority` without the `req` parameter.	Change the covered component in `{label}` from “@authority” to “@authority”;req and sign again.	Cloudflare’s verifier resolves @authority from the request, so a directory signed without req fails verification and your bot stays unverified.
`37A-WebBotAuthAuditor-408`	`FAIL`	The entry’s `expires` is in the past.	Sign the directory response on every request, or refresh the cached copy before `{expires}`, so that expires of `{label}` is in the future.	An expired directory signature is rejected like a missing one, so your bot loses verification until the directory is re-signed.
`37A-WebBotAuthAuditor-409`	`FAIL`	The entry’s `Signature` is missing, cannot be decoded, or does not verify with the key under Ed25519.	Sign exactly the components listed in Signature-Input `{label}` (RFC 9421 signature base, Ed25519) with the private key of `{thumbprint}` and publish the result in the Signature header.	A signature that does not verify means the published key and the signing key differ, so verifiers treat your bot as an impostor.
`37A-WebBotAuthAuditor-302`	`WARNING`	`Cache-Control` `max-age` or `s-maxage` is longer than the matching signature’s remaining validity (skipped when the signature is already expired).	Lower max-age below `{remaining}` s or extend expires of `{label}` beyond the cache lifetime of `{maxAge}` s.	A cache that outlives the signature serves an expired directory, and verifiers then reject your bot until the cache refreshes.
`37A-WebBotAuthAuditor-500`	`ERROR`	The URL is invalid or the directory could not be fetched (DNS, TLS, timeout).	Make sure `{directoryUrl}` resolves and answers over TLS, then run the audit again. Last error was `{error}`.	—
`37A-WebBotAuthAuditor-200`	`SUCCESS`	All rules passed. Message is `The key directory at "{directoryUrl}" serves {n} Ed25519 key(s) ({thumbprints}) and its signature(s) verified; "{label}" expires at {expires}.`	—	—
`37A-WebBotAuthAuditor-590`	`ERROR`	The key directory answered HTTP 403, HTTP 429 or a Cloudflare challenge, so 37AuditsBot was blocked before it could read the directory.	Allow 37AuditsBot in your firewall or bot protection: allowlist its user agent, or verify its Web Bot Auth signature. See https://www.37audits.com/bot.	While 37AuditsBot is blocked this audit cannot look at the site, so problems on it go unreported.

Output documentation

Status	Description	Test logic
SUCCESS	The directory serves Ed25519 keys and every key has a valid, unexpired directory signature	HTTP 2xx, JWKS of Ed25519 keys, one matching `Signature-Input`/`Signature` entry per key that verifies (`200`).
WARNING	The directory verifies but its transport, media type or cache lifetime is off	Fetched over http (`300`); wrong or missing `Content-Type` (`301`); `Cache-Control` outlives the signature (`302`).
FAIL	The directory is missing, malformed, unsigned, or its signature is wrong	Non-2xx status (`400`); body is not a JWKS (`401`); no keys (`402`); non-Ed25519 key (`403`); invalid `x` (`404`); no signature headers (`405`); no entry for a key (`406`); `@authority` without `req` (`407`); expired (`408`); verification failed (`409`).
ERROR	The auditor could not fetch the directory	Invalid URL, DNS, TLS or timeout (`500`); 37AuditsBot blocked (`590`).
INFORMATION	Specification pin	Always emitted (`100`); does not change overall status.

Risks and considerations

An unverifiable bot is an anonymous bot

Cloudflare and other origins verify your crawler by fetching this directory and checking the signature on it. A missing, unsigned or expired directory makes every signed request from your bot look unverified, and bot protection treats it like any unknown crawler.
Rotate keys by publishing the old and the new key together, each with its own directory signature, until the bot has switched.

Caching silently breaks signed directories

The directory response is signed with a created/expires window. A CDN, framework cache or static export that serves the same response past expires hands verifiers an expired signature. Sign on every request, or keep the cache lifetime below the validity window.

What this auditor does not check

The bot’s own signed requests (tag="web-bot-auth", the Signature-Agent header) leave the bot, not the site, so they cannot be audited from the outside. Use Cloudflare’s crawltest.com to test those.
The content-digest component the draft recommends is not required; Cloudflare’s validator does not require it either.
Registration with Cloudflare Verified Bots is a separate submission and is not verified.

Auditors

MCP Server

REST API

Web Bot Auth Auditor

Bot verification

Signed directory

How it works

What it audits

Output documentation

Risks and considerations

An unverifiable bot is an anonymous bot

Caching silently breaks signed directories

What this auditor does not check

Bot verification

Signed directory

​How it works

​What it audits

​Output documentation

​Risks and considerations

​An unverifiable bot is an anonymous bot

​Caching silently breaks signed directories

​What this auditor does not check

How it works

What it audits

Output documentation

Risks and considerations

An unverifiable bot is an anonymous bot

Caching silently breaks signed directories

What this auditor does not check