> ## Documentation Index
> Fetch the complete documentation index at: https://docs.37audits.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Web Bot Auth Auditor
> Verifies the signed Ed25519 key directory a bot owner publishes for Web Bot Auth
**Security Auditor** — verifies the signed key directory at `/.well-known/http-message-signatures-directory` that origins such as Cloudflare use to recognise your bot.
Cloudflare Verified Bots and any RFC 9421 verifier look up your public keys in this directory before trusting your crawler.
The directory response itself must be signed by each key it publishes, with a short, unexpired validity window.
This auditor is for sites that **operate a bot** and sign its requests with [Web Bot Auth](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/). It audits the key directory the site serves, not the requests the bot sends: only the directory is observable from the outside.
## How it works
The auditor ports the checks of Cloudflare's `http-signature-directory` validator:
1. **Locate** — builds `:///.well-known/http-message-signatures-directory` from the audited URL (its path is ignored) and fetches it with `Accept: application/http-message-signatures-directory+json`, following redirects.
2. **Blocked or missing** — HTTP 403, 429 or a Cloudflare challenge ends the run as a blocked bot (`590`). Any other non-2xx status fails (`400`): a bot owner without a reachable directory cannot be verified.
3. **Keys** — parses the body as a JWKS. Each key must be an `OKP` key on curve `Ed25519` with a 32-byte base64url `x`. The RFC 7638 thumbprint of each key is the `keyid` verifiers expect.
4. **Signatures** — reads `Signature-Input` and `Signature`. For every key there must be an entry with `tag="http-message-signatures-directory"`, `keyid` equal to the thumbprint, `created`, `expires` and the `("@authority";req)` component. The entry must not be expired, and the Ed25519 signature must verify over the RFC 9421 signature base built from the final response URL.
5. **Hygiene** — warns when the directory is served over `http`, with the wrong `Content-Type`, or with a `Cache-Control` lifetime longer than the signature's remaining validity (a cached copy would be served with an expired signature).
6. **Aggregate** — `ERROR` wins, then `FAIL`, then `WARNING`; if every rule passes, the audit is `SUCCESS`.
7. **Specification pin** — every run emits `37A-WebBotAuthAuditor-100` (`INFORMATION`) naming [draft-meunier-http-message-signatures-directory-03](https://datatracker.ietf.org/doc/html/draft-meunier-http-message-signatures-directory-03) and [RFC 9421](https://www.rfc-editor.org/rfc/rfc9421). That check does not change overall status.
## What it audits
Every row maps to one `checkRule*` method and one `CheckCode`. `37A-WebBotAuthAuditor-100` is emitted on every run, including fetch errors. Key rules (`403`–`409`) run once per published key, so one audit can carry several checks with the same code. `401` skips every later FAIL rule; `402` skips the key rules; a key that fails `403` or `404` skips `406`–`409`; `405` skips `406`–`409` for all keys; `406` skips `407`–`409` for that key; `407` skips `408` and `409`. CheckCode suffixes use status ranges: `100` information, `200` success, `300–399` warning, `400–499` fail, `500–599` error.
| CheckCode | Status | When it fires | Recommendation | Why it matters |
| - | - | - | - | - |
| `37A-WebBotAuthAuditor-100` | `INFORMATION` | Every run. Records that this auditor implements draft-meunier-http-message-signatures-directory-03 and RFC 9421. | — (recommendation is `null`; nothing for the site to change) | — |
| `37A-WebBotAuthAuditor-300` | `WARNING` | The directory was fetched over `http`. | Serve `{directoryUrl}` over HTTPS and point Signature-Agent at the https origin. | Over plain HTTP the directory can be swapped in transit, and Cloudflare only accepts https Signature-Agent origins, so the bot cannot be verified. |
| `37A-WebBotAuthAuditor-400` | `FAIL` | The directory answered a status other than 2xx (after the blocked check). | Publish the JWKS at `{directoryUrl}` so it answers HTTP 200 with media type application/http-message-signatures-directory+json. | Without a reachable key directory, origins such as Cloudflare cannot verify your bot's signatures, so it is treated as an unknown crawler and may be blocked. |
| `37A-WebBotAuthAuditor-301` | `WARNING` | `Content-Type` is absent or its media type is not `application/http-message-signatures-directory+json`. | Set Content-Type: application/http-message-signatures-directory+json on `{directoryUrl}`. | Verifiers that check the media type skip the directory, so keys may be ignored even though the signature is right. |
| `37A-WebBotAuthAuditor-401` | `FAIL` | The body is not a JSON object with a `keys` array. | Return a JSON object with a "keys" array of JWKs at `{directoryUrl}`, for example `{"keys":[{"kty":"OKP","crv":"Ed25519","x":"…"}]}`. | A directory that cannot be parsed hands verifiers no keys, so every signed request from your bot fails verification. |
| `37A-WebBotAuthAuditor-402` | `FAIL` | The `keys` array is empty. | Add the Ed25519 public key your bot signs with to the "keys" array at `{directoryUrl}`. | With no keys published there is nothing to verify your bot's signatures against, so verification always fails. |
| `37A-WebBotAuthAuditor-403` | `FAIL` | A key is not `kty=OKP` on curve `Ed25519`. | Replace key `{index}` (`{kid or thumbprint}`) with an Ed25519 OKP key (`{"kty":"OKP","crv":"Ed25519","x":"…"}`). | Cloudflare Web Bot Auth only verifies Ed25519 signatures, so requests signed with this key are rejected. |
| `37A-WebBotAuthAuditor-404` | `FAIL` | An Ed25519 key's `x` is missing, not base64url, or does not decode to 32 bytes. | Export the public key of key `{index}` again and publish its raw 32 bytes base64url-encoded (no padding) in "x". | A malformed public key cannot be imported by verifiers, so your bot's signatures can never be checked. |
| `37A-WebBotAuthAuditor-405` | `FAIL` | The response has no `Signature-Input` or no `Signature` header. | Sign the directory response: add Signature-Input: sig1=("@authority";req);alg="ed25519";keyid="`{thumbprint}`";created=…;expires=…;tag="http-message-signatures-directory" and the matching Signature: sig1=:…: header. | Verifiers only trust keys whose directory response is signed by them, so an unsigned directory leaves your bot unverified. |
| `37A-WebBotAuthAuditor-406` | `FAIL` | No `Signature-Input` entry has `tag="http-message-signatures-directory"`, `keyid` equal to the key's thumbprint, `created`, `expires` and an `@authority` component. | Add a Signature-Input entry for key `{thumbprint}` with tag "http-message-signatures-directory", keyid `{thumbprint}`, created, expires and the "@authority";req component, plus its Signature entry. | Verifiers ignore keys that do not have their own valid directory signature, so this key cannot vouch for your bot. |
| `37A-WebBotAuthAuditor-407` | `FAIL` | The matching entry covers `@authority` without the `req` parameter. | Change the covered component in `{label}` from "@authority" to "@authority";req and sign again. | Cloudflare's verifier resolves @authority from the request, so a directory signed without req fails verification and your bot stays unverified. |
| `37A-WebBotAuthAuditor-408` | `FAIL` | The entry's `expires` is in the past. | Sign the directory response on every request, or refresh the cached copy before `{expires}`, so that expires of `{label}` is in the future. | An expired directory signature is rejected like a missing one, so your bot loses verification until the directory is re-signed. |
| `37A-WebBotAuthAuditor-409` | `FAIL` | The entry's `Signature` is missing, cannot be decoded, or does not verify with the key under Ed25519. | Sign exactly the components listed in Signature-Input `{label}` (RFC 9421 signature base, Ed25519) with the private key of `{thumbprint}` and publish the result in the Signature header. | A signature that does not verify means the published key and the signing key differ, so verifiers treat your bot as an impostor. |
| `37A-WebBotAuthAuditor-302` | `WARNING` | `Cache-Control` `max-age` or `s-maxage` is longer than the matching signature's remaining validity (skipped when the signature is already expired). | Lower max-age below `{remaining}` s or extend expires of `{label}` beyond the cache lifetime of `{maxAge}` s. | A cache that outlives the signature serves an expired directory, and verifiers then reject your bot until the cache refreshes. |
| `37A-WebBotAuthAuditor-500` | `ERROR` | The URL is invalid or the directory could not be fetched (DNS, TLS, timeout). | Make sure `{directoryUrl}` resolves and answers over TLS, then run the audit again. Last error was `{error}`. | — |
| `37A-WebBotAuthAuditor-200` | `SUCCESS` | All rules passed. Message is `The key directory at "{directoryUrl}" serves {n} Ed25519 key(s) ({thumbprints}) and its signature(s) verified; "{label}" expires at {expires}.` | — | — |
| `37A-WebBotAuthAuditor-590` | `ERROR` | The key directory answered HTTP 403, HTTP 429 or a Cloudflare challenge, so 37AuditsBot was blocked before it could read the directory. | Allow 37AuditsBot in your firewall or bot protection: allowlist its user agent, or verify its Web Bot Auth signature. See [https://www.37audits.com/bot](https://www.37audits.com/bot). | While 37AuditsBot is blocked this audit cannot look at the site, so problems on it go unreported. |
## Output documentation
| Status | Description | Test logic |
| - | - | - |
| **SUCCESS** | The directory serves Ed25519 keys and every key has a valid, unexpired directory signature | HTTP 2xx, JWKS of Ed25519 keys, one matching `Signature-Input`/`Signature` entry per key that verifies (`200`). |
| **WARNING** | The directory verifies but its transport, media type or cache lifetime is off | Fetched over http (`300`); wrong or missing `Content-Type` (`301`); `Cache-Control` outlives the signature (`302`). |
| **FAIL** | The directory is missing, malformed, unsigned, or its signature is wrong | Non-2xx status (`400`); body is not a JWKS (`401`); no keys (`402`); non-Ed25519 key (`403`); invalid `x` (`404`); no signature headers (`405`); no entry for a key (`406`); `@authority` without `req` (`407`); expired (`408`); verification failed (`409`). |
| **ERROR** | The auditor could not fetch the directory | Invalid URL, DNS, TLS or timeout (`500`); 37AuditsBot blocked (`590`). |
| **INFORMATION** | Specification pin | Always emitted (`100`); does not change overall status. |
## Risks and considerations
### An unverifiable bot is an anonymous bot
* Cloudflare and other origins verify your crawler by fetching this directory and checking the signature on it. A missing, unsigned or expired directory makes every signed request from your bot look unverified, and bot protection treats it like any unknown crawler.
* Rotate keys by publishing the old and the new key together, each with its own directory signature, until the bot has switched.
### Caching silently breaks signed directories
* The directory response is signed with a `created`/`expires` window. A CDN, framework cache or static export that serves the same response past `expires` hands verifiers an expired signature. Sign on every request, or keep the cache lifetime below the validity window.
### What this auditor does not check
* The bot's own signed requests (`tag="web-bot-auth"`, the `Signature-Agent` header) leave the bot, not the site, so they cannot be audited from the outside. Use Cloudflare's `crawltest.com` to test those.
* The `content-digest` component the draft recommends is not required; Cloudflare's validator does not require it either.
* Registration with Cloudflare Verified Bots is a separate submission and is not verified.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.