> ## Documentation Index
> Fetch the complete documentation index at: https://docs.37audits.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Not Found Auditor
> Checks whether a nonexistent page returns a proper 404/410 status and a helpful custom 404 page
**SEO Auditor** — checks whether a nonexistent page returns a proper 404/410 status and a helpful custom 404 page.
A `noindex`ed, properly-statused 404 page keeps search engines from indexing junk URLs.
Helpful content, navigation, and a search box turn a dead end into a recoverable page.
This auditor requests a synthetic, never-existing path on the audited host, then evaluates the response independently: the HTTP status code must be 404 or 410 (not a soft 404), and — only once the status code is correct — the page itself is checked for custom not-found content, a `noindex` meta tag, navigation links, other helpful links, and a search box.
## How it works
1. **Generate a test URL** — appends `/this-page-should-not-exist-` to the audited host so the path is guaranteed not to exist.
2. **Fetch** — requests the test URL with jsoup (`ignoreHttpErrors`, redirects disabled, HTTP Basic Auth when provided) in a single request.
3. **Status rules** — a `200` response (soft 404) or any status other than `200`/`404`/`410` fails immediately; the page content is not evaluated in either case.
4. **Content rules** — once the status code is confirmed `404` or `410`, the page body is checked for custom not-found messaging (FAIL if missing), then `noindex` meta tag, navigation links, helpful links, and a search box (each WARNING if missing).
5. **Aggregate** — any `FAIL` rule wins over `WARNING`; if every rule passes, the audit is `SUCCESS`. Fetch errors (invalid URL, unreachable host) produce an `ERROR` response.
## What it audits
Every row maps to one `checkRule*` method and one `CheckCode`. CheckCode suffixes use status ranges: `200` success, `300–399` warning, `400–499` fail, `500–599` error. Content rules (`300`–`303`, `402`) are only evaluated once the status code rules (`400`, `401`) have passed.
| CheckCode | Status | When it fires | Recommendation |
| ----------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `37A-NotFoundAuditor-400` | `FAIL` | Requesting the nonexistent test URL returns HTTP 200 instead of 404/410 (a soft 404). | Configure the server to return HTTP 404 (or 410) for `{testUrl}` instead of HTTP 200, so browsers, crawlers, and monitoring tools can correctly detect the page is missing. |
| `37A-NotFoundAuditor-401` | `FAIL` | Requesting the nonexistent test URL returns a status code other than 200, 404, or 410. | Return HTTP 404 (Not Found) or 410 (Gone) for `{testUrl}` instead of HTTP `{statusCode}`. |
| `37A-NotFoundAuditor-402` | `FAIL` | The 404 page has no recognizable not-found messaging (checked against phrases such as "not found", "404", "oops", "sorry") or is too short to be meaningful. | Add a custom 404 page for `{testUrl}` with a clear message such as "Page not found" so visitors know the page is missing. |
| `37A-NotFoundAuditor-300` | `WARNING` | The 404 page has no `` tag. | Add `` to the 404 page at `{testUrl}` so search engines do not index it. |
| `37A-NotFoundAuditor-301` | `WARNING` | The 404 page has no navigation menu (`nav`, `.nav`, `.navigation`, `.menu`) and fewer than 3 links. | Add a navigation menu (or at least 3 links) to the 404 page at `{testUrl}` so visitors can navigate to other parts of the site. |
| `37A-NotFoundAuditor-302` | `WARNING` | The 404 page has no links to common destinations such as home, contact, help, or sitemap. | Add links to helpful destinations (e.g. home, contact, help, sitemap) on the 404 page at `{testUrl}`. |
| `37A-NotFoundAuditor-303` | `WARNING` | The 404 page has no search box (`input[type=search]` or a search-like `name`/`id`/`placeholder`). | Add a search box to the 404 page at `{testUrl}` so visitors can search for what they were looking for. |
| `37A-NotFoundAuditor-500` | `ERROR` | The audited URL is invalid, or the generated 404 test URL could not be fetched. | Retry the audit of `{url}`; verify the site is reachable. Last error was `{error}`. |
| `37A-NotFoundAuditor-200` | `SUCCESS` | Every rule above passed: proper status code, custom content, noindex tag, navigation links, helpful links, and a search box. | — |
## Output documentation
| Status | Description | Test logic |
| ----------- | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| **SUCCESS** | The 404 page returns the correct status and follows every content best practice | Status is 404 or 410, and the page has custom content, a noindex tag, navigation links, helpful links, and a search box (`200`). |
| **WARNING** | The status code is correct but the page is missing helpful, non-critical elements | Status is 404 or 410 and content is present, but one or more of noindex tag, navigation links, helpful links, or search box is missing (`300`–`303`). |
| **FAIL** | The status code is wrong, or the page has no real not-found content | Soft 404 (`400`); unexpected status code (`401`); missing custom content (`402`). |
| **ERROR** | The auditor could not generate or fetch the 404 test URL | Invalid audited URL or network/fetch failure (`500`). |
## Risks and considerations
### Soft 404s hide broken links from everyone
* A nonexistent page that returns HTTP 200 looks fine to a casual visitor but is invisible to tools that rely on status codes — broken-link checkers, uptime monitors, and search engine crawlers all misread it as a valid page.
* Soft 404s can also get indexed and compete with real pages in search results, diluting relevance signals for the site.
### Indexed error pages waste crawl budget and confuse rankings
* Without a `noindex` tag, search engines may crawl and index thousands of unique 404 URLs (especially on sites with many broken inbound links), wasting crawl budget that could go to real content.
### A dead end costs a visitor, not just a page view
* A blank or generic 404 page gives a lost visitor nothing to do but leave. Custom messaging, navigation, helpful links, and search all give them a path back into the site instead of bouncing.
### What this auditor does not check
* It does not verify that specific known-bad or previously-existing URLs return 404; it only tests a freshly generated, guaranteed-nonexistent path.
* It does not check HTTP→HTTPS redirects, canonical tags, or sitemap hygiene for stale URLs — those are covered by other auditors.