Skip to main content
SEO Auditor — locates the XML sitemap of the audited site (through robots.txt or the usual paths), parses it and checks that its latest <lastmod> is recent.

Discovery

Reads Sitemap: lines in robots.txt, then tries /sitemap.xml, /sitemap_index.xml and /sitemap/sitemap.xml.

Standards

Follows the Sitemaps XML format, protocol version 0.9, published at sitemaps.org.
An XML sitemap tells search engines which pages a site has and when each one last changed. Crawlers find it through the Sitemap: line of robots.txt or at a well-known path, and use its <lastmod> dates to decide what to recrawl. A missing sitemap leaves discovery to link following; a sitemap whose dates stopped moving tells crawlers nothing new is there. This auditor never fetches the audited page itself: it only reads robots.txt and the candidate sitemaps.

​
How it works

  1. Site root — takes scheme://host[:port] from the audited URL. A URL without a scheme or hostname ends the audit with 501.
  2. robots.txt — fetches <root>/robots.txt (as 37AuditsBot, with the site’s Basic Auth when configured) and collects every Sitemap: line. The probe never calls ensureNotBlocked: a 403 or any other answer on robots.txt is recorded in issues, not treated as a blocked audit.
  3. Candidates — the robots.txt URLs, or when there are none, /sitemap.xml, /sitemap_index.xml and /sitemap/sitemap.xml under the site root.
  4. Parse — fetches each candidate in turn (redirects followed, bodies up to 50 MB, 30 s per request, gzip detected by magic bytes even without Content-Encoding) until one answers HTTP 2xx with a <urlset> or <sitemapindex> root. Child sitemaps of an index are not fetched. The entries’ <lastmod> values are parsed (W3C Datetime with or without time zone, date-only read as end of that day) and the latest one is kept; values that do not parse are listed in issues.
  5. Rules — no candidate parsed fails (400). On the parsed sitemap, a latest <lastmod> older than the freshness window warns (300), and no parsable <lastmod> at all warns (301). Independently, a robots.txt that answered HTTP 200 without a Sitemap: line warns (302), even when a fallback path yielded a sitemap.
  6. Freshness window — SITEMAP_MAX_AGE_DAYS (Lambda environment, default 7 days).
  7. Aggregate — FAIL wins over WARNING; a sitemap whose latest <lastmod> is within the window emits 200, and the audit is SUCCESS when no warning fired.
  8. Specification pin — every run emits 37A-SitemapAuditor-100 (INFORMATION), naming the Sitemaps XML format, protocol version 0.9. It does not change the overall status.

​
What it audits

Every row maps to one checkRule* method and one CheckCode, in the order the auditor runs them. CheckCode suffixes use status ranges: 100 information, 200 success, 300–399 warning, 400–499 fail, 500–599 error.

​
Output documentation

Every check carries the same data map: siteBase, robotsTxtUrl, robotsTxtHttpStatus, robotsTxtListsSitemap, candidateSitemaps, selectedSitemapUrl, sitemapType (urlset or sitemapindex), sitemapHttpStatus, contentType, entryCount, latestLastmod (ISO instant or null), maxAgeDays and issues.

​
Risks and considerations

​
Why the sitemap matters

  • Search engines discover and prioritise pages from the sitemap; a site without one relies on internal links alone, and pages far from the home page may never be crawled.
  • <lastmod> is the signal crawlers use to recrawl changed pages first. Dates that never move, or are missing, make every page look equally stale.
  • Listing the sitemap in robots.txt is the one discovery mechanism every crawler shares; Search Console submissions only reach Google.

​
Freshness window

  • The window is SITEMAP_MAX_AGE_DAYS (default 7 days). A brochure site that rarely changes will warn with 300 even though nothing is wrong with its sitemap; the warning is a prompt to keep <lastmod> honest, not a defect of the site.
  • Date-only <lastmod> values (2026-10-16) are read as the end of that day in UTC, so a sitemap updated today is always inside the window.

​
What this auditor does not check

  • Only the first sitemap that parses is examined. Child sitemaps of a <sitemapindex> are not fetched, so their own <lastmod> values and URL counts are not seen.
  • The URLs listed in the sitemap are not fetched or validated, and the 50,000-URL and 50 MB limits of the protocol are not enforced.
  • Sitemaps in other formats (RSS, Atom, plain-text URL lists) are reported as an unknown root element and do not count.
  • A robots.txt that is missing (404) or blocked (403) is not a finding by itself; the auditor just falls back to the usual paths.

​
Changes from the previous version

  • An invalid audited URL (no scheme or hostname) is now an ERROR with 37A-SitemapAuditor-501; it used to be a FAIL with no check.
  • A sitemap without any parsable <lastmod> now gets its own warning, 37A-SitemapAuditor-301; it used to be reported as “out of date” together with the stale case.
  • A robots.txt that answers HTTP 200 without a Sitemap: line now warns with 37A-SitemapAuditor-302, so a site whose sitemap was only found at a fallback path is a WARNING rather than a SUCCESS.
  • The available field is gone from the check data; selectedSitemapUrl is null when no sitemap parsed.