sitemap.ai

SitemapAI Crawler

SitemapAI-Crawler helps site owners generate and validate XML sitemaps, discover public internal URLs, and identify crawl issues such as broken links, redirects, and missing metadata.

Requests identify the crawler with this HTTP header:

X-Crawler-Name: SitemapAI-Crawler

SitemapAI-Crawler uses browser-compatible User-Agent values and applies the wildcard User-agent: * robots.txt rules, including Allow, Disallow, Sitemap, and crawl-delay directives. If a site blocks the crawler through robots.txt or network protection, the crawler stops rather than attempting to bypass the site owner's preference.

Requests include identity headers such as X-Crawler-Name, X-Crawler-Purpose, X-Robots-Policy, and, when configured, a From contact address for operational issues.

Robots.txt

A wildcard rule applies to SitemapAI along with other crawlers. For example, this blocks all standards-compliant crawlers:

User-agent: *
Disallow: /

To allow or block SitemapAI specifically, match the published egress IPs or the X-Crawler-Name: SitemapAI-Crawler header at the firewall, CDN, or reverse proxy.

Firewall and WAF allowlisting

Public Cloudflare and server-administration instructions, including the current approved egress IP list, are available to everyone.

View allowlisting instructions

Owner-Authorized Crawls

Site owners can authorize SitemapAI crawls at their WAF, CDN, reverse proxy, or application server by matching a private request header configured for their account. For stronger verification, SitemapAI can also send a per-request HMAC signature over the request method, URL, and timestamp.

X-SitemapAI-Timestamp: <unix timestamp>
X-SitemapAI-Signature: <base64url hmac-sha256>
X-SitemapAI-Signature-Input: method url timestamp; alg=hmac-sha256

Signature input format:

METHOD
https://example.com/path
TIMESTAMP