sitemap.ai
← Crawler identity

Allowlist SitemapAI-Crawler

These instructions are public so site owners and IT teams can approve SitemapAI technical SEO crawls without creating an account.

Approved egress IPs

209.71.104.15

Machine-readable source: /.well-known/crawler.json

Cloudflare

  1. Create an IP list named sitemapai_crawler_ips containing the addresses above.
  2. Create a zone-level WAF custom rule before blocking/challenge rules.
  3. Match the SitemapAI crawler IP.
  4. Use Skip only for the protections preventing the crawl: rate limiting, Super Bot Fight Mode, selected managed rules, or Browser Integrity Check.
ip.src in $sitemapai_crawler_ips

Cloudflare Bot Fight Mode itself cannot be skipped; Super Bot Fight Mode can. Keep the exception scoped to the protections and public paths needed for the crawl.

nginx

geo $sitemapai_ip {
  default 0;
  <EGRESS_IP> 1;
}

# Use $sitemapai_ip with your rate-limit/WAF exception.

Security boundaries

Allowlisting does not grant application authentication. SitemapAI continues to respect robots.txt, crawl only public URLs, apply crawl limits, and back off when the origin is unstable. Never exempt admin, account, checkout, or other private paths solely because a request comes from our shared egress IP.