You need a list of every page on a website. The sitemap gives you one number, a crawler gives you another, and the person managing the CMS says both look wrong.
That disagreement is useful. Each source sees a different part of the site. A sitemap lists URLs for discovery. A crawler reports what it could find and fetch. A CMS export lists content stored in that system.
The most practical way to find a website's pages is to combine those lists, preserve where each URL came from, and investigate the gaps. Here's how to do that for a content audit, redesign, or migration—without mistaking a plausible page count for a complete inventory.
Which method should you start with?
- XML sitemap — Best use: Quickly collecting the site's declared URLs; Main limitation: May omit sections or contain outdated entries.
- Website crawler — Best use: Finding URLs and checking their current responses; Main limitation: Coverage depends on links, rendering, access, and crawl limits.
- CMS or route export — Best use: Finding published content that discovery methods miss; Main limitation: One system may not manage the entire website.
- Search and traffic records — Best use: Recovering URLs observed in the past; Main limitation: Historical activity is not a complete current inventory.
If you don't own the website, begin with its public sitemap and an appropriately scoped crawl. Without access to its publishing systems, you usually cannot verify that you've found every public page.
1. Decide what counts as a page
Before comparing totals, define the inventory you need. A content audit might include published articles and landing pages. A migration may also need old redirects, downloadable documents, and campaign URLs that still receive traffic.
Write down:
- Hosts: Does the project include
www.example.com,help.example.com, and other subdomains? - Content types: Are you counting HTML pages only, or also PDFs and other files?
- URL variations: Should pagination, filters, language versions, and tracking parameters be separate entries?
- Publication status: Are drafts, private pages, and archived content outside the scope?
Keep the original URLs even if you later group variants. For example, /shoes?page=2 may show different products from /shoes. Removing every query string before you inspect the pages can erase meaningful differences.
2. Find the XML sitemap and follow its index
Start at https://example.com/robots.txt and look for a Sitemap: declaration. If none appears, common locations worth checking include /sitemap.xml and /sitemap_index.xml. These are conventions, so a missing file at either path does not establish that the site has no sitemap.
If you don't know where the sitemap lives, the Sitemap Finder tool checks declarations and common locations. You can also check those paths yourself in a browser. For WordPress-specific locations, see our WordPress sitemap guide.
Check whether the file contains page URLs or an index of other sitemap files:
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-products.xml</loc>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-articles.xml</loc>
</sitemap>
</sitemapindex>In this example, the two entries identify sitemap files. You still need to collect the page URLs inside those files. Record which files you processed; a preview or sample of a large index is not a full export.
Keep a source column for each URL, such as sitemap-products.xml. If products are missing later, you'll know which part of the publishing system to investigate.
A sitemap is useful evidence of intended discovery, but inclusion does not prove a page is live or indexed. Google explicitly says that sitemaps do not guarantee crawling or indexing. Google's sitemap documentation explains that distinction.
3. Crawl the website and record the coverage
A crawler visits URLs and discovers additional ones, commonly through internal links and sitemap files. Its results let you check whether those URLs resolve, redirect, or fail.
Start a website crawl at the main public hostname and export the resulting URL list as CSV. That lets you compare individual URLs alongside your CMS and sitemap lists.
Alongside the export, record the crawl date, starting URL, page limit, excluded paths, and whether the crawl completed. Review failures and blocked requests. A crawl that stops at 500 pages tells you it reached its limit; it doesn't establish that the website has 500 pages.
If a familiar section is absent, inspect a representative listing page. Are its links present in the initial HTML? Do they appear only after JavaScript runs? Does the next batch require clicking a button? Browser rendering can uncover additional links, but it does not automatically reproduce every interaction or logged-in state.
A discovered URL is not necessarily an internally linked URL
Many crawlers use multiple discovery sources. A URL found in a sitemap may appear in the crawl even if the crawler never encountered an internal link to it.
For an orphan-page investigation, retain discovery-source and inlink evidence where available. If your tool combines sources without identifying them, use a separate link-only crawl where supported. Label pages with no observed inlinks as candidates until you confirm adequate crawl coverage.
4. Export published URLs from your CMS
If you own the site, ask for an export of published pages and their public URLs. Include all relevant content types: products, categories, articles, service pages, documentation, and campaign landing pages.
Request the content ID and publication status as well. A list of slugs alone may be insufficient when routes depend on categories, locales, or custom routing rules.
For a custom application, the equivalent may be a route list generated from the production content database. Ask the developer which public routes are created outside the CMS. A marketing site, documentation portal, and ecommerce catalog might each have a different source of truth.
Keep unpublished records separate. The aim is to identify missing public content without accidentally turning drafts or private URLs into crawl targets.
5. Add historical URLs where they help
For a migration or cleanup, add URLs observed in Search Console, analytics landing-page reports, and server access logs. These can reveal an old campaign page or retired article that is no longer linked from the current site.
Attach the source and date range to every historical import. A URL requested last year may redirect today. A path in a server log may be a typo or an automated probe rather than a page that ever existed.
A Google search such as site:example.com is useful for spot checks, but its result list is not exhaustive—even for indexed URLs. Don't use its displayed count as your website's page total. Google documents this limitation.
6. Combine the lists without losing their differences
Put the exports into a spreadsheet and build a master list of distinct URLs. Add columns for presence in each source, then join in current crawl results. Keep the raw exports so you can trace unexpected rows.
Compare the actual URLs before comparing totals. Two lists can each contain 800 entries while disagreeing on 100 of them. A matching count can hide an entire missing section.
A useful inventory has these fields:
- Original URL and comparison URL: Preserve the original; document any normalization used for matching.
- Source flags: Sitemap, crawl, CMS, and historical records.
- Current response: HTTP status, final destination, and check date, where verified.
- Indexing signals: Observed noindex and declared canonical, where collected; leave unknown values explicit.
- Decision: Keep, improve linking, investigate, redirect, or remove from the sitemap.
Don't merge paths solely because their capitalization or trailing slashes look similar. Confirm how the site handles them. Keep canonical grouping separate from exact-URL matching: a declared canonical is a preference signal, not proof that two addresses are interchangeable.
Example: why three lists produce three different counts
Take this example inventory of five URLs.
/— Sitemap: Yes; Crawl: Yes; Published CMS export: Yes; Next check: Verify the response./services— Sitemap: Yes; Crawl: Yes; Published CMS export: Yes; Next check: Verify the response./guides/setup— Sitemap: Yes; Crawl: No; Published CMS export: Yes; Next check: Check crawl coverage and access./campaign/summer— Sitemap: No; Crawl: No; Published CMS export: Yes; Next check: Check whether it is still public and should be linked./old-services— Sitemap: No; Crawl: Yes; Published CMS export: No; Next check: Inspect its response and the link pointing to it.
The sitemap contains three URLs, the crawl found three, and the CMS lists four. Together they identify five distinct URLs to investigate. None of those counts alone tells you how many live, indexable pages exist.
The missing guide could reflect a timeout or crawl exclusion. The campaign could be intentionally excluded from search. The old services URL could redirect correctly. The comparison creates specific questions you can resolve.
What to do with the URLs that don't match
Work through disagreements by section. If every missing URL starts with /guides/, inspect the guides index, its sitemap file, and the CMS settings for that content type. That is usually a more productive starting point than checking 200 guide pages individually.
Separate “not found by this crawl” from “not indexed by Google.” For a site you manage, Search Console's URL Inspection tool can help with the second question. Its indexed result and live test also differ: a successful live test does not establish that the URL is indexed. See Google's URL Inspection guidance.
Don't automatically add every unmatched URL to your sitemap. An old redirect, a private campaign page, and a newly published guide need different decisions. Find out why the URL exists and what should happen when someone visits it.
When is the inventory good enough to use?
For a small content cleanup, a completed crawl reconciled with published CMS records may be enough. Before a migration, spend more time on historical URLs and sections managed outside the main CMS: those are easy to forget when the new site launches.
Save a short coverage note with the spreadsheet. For example: “Main website and help subdomain; published HTML pages; all listed sitemap files processed; CMS export from October 18; three blocked URLs still unresolved.”
That note makes the inventory usable by someone who wasn't involved in collecting it. They can see what the list covers, which gaps remain, and where to look when another page turns up.