Practical guide · 5 minute read

How to Monitor Sitemap Changes on Any Website

Compare public sitemap snapshots to spot added URLs, removed URLs and changes to lastmod dates.

Key takeaways

  • To monitor sitemap changes, save a snapshot of a site's sitemap, fetch it again later, and compare the two. Look for new URLs, missing URLs and changes to reported lastmod dates.
  • It is one of the cheapest ways to watch a competitor's new pages, a publisher's new articles or your own site for accidental removals.
  • The hard part is storing snapshots and producing a clean list of changes every run.
  • My Sitemap Change Monitor does this at $0.01 per change row, and a run with no changes produces no change rows.

Want a change feed without building one? Run the Sitemap Change Monitor on Apify.

What a sitemap tells you

A sitemap is an XML file a website publishes to list its pages. Each entry has a URL in a <loc> tag and often a <lastmod> date saying when the page last changed. Big sites usually split this into several sitemaps tied together by a sitemap index file. You can often find a site's sitemap at /sitemap.xml or listed in its robots.txt.

A sitemap lists the pages the site chooses to show search engines. Its coverage and the accuracy of its lastmod dates depend on what the site publishes.

What "monitoring sitemap changes" means

Monitoring sitemap changes means checking the same sitemap on a schedule and recording three kinds of change.

  • Added: a URL appears that was not there last time.
  • Removed: a URL that was there last time is gone.
  • Lastmod changed: the URL is still there but its lastmod date moved.

These three signals can help you spot new product pages, new blog posts, retired pages and reported updates.

How to compare sitemap snapshots

  • Find the sitemap. Try /sitemap.xml and check robots.txt for a Sitemap: line.
  • Handle sitemap indexes. If the file lists other sitemaps instead of pages, fetch each child sitemap too.
  • Build the snapshot. Collect every <loc> with its <lastmod> into a URL-to-date map and save it.
  • Fetch again later. Build a fresh map the same way.
  • Diff the two maps. URLs only in the new map were added. URLs only in the old map were removed. URLs in both with a different lastmod were changed.
  • Emit one row per change and replace the old snapshot with the new one.

The logic fits in a short script. Keeping the snapshots, the schedule and the edge cases (compressed sitemaps, nested indexes, sites that time out) working long term is the real work.

Your options compared

FeatureChecking the sitemap by handYour own scriptSitemap Change Monitor (Apify)
Detects added URLsOnly if you compare by eyeIf you write the diffYes
Detects removed URLsHard to spot by eyeIf you write the diffYes
Detects lastmod changesHard to spot by eyeIf you write the diffYes
Stores the previous snapshotYou save a copyIf you build storageYes
OutputNotesWhatever you writeOne record per change
PriceYour timeYour time plus hosting$0.01 per change row

Which option is best for whom

  • Checking by hand is best for a quick look at a small site's sitemap once. Not for anything with hundreds of URLs, because spotting a removed URL by eye is close to impossible.
  • Your own script is best for developers who want to own the code and already run scheduled jobs. Not for people who do not want to maintain snapshot storage and edge cases.
  • The Sitemap Change Monitor is best for SEOs, content teams and researchers who want a clean added, removed and changed feed from public sitemaps. Not for tracking pages a site leaves out of its sitemap, because it only sees what the sitemap lists.

A worked example with real numbers

I live-tested the monitor on Apple's sitemap, which had 871 URLs at the time. The first run built the baseline snapshot. The second run correctly reported zero changes. The second run checked whether the monitor would report changes when there were none.

The published live test covers baseline creation and the no-change case.

At the listed rate, a run that finds 40 changes costs $0.40 in change rows. A run with zero changes produces zero change rows.

Point it at a sitemap you care about: open the Sitemap Change Monitor on Apify.

How to choose

  • One small site, one look: open the sitemap in your browser.
  • You run your own scheduled jobs and want full control: build it with the method above.
  • You want an added, removed and changed feed across one or many sitemaps: use the Sitemap Change Monitor.

"Why not just crawl the whole site?"

Crawling finds pages the sitemap leaves out, so it can be more complete. It is also slower, heavier on the site and much more work to diff. If the site keeps a decent sitemap, watching it gets you most of the signal for a fraction of the effort. If you need pages the sitemap does not list, you need a crawler instead.

FAQ

How do I find a website's sitemap?

Try /sitemap.xml on the domain first. If that does not work, open /robots.txt and look for a line starting with Sitemap:.

How can I see when a competitor adds new pages?

Monitor their sitemap and watch for added URLs. The Sitemap Change Monitor emits one record for each added, removed or lastmod-changed URL against the previous snapshot.

What does lastmod mean in a sitemap?

It is the date the site says a page last changed. It is only as accurate as the site keeps it, so treat it as a hint rather than proof.

Do I pay when nothing changed?

The price is $0.01 per change row, so a run with zero changes produces zero change rows. Check the store page for the full pricing terms.

Can I monitor sitemap changes for my own site?

Yes. It is a good way to catch pages that were removed from your sitemap by accident.

Start monitoring

Sitemap Change Monitor on the Apify Store, $0.01 per change row.

Read more on the Sitemap Change Monitor product page.

Checklist sign-off. The guide uses the primary keyword in its slug, title and H1, names the author, gives both dates, leads with the answer, defines the topic, explains the method, compares three options in an HTML table, describes who each option suits, works through the reported Apple test and listed-price arithmetic, explains how to choose, addresses crawling, answers five FAQs and links to the product page and Apify with four calls to action. The worked example uses the live-test results reported in the promotion brief.