Key takeaways
- To monitor sitemap changes, save a snapshot of a site's sitemap, fetch it again later, and compare the two. Look for new URLs, missing URLs and changes to reported lastmod dates.
- It is one of the cheapest ways to watch a competitor's new pages, a publisher's new articles or your own site for accidental removals.
- The hard part is storing snapshots and producing a clean list of changes every run.
- My Sitemap Change Monitor does this at $0.01 per change row, and a run with no changes produces no change rows.
Want a change feed without building one? Run the Sitemap Change Monitor on Apify.
What a sitemap tells you
A sitemap is an XML file a website publishes to list its pages. Each entry has a URL in a <loc> tag and often a <lastmod> date saying when the page last changed. Big sites usually split this into several sitemaps tied together by a sitemap index file. You can often find a site's sitemap at /sitemap.xml or listed in its robots.txt.
A sitemap lists the pages the site chooses to show search engines. Its coverage and the accuracy of its lastmod dates depend on what the site publishes.
What "monitoring sitemap changes" means
Monitoring sitemap changes means checking the same sitemap on a schedule and recording three kinds of change.
- Added: a URL appears that was not there last time.
- Removed: a URL that was there last time is gone.
- Lastmod changed: the URL is still there but its lastmod date moved.
These three signals can help you spot new product pages, new blog posts, retired pages and reported updates.
How to compare sitemap snapshots
- Find the sitemap. Try
/sitemap.xmland checkrobots.txtfor aSitemap:line. - Handle sitemap indexes. If the file lists other sitemaps instead of pages, fetch each child sitemap too.
- Build the snapshot. Collect every
<loc>with its<lastmod>into a URL-to-date map and save it. - Fetch again later. Build a fresh map the same way.
- Diff the two maps. URLs only in the new map were added. URLs only in the old map were removed. URLs in both with a different lastmod were changed.
- Emit one row per change and replace the old snapshot with the new one.
The logic fits in a short script. Keeping the snapshots, the schedule and the edge cases (compressed sitemaps, nested indexes, sites that time out) working long term is the real work.
Your options compared
| Feature | Checking the sitemap by hand | Your own script | Sitemap Change Monitor (Apify) |
|---|---|---|---|
| Detects added URLs | Only if you compare by eye | If you write the diff | Yes |
| Detects removed URLs | Hard to spot by eye | If you write the diff | Yes |
| Detects lastmod changes | Hard to spot by eye | If you write the diff | Yes |
| Stores the previous snapshot | You save a copy | If you build storage | Yes |
| Output | Notes | Whatever you write | One record per change |
| Price | Your time | Your time plus hosting | $0.01 per change row |
Which option is best for whom
- Checking by hand is best for a quick look at a small site's sitemap once. Not for anything with hundreds of URLs, because spotting a removed URL by eye is close to impossible.
- Your own script is best for developers who want to own the code and already run scheduled jobs. Not for people who do not want to maintain snapshot storage and edge cases.
- The Sitemap Change Monitor is best for SEOs, content teams and researchers who want a clean added, removed and changed feed from public sitemaps. Not for tracking pages a site leaves out of its sitemap, because it only sees what the sitemap lists.
A worked example with real numbers
I live-tested the monitor on Apple's sitemap, which had 871 URLs at the time. The first run built the baseline snapshot. The second run correctly reported zero changes. The second run checked whether the monitor would report changes when there were none.
The published live test covers baseline creation and the no-change case.
At the listed rate, a run that finds 40 changes costs $0.40 in change rows. A run with zero changes produces zero change rows.
Point it at a sitemap you care about: open the Sitemap Change Monitor on Apify.
How to choose
- One small site, one look: open the sitemap in your browser.
- You run your own scheduled jobs and want full control: build it with the method above.
- You want an added, removed and changed feed across one or many sitemaps: use the Sitemap Change Monitor.
"Why not just crawl the whole site?"
Crawling finds pages the sitemap leaves out, so it can be more complete. It is also slower, heavier on the site and much more work to diff. If the site keeps a decent sitemap, watching it gets you most of the signal for a fraction of the effort. If you need pages the sitemap does not list, you need a crawler instead.
FAQ
How do I find a website's sitemap?
Try /sitemap.xml on the domain first. If that does not work, open /robots.txt and look for a line starting with Sitemap:.
How can I see when a competitor adds new pages?
Monitor their sitemap and watch for added URLs. The Sitemap Change Monitor emits one record for each added, removed or lastmod-changed URL against the previous snapshot.
What does lastmod mean in a sitemap?
It is the date the site says a page last changed. It is only as accurate as the site keeps it, so treat it as a hint rather than proof.
Do I pay when nothing changed?
The price is $0.01 per change row, so a run with zero changes produces zero change rows. Check the store page for the full pricing terms.
Can I monitor sitemap changes for my own site?
Yes. It is a good way to catch pages that were removed from your sitemap by accident.
Start monitoring
Sitemap Change Monitor on the Apify Store, $0.01 per change row.
Read more on the Sitemap Change Monitor product page.
Checklist sign-off. The guide uses the primary keyword in its slug, title and H1, names the author, gives both dates, leads with the answer, defines the topic, explains the method, compares three options in an HTML table, describes who each option suits, works through the reported Apple test and listed-price arithmetic, explains how to choose, addresses crawling, answers five FAQs and links to the product page and Apify with four calls to action. The worked example uses the live-test results reported in the promotion brief.