For web scrapers and researchers, sitemaps are invaluable. Rather than crawling a site page-by-page, you can read its sitemap to get a complete list of URLs upfront, including last-modified dates that tell you which pages have changed recently. — Permalien