Home › Actors › Every URL in a sitemap, with an optional broken-link check › Data and schema
This is the machine-readable side of Every URL in a sitemap, with an optional broken-link check: the columns its dataset contains, taken from a real sample run of 12 rows and 16 columns. Fill rate counts how many of the 12 sample rows have a value. Contact fields are withheld here.
| column | type | filled | example |
|---|---|---|---|
| recordType | text | 12/12 | url |
| site | text | 12/12 | crawlee.dev |
| url | text | 12/12 | https://crawlee.dev/blog |
| changefreq | text | 12/12 | weekly |
| priority | number | 12/12 | 0.5 |
| sourceSitemap | text | 12/12 | https://crawlee.dev/sitemap.xml |
| sitemapDepth | integer | 12/12 | 0 |
| pathGroup | text | 12/12 | /blog |
| pathDepth | integer | 12/12 | 1 |
| hasQuery | boolean | 12/12 | False |
| imageCount | integer | 12/12 | 0 |
| videoCount | integer | 12/12 | 0 |
| hreflangCount | integer | 12/12 | 0 |
| robotsDisallowed | boolean | 12/12 | False |
| isDuplicate | boolean | 12/12 | False |
| checked | boolean | 12/12 | False |
| recordType | site | url | changefreq | priority | sourceSitemap | sitemapDepth | pathGroup |
|---|---|---|---|---|---|---|---|
| url | crawlee.dev | https://crawlee.dev/blog | weekly | 0.5 | https://crawlee.dev/sitemap.xml | 0 | /blog |
| url | crawlee.dev | https://crawlee.dev/blog/archive | weekly | 0.5 | https://crawlee.dev/sitemap.xml | 0 | /blog |
| url | crawlee.dev | https://crawlee.dev/blog/authors | weekly | 0.5 | https://crawlee.dev/sitemap.xml | 0 | /blog |
| url | crawlee.dev | https://crawlee.dev/blog/building-gradcracker-scraper-with-c... | weekly | 0.5 | https://crawlee.dev/sitemap.xml | 0 | /blog |
| url | crawlee.dev | https://crawlee.dev/blog/common-problems-in-web-scraping | weekly | 0.5 | https://crawlee.dev/sitemap.xml | 0 | /blog |
Download the full sample CSV (raw file on GitHub). Long values are shortened above.
From the actor's current input schema on Apify.
| field | type | label | default |
|---|---|---|---|
| startUrls (required) | array | Domains or sitemap URLs | ["https://crawlee.dev"] |
| maxUrlsPerSite | integer | Max URLs per site | 1000 |
| checkMode | string | Link check mode | "off" |
| checkPageStatus | boolean | Check page status (HTTP 200/404/redirect) | false |
| maxSitemapFiles | integer | Max sitemap files per site | 50 |
| maxCrawlingDepth | integer | Max crawling depth (nested sitemap levels) | 6 |
| includeDuplicates | boolean | Emit duplicate URL rows | false |
| pageUrls | array | Pages to scan (pageLinks mode) | |
| maxLinksToCheck | integer | Max links to check | 100 |
| checkIndexability | boolean | Read noindex signals | true |
| checkCanonical | boolean | Flag canonical mismatches | false |
| checkConcurrency | integer | Check concurrency | 5 |
| requestDelayMs | integer | Delay between requests to one host (ms) | 300 |
| requestTimeoutSecs | integer | Request timeout (seconds) | 20 |
| timeBudgetSecs | integer | Time budget (seconds) | 540 |
| userAgent | string | User-Agent | "SitemapLinkInventoryBot/1.0 (+https://a |
| monitorMode | boolean | Monitor mode (only new or changed) | false |
| reportRemoved | boolean | Also report removed (monitor mode) | false |
| monitorName | string | Monitor name | "default" |
$0.001 per sitemap file parsed (pay per event). Read from the Apify actor record on 2026-10-14; the live price on the actor page always wins.
Successful builds of the published actor, newest first (16 builds in total). Apify records build numbers and dates, not release notes, so this lists what shipped and when.
Run it on Apify About this actor
Disclosure: we build and maintain this actor.
This actor
Comparison
Guides with real sample rows from this actor
Some links are affiliate links.