Direct answer: using a fixed discovery rule across 25 sites, 15 sites yielded a usable sitemap and 73 URLs were sampled: 67 (91.8%) returned 200, 5 (6.8%) returned 404, and one timed out on all three attempts. Twelve sites were fully 200; three had at least one problem, with four of the five 404s on a single site (E11).
Five responses redirected (all on E29, none cross-host) and two returned 200 with bodies under 500 bytes. Discovery found no usable sitemap on 10 sites (eight 404s, two 200 responses with no <loc>); 10 sitemaps came from a robots declaration and five from the default path.
Method: fixed discovery, five URLs per site, three attempts
The sites are the 25 equipment manufacturers from study 4 that had an analysable homepage. A sitemap is discovered in only two ways: a Sitemap line in robots.txt, or the default /sitemap.xml; for a sitemap index the first child is used. After parsing <loc>, five URLs are picked at even positions (all if fewer), fetched serially with at least 0.6 s between requests and redirects followed. Non-200 responses and bodies under 500 bytes are retried twice (three attempts total) to confirm the status.
| Item | Rule | Boundary |
|---|---|---|
| Discovery | robots-declared sitemap or /sitemap.xml | Other paths not tried |
| Sampling | Five evenly spaced positions | Not a share of the whole sitemap |
| Retries | Two extra attempts for non-200 or <500 bytes | 404 is deterministic |
| Redirect | Final URL differs from the sitemap URL (fragment ignored) | Canonicality not judged |
| Classification | 200 / 404 / other 4xx / 5xx / network error | No cause inferred |
Result: nine in ten sampled URLs open
| Outcome | Count | Share of 73 |
|---|---|---|
| 200 | 67 | 91.8% |
| 404 | 5 | 6.8% |
| Network error (3 timeouts) | 1 | 1.4% |
| 200 after redirect (included in 67) | 5 | 6.8% |
| 200 with body under 500 bytes (included in 67) | 2 | 2.7% |
All five redirects belong to E29 and stay on the same host; no other site redirected across hosts. The two small 200s are 88 and 138 bytes, possibly interstitials or placeholders, and are still counted as 200.
Problems cluster on three sites
| Site | Sampled | 200 | 404 | Network error | Note |
|---|---|---|---|---|---|
| E11 | 5 | 1 | 4 | 0 | All four 404 pages are 548 bytes |
| E16 | 5 | 4 | 1 | 0 | One 404 with a 4.7 KB body |
| E02 | 5 | 4 | 0 | 1 | The same URL timed out three times |
The four E11 404s reproduce consistently, so those sitemap entries cannot be opened on the current site; E02's URL timed out on all three attempts, which is a network error, not a 404.
Sitemap discovery: 15 of 25 usable
| Case | Sites |
|---|---|
| Robots-declared sitemap, usable | 10 |
Default /sitemap.xml, usable | 5 |
| Both paths return 404 | 8 |
200 but no <loc> | 2 |
Usable sitemaps list between 3 and 1,043 URLs, with a median around 180; E18 lists only three, so it contributed three samples. Sitemap paths are outside this study's scope — variants such as /sitemap_index.xml were not tried, so "no usable sitemap" is a finding of this rule, not of the site.
Compared with the study-2 structure scan
| Measure | Study 2 (structure) | This study (resolution) |
|---|---|---|
| Sites | 25 | 25 |
| Usable sitemaps | 13 | 15 (slightly wider rule) |
| What was checked | XML validity, URL count, lastmod | HTTP result of sampled URLs |
| Sampled URLs | 0 | 73 |
| Problems found | Counts and lastmod coverage | 5 × 404, 1 × timeout |
Study 2 described the files; this one asks whether the listed URLs open. The same sites four days apart moved from 13 to 15 usable sitemaps, which shows that discovery rules and site state both move the number and should not be read as an improvement.
What this means for acceptance
- Make "sitemap entries open" part of acceptance. A file existing is not the same as working entries; 6 of 73 were unusable or odd.
- A five-URL sample finds clusters. Five per site were enough to expose E11's four 404s at low cost.
- Keep 404 and timeout apart. E02's URL timed out three times and must not be reported as missing.
- Watch small 200s. Two bodies under 500 bytes still return 200 and may be interstitials or placeholders.
- Count redirects separately. Five redirected responses stayed on the same host but no longer matched the sitemap URL.
Limits
- Fixed discovery: only robots declarations and
/sitemap.xml; sitemaps at other paths may be missed. - Five per site: a signal of whether problem entries exist, not a failure rate for the whole file.
- One snapshot: captured on the evening of 2026-11-05; timeouts and protections vary with time and network.
- No cause inferred: a 404 means the URL does not open now, not why.
- 200 is not content quality: JS was not rendered and soft 404s were not judged.
- 25 sites: a pilot, no industry claim.
Data and reproduction
Anonymised data: equipment-sitemap-url-resolution-2026.csv (73 rows × 13 columns: site id, sitemap source and status, URL count, sample index, HTTP status, result class, redirect, cross-host, bytes, small-body flag, attempts, error type). The repository keeps docs/research/2026-11-sitemap-url-resolution.csv; the public CSV contains no URLs.
Parser and classifier: scripts/sitemap_url_probe.py; unit tests: scripts/test_sitemap_url_probe.py (no network, 12 tests). Run:
python -m unittest scripts.test_sitemap_url_probe
Related: the sitemap structure pilot, the robots scan, and the sitemap checker.
Sources and definitions
- Sites: the 25 analysable homepages from study 4 (2026-10-31).
- Capture: 2026-11-05 22:41–22:44, serial, ≥0.6 s apart, redirects followed.
- Retries: three attempts for non-200 responses and bodies under 500 bytes.