Direct answer: Of the same 30 equipment makers, 24 sites could be tested. Nine (37.5%) declare a sitemap in robots.txt. In the end 13 sitemaps were usable (12 urlset + 1 sitemapindex, 54.2%), 9 returned 404, and 2 returned 200 without being a sitemap — one empty body and one HTML page. The 12 urlsets have a median of 206.5 URLs (107–1,098), and all 13 usable files carry lastmod. This is a pilot, not a benchmark.
Earlier rounds looked at pages: robots policy, product entry pages, contact paths. This one looks at how a site hands its URL list to crawlers. It is the most overlooked part of discovery, and it is well suited to verdicts based on status codes and XML roots.
Method: two requests per site
The frame follows the earlier scan: positions 71–100 of the 2024 Changzhou first-set major equipment list, 30 companies. Four had no identifiable official site, one robots.txt returned 502, and one returned 503 on the scan day and was skipped conservatively, leaving 24.
Each site: fetch /robots.txt, read every Sitemap: line, fetch the first declared URL, or fall back to /sitemap.xml when nothing is declared. Only one sitemap per site; sitemapindex children are not followed. UA LingyiResearchBot/1.0, requests 6 seconds apart, 20-second timeout, no JavaScript. Raw responses stay outside the repository.
| Rule | Definition |
|---|---|
| Declared | At least one Sitemap: line in robots.txt, case-insensitive, first token after the colon |
| Usable | HTTP 200 and parses as XML whose root is urlset or sitemapindex |
| URL count | url nodes under a urlset; an index records child sitemaps, not pages |
| lastmod | A direct child lastmod of a url node |
| Same host | Every loc host matches the site host; an empty list is false |
Result 1: declared is not the same as usable
| Link in the chain | Count | Note |
|---|---|---|
| robots.txt returns 200 | 16 / 24 | 8 sites return 404 for robots.txt |
| robots.txt declares a sitemap | 9 / 24 | One declares 20 URLs at once (child sitemaps of an index) |
| Declared sitemap is usable | 8 / 9 | One declared URL returns 404: writing an address is not creating the file |
| No declaration, /sitemap.xml fallback usable | 5 / 15 | A file can exist without a declaration, but less often here |
| Fallback returns 200 but is not a sitemap | 2 / 15 | One empty body, one HTML page |
| Fallback returns 404 | 8 / 15 | No file at all |
robots.txt alone overstates. Nine declare, eight are usable. A status-code check alone understates too: two fallbacks returned 200 with an empty body or HTML. Both questions matter: what is declared, and what comes back that actually parses.
One more detail: of the 8 sites without robots.txt, only 1 still served a usable file at /sitemap.xml. Missing robots.txt and missing sitemap overlap heavily here, though a sample cannot establish cause.
Result 2: what the 13 usable files look like
| Feature | Result |
|---|---|
| Usable files | 13: 12 urlset + 1 sitemapindex |
| URLs per urlset | Median 206.5, min 107, max 1,098 |
| sitemapindex | 1 file with 20 child sitemaps; not followed, so page count is unknown |
| Carries lastmod | 13 / 13 |
| All locs same host | 12 / 13; one lists a different domain of the same site |
| Median response size | 27.8 KB across 200 responses |
lastmod at 13/13 is an interesting contrast: the file exists and the field is filled, but "has lastmod" is not "lastmod is recent". This scan records presence only, not freshness, and never infers an update date from a copyright year.
The file whose locs all point to another domain deserves a note: when the site host and the listed host differ, the search engine treats each URL by its own host. If both domains serve the same content, that can split crawling or create duplicates. This scan records "different host" and stops there.
Result 3: two kinds of fake sitemap
- Empty body. HTTP 200 with a
text/xmlcontent type and zero bytes. A status-only monitor calls it healthy. - HTML page. HTTP 200 returning 10 KB of
text/html, most likely a soft-404 or a front-end fallback. XML parsing fails immediately.
These are exactly what automated checks miss, because "200 = fine" is the default assumption. A sitemap verdict needs three things: status code, content type, and root element.
Five actionable conclusions
- Declare the sitemap in robots.txt. 8/9 declared files were usable, against 5/15 that relied on the fallback.
- Fetch what you declared. One site declared a 404 URL; no crawler will fix that for you.
- Judge by the XML root. 200 plus
text/xmlcan still be an empty file; parse and checkurlset/sitemapindex. - Check the loc host. 1 of 13 pointed entirely at another domain; nothing errors, crawling just scatters.
- Do not read lastmod as freshness. Every usable file carries it, and none of it was verified here.
Limits
- Non-random sample: one list, one city, list-order selection, 24 testable sites; a pilot, not the industry.
- First declaration only: sites may split sitemaps by section; not enumerated.
- No index recursion: a
sitemapindexis counted as child sitemaps, not pages. - Single fetch: one site's robots.txt returned 503 on the day and was skipped; transient states change the denominator.
- No indexing claim: a usable sitemap says nothing about crawling, indexing or traffic.
- No naming: the public data carries sample_id and fields only, never company names, domains or file paths.
Data and recalculation
Anonymised data: equipment-sitemap-scan-2026.csv (24 rows, 15 fields, including sample_id and capture time). The repository also keeps docs/research/2026-11-sitemap-scan.csv.
Probe and rules: scripts/sitemap_probe.py; tests: scripts/test_sitemap_probe.py (16 tests, no network). Run:
python -m unittest scripts.test_sitemap_probe
Related: the 30-site robots scan (same frame), the per-bot robots matcher and the 12 acceptance clauses.
Sources and definitions
- Sitemap protocol (sitemaps.org): sitemaps.org.
- robots.txt and Sitemap declarations: RFC 9309: rfc-editor.org.
- Capture window: 2026-11-01 20:41–20:46 Asia/Shanghai, UA
LingyiResearchBot/1.0, two requests per site, 6 seconds apart.