Direct answer: Rechecking the same URLs 7–12 hours later: zero stable robots flips (one site returned 404 once, then four rechecks returned 200, so it is recorded as transient); zero sitemap status, structure or byte changes (24 files, lastmod included); zero contact-page status or structure changes, but 9 of 20 pages changed at the byte level. A short window, not proof of stability.
This is the C track's first real same-URL recheck. The goal is not to grade sites but to test whether the same UA and the same rules can produce an item-by-item diff — and whether "really changed" can be separated from "looks changed". The answer: a retry step is mandatory, otherwise one transient 404 becomes a "policy reversal".
Method and window
Baselines come from earlier published data: robots (2026-10-31 to 11-01), sitemap (11-01 20:41–20:46) and contact pages (11-01 18:41–18:47). The recheck ran around 2026-11-02 03:40 with the same sample_ids, the same UA LingyiResearchBot/1.0 and the same rules; at most three requests per site, six seconds apart.
| Layer | Rechecked | Window | Compared |
|---|---|---|---|
| robots | 25 | ~12 hours | Per-bot allowed / disallowed / absent |
| sitemap | 24 | ~7 hours | HTTP status, root element, URL count, byte SHA-256 |
| Contact page | 20 | ~9 hours | HTTP status, H1 count, form count, byte SHA-256 |
Result: stable, transient and byte-level
| Layer | Status change | Structure change | Byte change | Note |
|---|---|---|---|---|
| robots | 0 stable | 0 verdict flips | — | One transient 404, recovered, not counted |
| sitemap | 0 | 0 | 0 | All 24 byte-identical |
| Contact page | 0 | 0 | 9 / 20 | H1 and form counts unchanged |
The transient 404: the mistake a recheck invites
One site returned 404 for /robots.txt during the recheck. Under the rules, that becomes "absent" and five bots move from allowed to absent — a policy reversal on paper. But the next four requests all returned 200 with 2,696 identical bytes.
Reporting that as a flip would invent a change that did not happen. This round records it as transient unavailability, excludes it from the flip count and keeps the raw verdicts for review. A recheck needs a retry rule: on a non-200, retry after a few seconds at least three times before calling it a change.
Sitemap zero change: static file, not "no updates"
The 24 sitemaps were byte-identical for about seven hours, lastmod included. They are static files that are not rebuilt on a schedule; alternatively the sites genuinely published nothing in that window. Bytes cannot tell the two apart — which is exactly why lastmod proves what the file says, not when the content changed.
Nine contact pages changed at byte level, none in structure
Nine of 20 contact pages changed hash while H1 and form counts stayed identical. Common causes: timestamps, CSRF tokens, rotating recommendations, CDN injection or A/B tests. For acceptance: a byte hash is good for "is this the same version", not for "did the content really change"; the latter needs structural fields or manual comparison.
Five actionable conclusions
- Retry before declaring change. On a non-200, retry at least three times; log a single failure as transient.
- Record status, structure and bytes separately. They mean different things; folding them into "changed" misleads.
- Byte change is not content change. A dynamic token changes the hash; structure fields are steadier.
- A short window is not a stability claim. 7–12 hours validates the procedure; day/week/month windows support stability.
- Write the window into the record. A recheck without a window cannot be compared or explained.
Limits
- Short window: 6–12 hours; no daily, weekly or monthly claim.
- Single baseline: the baseline is itself one measurement, so its error carries over.
- Byte noise: dynamic content changes hashes; substantive updates cannot be separated.
- Fixed URLs only: new pages, new sitemaps and redesigned paths are not discovered.
- Non-random sample: 25/24/20 sites; a pilot.
Data and recalculation
Anonymised data: equipment-recheck-12h-2026.csv (25 rows of per-sample robots/sitemap/contact change flags, including robots_transient). The repository keeps docs/research/2026-11-recheck-12h.csv.
Diff logic: scripts/recheck_compare.py; tests: scripts/test_recheck_compare.py (10 tests, no network). Run:
python -m unittest scripts.test_recheck_compare
Related: the 30-site robots scan (baseline), the 24-site sitemap scan (baseline) and the contact-path pilot (baseline).
Sources and definitions
- robots verdicts: RFC 9309: rfc-editor.org.
- Baselines: the published CSVs
2026-11-equipment-robots-scan.csv,2026-11-sitemap-scan.csvand2026-11-contact-path-pilot.csv. - Recheck: 2026-11-02 03:40 Asia/Shanghai, UA
LingyiResearchBot/1.0, six seconds between requests.