Direct answer: Fifteen homepages were checked against robots first, then requested once with each of five AI crawler UAs. Fourteen are comparable: 5 identical (status, bytes, title), 7 byte-only differences with the same title, 1 returns 403 to Bytespider, and 1 fails to connect for two OpenAI UAs (via a local proxy, cause unknown). One site disallowed all five bots, so no bot requests were sent.
The earlier robots scan measured the declaration; this one measures delivery: same path, different UA. Most sites treat them the same, but there is a real gap between what robots says and what the server does.
Method: robots first, then the request
The first 15 analysable homepages by sample_id, chosen deterministically. Each site's robots.txt is fetched fresh with the own UA; a bot UA is sent only when that bot is allowed (or robots is absent). Disallowed or unknown bots are skipped, not bypassed.
UAs: GPTBot, OAI-SearchBot, PerplexityBot, Bytespider and ClaudeBot, each with a realistic version and info URL. At most seven requests per site, six seconds apart, 20-second timeout, no cookies, no JavaScript. We record status, final URL, body SHA-256, <title> and bytes.
| UA | 200 | non-200 | robots skips |
|---|---|---|---|
| Own (control) | 15 | 0 | 0 |
| GPTBot | 13 | 1 connection failure | 1 |
| OAI-SearchBot | 13 | 1 connection failure | 1 |
| PerplexityBot | 14 | 0 | 1 |
| Bytespider | 13 | 1× 403 | 1 |
| ClaudeBot | 14 | 0 | 1 |
All five skips come from one site that blocks every bot with a single rule; no bot request was sent there.
Site level: three outcomes
| Type | Sites | What happened |
|---|---|---|
| Identical | 5 | Same status, bytes and title for all five bots |
| Byte-only | 7 | Same status and title, different hash; mostly the same byte size |
| Status difference | 2 | One 403, one connection failure |
| Not comparable | 1 | robots blocked all five; only the own UA was sent |
The byte-only group is easy to misread. In most of those seven the byte size is identical to the own UA (e.g. 214,244 and 137,484 bytes) and only the hash differs — timestamps, random recommendations or CSRF tokens rather than a different page for AI. It is not recorded as cloaking.
Two cases
Bytespider 403. One site returns 403 (a 307-byte Forbidden page) to Bytespider while returning 200 with identical content to the own UA and the other four bots. Three follow-up attempts gave the same result. That is a clear UA-related difference, though it may be a generic WAF rule rather than an AI-specific block.
OpenAI UA connection failure. On another site, GPTBot and OAI-SearchBot requests raised ProxyError through the local HTTP proxy, while PerplexityBot, Bytespider, ClaudeBot and the own UA succeeded at the same time. Three attempts matched. The site's WAF and the local proxy cannot be separated here, so it is recorded as an observation, not as "the site blocks OpenAI".
Five actionable conclusions
- robots allow does not guarantee content. One site returned 403 to Bytespider despite allowing it.
- Check UA rules before page content. 403s and connection failures usually sit in the WAF layer.
- Read byte differences with title and size. Same title and size usually means dynamic content, not cloaking.
- Run a small UA test. Two or three requests per site are enough to spot obvious UA blocking.
- Record the environment. Proxy, region and time all matter; the proxy error here is an example.
Limits
- Single request: WAF and CDN behaviour may vary by rate and region; only the two anomalies were retried three times.
- Proxy variable: requests go through a local HTTP proxy, so a connection failure cannot be fully attributed to the site.
- Homepage only: deep links, APIs and forms are out of scope.
- robots skips: one site blocked all five bots, shrinking the sample.
- 15-site pilot: deterministic sample, not the industry.
Data and recalculation
Anonymised data: equipment-ua-response-pilot-2026.csv (90 rows: 15 sites × 6 UAs, with a notes column). The repository keeps docs/research/2026-11-ua-response-pilot.csv.
Comparison logic: scripts/ua_response_probe.py; tests: scripts/test_ua_response_probe.py (9 tests, no network).
python -m unittest scripts.test_ua_response_probe
Related: the 30-site robots scan (declaration layer), the robots.txt matcher and the 24-site four-layer review.
Sources and definitions
- robots verdicts: RFC 9309: rfc-editor.org.
- HTTP status and redirects: RFC 9110: rfc-editor.org.
- Capture: around 2026-11-02 07:00 Asia/Shanghai, at most seven requests per site, six seconds apart; two anomalies retried three times.