Direct answer: On 31 October and 1 November 2026 we scanned the websites of 30 Changzhou companies from the 2024 Su-Xi-Chang first-set major equipment list. 25 had an analyzable homepage: 17 allow all five AI crawlers, 1 blocks all five, 7 have no robots.txt (counted as absent, not as an explicit allow), and 0 block only some bots. Another 4 produced no identifiable official site and 1 returned a 502 for robots.txt.
This is not an industry benchmark or a website ranking. Only 25 analyzable homepages are below our 30-website threshold, so we call it a scan. The frame, search rules, fetch rules, engine change, anonymized data and limits are published. No company is named or evaluated.
No robots.txt is not "allow"; it is "no declared policy".
Allowing crawlers does not mean being cited by AI.
A full block does not prove the company does not want to be found; the rule may come from a template or legacy configuration.
Frame: 30 Changzhou companies from the first-set list
The frame is the 2024 Su-Xi-Chang first-set major equipment list: 40 items for Suzhou, 30 for Wuxi and 30 for Changzhou, 100 in total. We take the 30 Changzhou companies (list positions 71–100) as a census, not a random sample and not a selection.
The list was jointly issued by the Changzhou, Suzhou and Wuxi industry bureaus, with a public comment period from 16 to 22 December 2024; we checked a public reproduction and its xlsx attachment. Limits: the list is from 2024, not the latest 2026 list, and 30 companies do not represent Changzhou manufacturing as a whole.
Method: robots first, then decide whether to fetch
For each site we fetched robots.txt, matched the five AI crawler user agents, and then checked whether our own agent was allowed on the homepage. One fetch, no JavaScript, no login, no form submission, at least five seconds between sites.
Site identification: two queries (legal name + 官网 / 官方网站 首页), top 10 results; B2B directories, company registries, job boards, social networks, news, government and exhibition pages were excluded. If neither query produced an acceptable site, we recorded "no official site identified"; group/brand sites were accepted and noted.
robots verdict: each of GPTBot, OAI-SearchBot, PerplexityBot, Bytespider and ClaudeBot was matched against its user-agent group, falling back to *; the path checked was /. Allowed, disallowed, absent (no robots.txt) and unknown (fetch failure) are separate values. Absent and allowed are counted separately.
Method change: 360 Search repeatedly returned zero results for our agent (rate limiting), so many samples used a fallback search integration with the same queries and exclusion rules, recorded per sample. This is one of the scan's largest limitations.
Result 1: company-level availability
| Result | Companies | Rule |
|---|---|---|
| Analyzable homepage | 25 | Homepage GET 200 and parseable static HTML |
| No official site identified | 4 | No acceptable site in the top 10 of two queries; not proof that no site exists |
| robots unreachable | 1 | robots.txt returned 502; page not fetched, conservatively |
| Site-wide block for our agent | 0 | No sample blocked our own agent site-wide |
Company-level counts cover 30; robots verdicts are counted on the 25 sites only. A 502 is a server error, not a policy refusal, and must not be written as "blocking AI".
Result 2: policy distribution by bot
Of 25 homepages, 17 allow all five AI crawlers, 1 blocks all five, 7 have no robots.txt (absent) and 0 block only some. Per bot, the counts are identical: 17 allowed, 1 disallowed, 7 absent.
| Crawler | allowed | disallowed | absent |
|---|---|---|---|
| GPTBot | 17 | 1 | 7 |
| OAI-SearchBot | 17 | 1 | 7 |
| PerplexityBot | 17 | 1 | 7 |
| Bytespider | 17 | 1 | 7 |
| ClaudeBot | 17 | 1 | 7 |
Identical counts describe these 25 sites only; they do not prove that companies treat all AI crawlers as one category. Zero partial blocks is also an observation of this sample.
The seven absent sites deserve a separate note. They have no robots.txt, which common practice treats as no restriction, but that differs from an explicit Allow: the first expresses no policy, the second expresses permission. We keep them apart to avoid turning "not configured" into "welcoming AI".
Result 3: homepage structure and JS shells
Of 25 homepages, 12 have no H1, 8 have exactly one and 5 have more than one; 5 have JSON-LD and 1 declares hreflang. Three are under 20KB, including one near-empty shell.
| Signal (n=25) | Count | Note |
|---|---|---|
| H1 = 0 | 12 | No H1 in static HTML |
| H1 = 1 | 8 | Unique H1 |
| H1 > 1 | 5 | From 2 to 10 |
| JSON-LD present | 5 | Types and completeness not graded |
| hreflang present | 1 | The only one in this sample |
| HTML < 20KB | 3 | One is 1.3KB with a title, H1=0, likely a JS shell |
H1, JSON-LD and hreflang are counted from static HTML only; JavaScript injection is undercounted. Small HTML does not always mean JS dependence and needs a content check.
Equating "no H1" with "bad SEO" is too strong; but 12 of 25 having no H1 suggests semantic headings are not the default in this set. For tools that parse static HTML, that makes the page topic harder to determine.
What it means for a manufacturer
- Decide whether you have a policy. Sites without robots.txt should choose which AI crawlers to allow, which to refuse and why. Doing nothing leaves the decision to the other side.
- Do not rely on one blanket rule. If you want search-type AI but not training-type crawling, configure by bot/purpose; the finer the rule, the more often it needs review to avoid accidental blocks.
- Readability comes after access. H1, JSON-LD, static body text and inquiry paths determine whether an allowed crawler can understand the page; robots is only the first gate.
For a self-check, use the 20-check AI readability self-audit. The earlier 21-company AI-device pilot offers a different frame for comparison, and the SEO and GEO guide covers crawling, structure and measurement in more depth.
Limits: do not treat 25 homepages as an industry conclusion
- Specific frame: 30 Changzhou companies from the 2024 Su-Xi-Chang first-set list, not Changzhou manufacturing and not China.
- Website sub-sample 25 < 30: not a benchmark; every count describes these 25 homepages.
- Search-dependent: rate limits and index coverage affect the "no official site" count; many samples used a fallback engine.
- One fetch, one user agent, one region: robots may vary by UA or IP; TTFB is a proxy, not CrUX.
- Homepage and robots only: product and English pages were not fetched, and indexing, AI mentions, clicks and inquiries were not measured.
- No motive inference: a full block or absent file is recorded as a rule outcome, not as intent.
Data and reproduction
The anonymized CSV (30 rows, sample IDs E01–E30, no company names, URLs or contact details) is at equipment-website-robots-scan-2026.csv. Fields include result, http_status, H1, JSON-LD, hreflang, inquiry form and the five bot verdicts. Raw HTML is not published.
Recheck: five samples were re-parsed from stored HTML with the same parser; robots verdicts and structural fields matched. Other samples keep raw responses and timestamps for internal review. Empty cells mean "no analyzable homepage", not zero.
Sources and definitions
- 2024 Su-Xi-Chang first-set major equipment list notice (Changzhou, Suzhou and Wuxi industry bureaus, 2024-12-16): notice.
- RFC 9309, Robots Exclusion Protocol: rfc-editor.org.
- schema.org Organization (reference for the structural signals): schema.org.