Direct answer: we requested three paths on each of 15 equipment-company websites — root /llms.txt, /llms-full.txt and /.well-known/llms.txt — 45 requests in total. The root path returned a usable text/plain + Markdown file on 4/15 (26.7%) sites (424–12,392 bytes); one more returned 200 but 58,326 bytes of HTML and does not count; nine returned 404 and one timed out. /llms-full.txt and /.well-known/llms.txt had 0/15 valid files.
The point is not “no llms.txt means no AI citation”. It is that llms.txt is still uncommon in this sample, and a 200 status alone can mistake a 58 KB HTML page for an llms.txt. Acceptance needs the path, Content-Type and body type together.
Method: three paths, original scheme, body-type check
The sample is the same 15 equipment-company sites used in the AI-crawler UA study. Each site keeps its original URL scheme (three sites are http://; forcing HTTPS would create SSL errors unrelated to llms.txt). Each site requested the three paths in order, with 1 s between paths and 1 s between sites; redirects were followed, the timeout was 20 s and at most 1 MB was read.
Status alone is not enough: only 200 + text/plain + Markdown body (starts with # or contains Markdown links) counts as valid. A 200 with text/html is recorded as an HTML soft 404. 404 and 5xx/timeouts are separate; the content quality, accuracy or AI usefulness of the file is not judged.
| Path | Valid text/plain + Markdown | 200 but HTML | 404 | 5xx / timeout |
|---|---|---|---|---|
/llms.txt | 4 | 1 | 9 | 1 |
/llms-full.txt | 0 | 1 | 14 | 0 |
/.well-known/llms.txt | 0 | 0 | 13 | 2 |
Result: four valid at root, none on the extended paths
| Site | /llms.txt | /llms-full.txt | /.well-known/llms.txt |
|---|---|---|---|
| E01 | 404 | 404 | 404 |
| E02 | 404 | 404 | 404 |
| E03 | 200 text/plain 424 B | 404 | 404 |
| E04 | 404 | 404 | 500 |
| E06 | 200 text/plain 7,753 B | 404 | 404 |
| E08 | 404 | 404 | 404 |
| E09 | 200 text/plain 702 B | 404 | 404 |
| E11 | 404 | 404 | 404 |
| E12 | 200 HTML 58,326 B | 200 HTML 58,326 B | 404 |
| E13 | 404 | 404 | 502 |
| E14 | 200 text/plain 12,392 B | 404 | 404 |
| E16 | 404 | 404 | 404 |
| E17 | 404 | 404 | 404 |
| E18 | 404 | 404 | 404 |
| E19 | timeout | 404 | 404 |
The four valid files were 424, 7,753, 702 and 12,392 bytes, 21,271 bytes in total, median 4,227.5 bytes. All start with a # heading and contain site/business description and links; content completeness is not judged.
Three ways to misread it
- 200 is not llms.txt. E12 returned 200 at both root and
llms-full.txt, but Content-Type wastext/htmland the body was 58,326 bytes of page HTML. Counting status alone would raise the valid rate from 4/15 to 5/15. - Extended paths are not mirrors of the root. None of the four root files had an
llms-full.txt;/.well-known/llms.txthad no valid file, plus 500, 502 and a timeout. Test each path separately. - Keep the original scheme. Three sites start with
http://; forcing HTTPS produced SSL errors and would mistake “no llms.txt” for “unreachable”.
What this means for acceptance
- Test root first, then full and well-known. Only the root had valid files in this sample; record the extended paths separately rather than copying the root result.
- Add Content-Type and body checks to 200.
text/plain+ Markdown counts; atext/html200 is a soft 404. - Keep each site's original scheme. Do not force HTTPS on HTTP-only sites; SSL errors contaminate the adoption result.
- Keep llms.txt separate from robots. robots says whether a crawler may fetch; llms.txt is a site note for AI; neither replaces the other.
- Treat absence as normal; do not promise an effect. 4/15 describes these 15 sites; no llms.txt does not mean no citation, and an llms.txt does not guarantee one.
Limits
- 15-site pilot: the same sites as the AI-crawler UA study; not the equipment industry and not all corporate websites.
- Three paths, one request: one request per path per site; redirect chains, CDN differences, regions and long-term availability were not tested.
- Reachability and body type only: content quality, link validity, AI use, citation or ranking are not measured.
- One timeout: E19's root path timed out twice and is recorded as unreachable; its other two paths returned 404, so the timeout is not rewritten as 404.
- Non-standard: llms.txt is not an HTTP standard; this study records paths, status, Content-Type and body type only.
Data and reproduction
Anonymised data: equipment-llms-txt-adoption-2026.csv (45 rows × 11 columns: site, path, status, final path, redirected, Content-Type, downloaded bytes, body type, Markdown marker, error type, time). In-repo copy: docs/research/2026-11-llms-txt-adoption.csv.
Collection: Python requests, original scheme, three paths in order, 1 s between paths and sites, 20 s timeout, at most 1 MB read, redirects followed. Classification follows the method table. Site names and full URLs are not published.
Related: the AI-crawler UA pilot, the AI-terminal website readability pilot, and the product entry-page AI-readability pilot.
Sources and definitions
- Sample: 15 equipment-company sites from the AI-crawler UA study (fetched around 2026-11-01; original scheme preserved).
- Requests: 19:48–19:50 on 2026-11-07, 3 paths × 15 sites, 45 requests, Python requests, 20 s timeout.
- Rule: 200 +
text/plain+ Markdown body counts; 200 HTML is a soft 404; 404 and 5xx/timeout are separate.
Frequently asked questions
Is llms.txt an HTTP standard?
No. It is a proposed convention, not an HTTP standard; this study records status, Content-Type and body type on three paths, not its official status or effect.
Why four valid rather than five?
Five root paths returned 200, but E12 was 58,326 bytes of text/html rather than text/plain + Markdown; under the method rule only four count.
Why are llms-full.txt and .well-known/llms.txt both zero?
In these 15 sites only four root paths had a valid llms.txt. llms-full had one 200 HTML and 14 404s; well-known had 13 404s, one 500 and one 502.
Does a 200 status prove an llms.txt exists?
No. E12 returned 200 at both root and llms-full, but Content-Type was text/html and the body was 58 KB of page HTML; Content-Type and body type are required.
Why are some sites tested over http?
Three sites in the sample start with http://. Forcing HTTPS produced SSL errors and would mistake “no llms.txt” for “unreachable”, so the original scheme is kept.
Can the data be reproduced?
Yes. The 45-row × 11-column anonymised CSV is in the repository and records time, path, status, Content-Type, bytes, body type and error type row by row; site names and full URLs are not published.