Watched our server logs for 60 days to see who actually fetches llms.txt. The honest answer.
Tired of opinion threads on whether llms.txt 'works', so I instrumented it and watched the logs.
Setup: llms.txt deployed on 3 sites I run (one B2B SaaS, one open-source project, one personal blog). Custom log line on every fetch with user-agent and IP. 60-day window, mid-April to mid-June 2026.
Who actually fetched llms.txt:
- PerplexityBot: 47 times across the 3 sites. Most consistent fetcher by far.
- OAI-SearchBot: 12 times. All on the SaaS site, never on the OSS or blog.
- ClaudeBot / Claude-Web: 0. Not once.
- Google-Extended / Google-Search: 0.
- Various hobby crawlers and scrapers identifying as 'curl', 'python-requests', 'GPT-something-bot': ~80 fetches, hard to tell what's real.
What did NOT happen: no Anthropic-identified bot ever touched it. No Google bot ever touched it. ChatGPT searched the site directly on every query I tested instead of fetching the manifest.
My read: llms.txt is currently a Perplexity signal and a low-effort one-time write-up exercise. Anyone telling you it's a 'standard' the major engines respect is either ahead of where the actual adoption is, or wishful.
Will re-run this in 6 months. Happy to share the log parser if anyone wants to replicate.
2 replies
- Marcus B.
Would steal your log parser. We've been doing this manually on grep and it's painful. The PerplexityBot 47 vs ClaudeBot 0 split is the kind of finding that should be on a public dashboard somewhere — it would settle a lot of conference debates in one screenshot.
Petr here — your fetch counts match what we see across the monitored fleet to within margin of error. The PerplexityBot pattern (regular, predictable) is the cleanest signal. The OAI-SearchBot pattern (irregular, query-triggered) tells you it's used for live retrieval rather than scheduled crawl. The Anthropic absence is the most quietly important data point in this thread — if you're optimizing for Claude, llms.txt is not the move.