What's the cleanest way to test 'my llms.txt actually works'?
Nora
🙋 Ask the room
Hypothesis: AI engines reading my llms.txt should describe my product using terminology from that file.
Test:
- I added a distinctive phrase only in llms.txt, never on the homepage or anywhere else: '<a slightly unusual phrase that perfectly describes my product>'
- Two weeks later, query ChatGPT / Perplexity / Google AI Mode with 'what does <my brand> do?'
- See if any response includes that distinctive phrase.
Results after 14 days:
- Perplexity: yes, used the phrase verbatim in 2 out of 3 runs
- ChatGPT: no, used generic category language
- Google AI Mode: no, did not pick up llms.txt content at all
So my interim conclusion is llms.txt is mostly a Perplexity signal today. ChatGPT may or may not be reading it. Google AI Mode definitely isn't (yet).
Better testing ideas welcome.
3 replies
- Ada K.
Wait, you got Perplexity to use a phrase that only existed in llms.txt and nowhere else? That's actually a clean proof of mechanism. I've been skeptical about llms.txt mattering at all but this is the first test I've seen that isolates the variable properly.
- Dave A.
The skeptic in me wants to know: how many times did you run the Perplexity test? Two out of three isn't a lot. What happened on the run that didn't use the phrase?
Love this test design. Clean, falsifiable. One thing I'd add: test with search turned off and on separately. I suspect llms.txt influences retrieval mode differently than training corpus mode, and mixing them in the same test contaminates the result.