ChatGPT search-on vs search-off returns opposite citations for the same query. I'm tracking both now.
Spent a week trying to understand why my citation rates were so inconsistent across runs on the same query. Figured out it wasn't temperature variance — it was search mode.
When ChatGPT has web search enabled, it behaves like a fresh-crawl retrieval system. My recent content (comparison pages published 3 weeks ago) shows up. When search is off, it's pure training data, and my brand is either not there or gets confused with older competitors.
So now I have two separate tracking columns:
- 'ChatGPT search-on' mention rate
- 'ChatGPT search-off' mention rate
They're completely different numbers. Search-on: I'm at 31% mention rate on buyer-intent queries. Search-off: 7%.
The gap is actually useful information. The 24-point spread tells me that my content is being crawled and read recently, but I haven't built enough training-corpus presence yet for the baseline model to know I exist.
Search-on is the leading indicator. If search-on moves, search-off should follow in 3-6 months as the new content gets ingested.
If you're only tracking ChatGPT and getting confused by variance — check whether your test runs have search enabled or not. It's probably not variance, it's two different data sources answering the same prompt.
3 replies
- Leo H.
how do you ensure your test runs are consistently search-on vs consistently search-off? i keep forgetting which setting i had when i ran a test and it contaminates my week-over-week comparisons
- Dave A.
The search-on as leading indicator for baseline is a useful mental model but I'd caveat it: your content still needs to earn sustained links to stay in Perplexity and search-on ChatGPT. A page that gets crawled but doesn't get linked can fall back out within 4-6 weeks based on what I've seen.
The separate column for search-on vs search-off is something I've been meaning to set up for months. The 24-point spread you're seeing is actually a really clear diagnostic. That spread = 'I'm doing recent content work well but haven't built enough entity presence in the training corpus yet'. Very actionable framing.