Each assistant is a different model
ChatGPT, Claude, Perplexity, Gemini and Google’s AI answers are built by different companies. Each runs on its own models, trained on its own data, up to its own cut-off date, with its own habits about how long an answer should be and how many options to offer. What one model knows about your category before it searches for anything is not what another knows.
The model behind a name also changes. Each vendor offers several models and replaces them over time. An answer from the same assistant this month may come from a different model than an answer three months ago, and nothing in the answer tells you so.
In Teddit you choose which model each assistant answers with. If you change it, expect the answers to shift, and remember the date when you read the history later.
Live web search, and what it brings back
For questions about products and services, assistants commonly search the web and build the answer from the pages they find. That adds three steps, and each one can go differently.
- What to search for. The assistant turns the question into one or more search queries of its own. Nothing makes two assistants write the same ones.
- What comes back. Each assistant searches in its own way. The same query can return different pages in different products, and different pages next week than this week.
- What gets used. From the results, the assistant picks which pages to read, which to rely on and which to cite.
A change at any step changes the answer. A new comparison article is published. A review site updates its list. A page of yours is rewritten. The assistant did not change its mind about you. The material it was handed changed.
This is why Teddit stores more than the text. For every answer it keeps the pages the assistant cited and the web searches it ran, where the assistant discloses them. When an answer changes, the list of sources is usually the first place worth comparing.
The wording of the question
An answer belongs to the exact question that was asked. Compare two ways of asking about the same need:
What is the best email marketing tool?
I run a small online shop and send one newsletter a week. Which email marketing platform should I use?
The second question gives the assistant a size, a use and a frequency. It will write different search queries, find different pages, and often name a narrower set of brands. A made-up platform like Brightloop could be absent from the first answer and present in the second, or the other way round. Neither result is wrong. They are results for two different prompts.
Language and place count as wording too. Teddit asks every prompt of a site in that site’s language and from the site’s country, because a question from another country can get another answer.
Randomness
Even with the same model, the same question and the same pages, an assistant does not produce the same text twice. Language models choose their words with some randomness built in. The answer may list four brands one time and six the next, or put them in a different order.
It follows that brands at the edge of an answer are the least stable. A brand the assistant treats as an obvious choice tends to appear every time. A brand that sits at the end of a longer list can come and go without anything in the world having changed.
What this means for measuring
If a single answer can differ for four reasons you do not control, a single answer is an anecdote. A screenshot from Tuesday proves very little. Measurement has to be built so that the noise shows up as noise.
- Keep the questions fixed. Track the same prompts with the same wording, week after week. Then wording is no longer a reason for change, and what moves comes from the assistants and the web.
- Ask on a schedule. Teddit runs scans by hand or daily, weekly or monthly. Weekly suits most brands: often enough to catch a change, slow enough to read what came back.
- Read runs, not single answers. A scan collects one answer per assistant per prompt. One square can be luck. Three weeks in the same state is a pattern.
- Read each assistant separately. They are different products with different sources. An average across them hides the one that changed.
- When a change holds, open both sides of it. Compare the answer before and the answer after, and the pages each one cites. A source that appeared or disappeared often explains the change.
The states each answer can be in, and how to read them across weeks, are covered in Named, named first, linked. To set up prompts and a schedule, start with the quick start.