By Rob Manzano, founder and CEO of reOS. 15+ years leading UX research at SoundCloud, Zalando and Spotify. Updated September 2026.
An AI-generated research insight is verified when you can trace it to the source quote, confirm the AI analysed the full dataset, and show what in the data contradicted it. An insight that fails any of those checks is unverified and should be labelled that way before a stakeholder sees it. Keeping that trail intact is what makes research defensible.
Quick summary
-
Rob Manzano (opens in a new tab), who led research at Zalando and Spotify, runs three checks on any AI insight before it reaches a stakeholder.
-
General LLMs like ChatGPT, Claude and Gemini help with synthesis. They also don't cite their evidence accurately, can skip parts of long inputs, and drift between runs.
-
A finding with no evidence behind it is just your opinion against a more senior one. A finding people can click into changes who they're arguing with.
-
Label unverified findings on the slide. People will trust the verified ones more because you did.
What makes a research insight defensible?
A defensible insight stays linked to its source. Anyone in the room can follow the trail from the finding back to the customers who said it. The reasoning in between stays visible too.
Most slides throw that trail away. The finding is there. The interviews, the coding choices and the outliers you set aside sit in a folder nobody opens.
So when a stakeholder asks "where did the customer say that?", you have to prove something in a format that already threw the proof away.
AI adds a second layer. Synthesis used to be your own thinking. Now large parts of synthesis happens inside a model, and if you can't see what the model did, neither can stakeholders.
Why do AI-generated insights fail in stakeholder meetings?
They fail because they sound confident and show no trail. A stakeholder can't tell a verified finding from a plausible one.
Most of the time, that missing trail starts with the tool. ChatGPT, Claude and Gemini are what most researchers reach for first. That's why so many researchers paste transcripts into them. Used that way, they have four habits worth knowing.
-
They don't cite themselves. Ask for the themes across eight interviews and you get themes. You won't be told which line of which transcript produced each one unless you ask, and then check.
-
They can skip parts of a long input. Long transcripts may get summarised or under-weighted. You won't be told when that happened.
-
Their output drifts. Ask the same question twice and you can get two sets of themes. The first answer was one reading of the data.
-
They aggregate. What several people said becomes a theme. What one person said gets dropped, even when that one comment is the important one. A design lead I spoke to keeps a private file of those, because in his experience they turn into project-driving themes.
None of this makes them bad tools. It makes their output a hypothesis until you've checked it.
What are the three questions to ask before trusting an AI insight?
Ask three things: can I trace it to the source, did the AI read all the data, and did anything contradict it. This works with an LLM and a spreadsheet. It's also the standard we built reOS around.
1. Can I trace it to the exact source?
Name the interview and the timestamp. "It's somewhere in the transcripts" means you have a hypothesis.
Ask the AI to quote its evidence. Then open the transcript and check the quote is real and means what the summary says. In reOS, every insight links to the exact second it came from (opens in a new tab), so you click the insight and hear the customer say it.
2. Did the AI read all the data, or part of it?
Check what went in. How many transcripts, and did the tool read each one? Or did it summarise first and then analyse the summaries?
This is also where the process matters. A single AI pass gives you no way to see what got dropped. In reOS, analysis runs through four stages and every claim gets checked against its source (opens in a new tab) as a separate step.
3. Did anything contradict this finding?
Ask outright: which participants disagreed, and what points the other way? Zero counter-evidence across a dozen interviews usually means an incomplete search.
Interviews are also only one input. Support tickets, emails and forms are where the counter-evidence usually turns up, which is why reOS pulls them into the same analysis (opens in a new tab). Every insight shows the evidence that supports it AND evidence that argues against it.
If you can answer all three, the insight is verified. If one answer is missing, write "unverified, based on partial data" on the slide. That label costs you nothing and buys trust for everything else in the deck.
How does an evidence trail change the meeting?
It changes who your stakeholder is arguing with. When they can click through and watch the customer say it, the argument moves from your judgement to the customer's words.
At Zalando I had a hard rule for the product teams: no tourists.
We wouldn't run research on their behalf. If they wanted research, they did it with us, taking notes, moderating, or running the debrief. I'm writing more about that rule on LinkedIn next month.
👉 Follow along here (opens in a new tab)
FAQ
Can I use ChatGPT for user research analysis?
Yes, for a first pass. Treat the output as a hypothesis. Run the three checks by hand before it goes near a stakeholder.
What do I do with an insight that fails a check?
Present it and label it unverified, or drop it. Presenting it as verified is the one option that damages every other finding in the deck.
How is defensible research different from traceable research?
Traceability is the link from insight to source. Defensible research keeps that link visible all the way to the decision, along with the reasoning in between, and adds a human sign-off.
Final thoughts
An AI research insight is verified when you can trace it to its source, confirm the AI read everything, and show what contradicted it. General LLMs don't run those checks for you. So you do them by hand, or you build them into the workflow.
Whichever route you take, start by finding out how your current work would score.
- Pick your last AI-assisted analysis and run the three checks on it.
- Label anything that fails as unverified before it goes further.
- Decide which check costs you the most time, and fix that one first. If you'd rather not run those checks by hand, reOS is the insights platform for user researchers. Book a demo and we'll trace one of your own insights back to the second it was said.
👉 Book your free demo (opens in a new tab)
Not ready for that? Our monthly newsletter goes deeper on topics like defensible research. Subscribe here. (opens in a new tab)
