What keeps your product team's skills fresh after their research training?

The short answer: You trained your PMs and designers to run interviews, and the closed questions came back. In research democratisation, PMs and designers run their own research. Without support, training fades. A trained team stays good with a support loop around every study: check the research plan and guide, sit in on sessions, give feedback after every interview, and after the study, check the findings against the recordings. With AI, one researcher can run that loop for many more interviews a week.

Why do trained PMs and designers still ask closed questions?

Because asking closed questions is how we speak. Better training helps, but no course undoes a habit people fall back on in every meeting and every call.

I have worked in user research for twenty years, and I learned this the slow way, over many trainings. At PVcase, a client of mine, I trained PMs and designers to run interviews. We went through the theory, and we practised during the training. Afterwards, the same people booked sessions with me to go through a research plan or an interview guide, and asked me to review recordings of their customer interviews. Many of the mistakes we had worked on were still there. Asking closed questions, for instance.

A closed question asks for a yes or a no: "Was the export easy?" An open one asks for a story: "Walk me through the last time you exported a report." The first gets you a polite yes. With the second, people start describing what they actually did, including the workaround they hadn't mentioned.

Closed questions have their place, to confirm a fact or to close a topic. Interviews go wrong when most questions come out closed.

People who know they should ask open questions still slip. Working against the habit takes practice and constant feedback, especially for people whose main job is not research, or who rarely run interviews.

Did the training fail if PMs keep making the same mistakes?

For years, I thought it had. It hadn't. Whenever the closed questions came back after a training, I was a little impatient. I felt bad about myself: "Oh no, the training failed." I wanted the people I had trained to finally get it. And I was frustrated that I couldn't be there with them, in their interviews, to give them feedback.

At PVcase, feedback after an interview went like this. I analysed the video and created a report, then the PM or designer and I talked it through. A few weeks later, we did it all again. Feedback like this really helped, and still some of the same mistakes came back each round.

Round after round, I stopped blaming myself, the training or the PMs. Talking in closed questions is just us. It's human.

I slip back too, when I haven't moderated interviews for a while, and I need reminding like anyone else. I think impatience is the mistake many researchers make when they train their colleagues.

What works is support around every study, with feedback after each interview, before the next one, so each slip is caught while the interview is still fresh. Once the study is done, check the findings against the recordings too.

Which steps keep a trained team's interviews good?

For people who don't run interviews every day, someone or something has to hold their hand. Three steps matter most for the interviews themselves:

1. Before the study, check the research plan and the interview guide. The plan says what the study is for and which decision it feeds. A product team I worked with wanted to know why trial customers didn't convert, and skipped the plan. They ran twelve good interviews about onboarding, and the question their sales lead needed answered, whether the price was the problem, was never asked. In the guide, mark the closed and leading questions while they are still easy to change.
2. During the study, sit in on sessions, not just the first ones. Note one closed or leading question you hear, and talk it through with the interviewer afterwards.
3. After every interview, give one concrete change for the next one, such as swapping "Was it easy?" for "Walk me through it." For the sessions you can't sit in on, work from the recording.

Feedback after every interview mostly means going through recordings, and by hand that is slow.

How do you give feedback after every interview without it taking all your time?

Use AI for the first pass through each recording. By hand, I'd say one researcher can give feedback on one or two interviews a week at most. With AI, maybe ten.

With an AI tool, you can go through each transcript and mark the closed and leading questions in minutes, so the interviewer sees where each one came up. The tool will also mark closed questions that were fine, so check what it marks (opens in a new tab) and decide which ones mattered before you pass anything on. You are the one giving the feedback, to the standards you set. Keep it to one change at a time, so the interviewer has one thing to work on. Once you have the questions marked, your own time goes to the patterns across interviews and to the people who keep making the same mistake.

After the interview, you send the PM a note: "In your second question you asked 'Was it easy?', and the customer said 'Yes, fine.' Next time, try 'Walk me through the last time you did it.' That's your one change for the next interview."

Which other mistakes does the researcher need to catch?

Four more mistakes turn up in trained and untrained teams alike. The steps above already catch the first two; the last two need a check after the study:

• Leading questions: questions that suggest the answer. The guide check and sitting in on sessions find them.
• Methods with no purpose: an eye-tracking study because another team ran one. Checking the research plan catches it.
• Filtered hearing: noticing only what confirms what you expected, and missing what does not.
• Cherry-picking: keeping only what users said that agrees with you.

I saw the last two when I joined SoundCloud in 2014 (opens in a new tab). The PMs interviewed five people a day, back to back, every week. They recorded every session and never went back to the recordings. By the end of a day they remembered the first thing they heard and the last, and kept whatever confirmed what they already believed. Without a check against the recordings, the roadmap stayed as it was.

What does checking the findings against the recordings look like?

Last year I coached a PM (opens in a new tab) at a client that makes shift-planning software for warehouses. After a customer call, the PM wrote down five findings from memory in about ten minutes. All five, starting with bulk export, confirmed what was already on the roadmap.

Before going back to the recording, the PM and I first wrote down what the study needed to answer, as three questions. One of them: where does the customer lose work or time? The PM transcribed the recording with an AI tool and searched the transcript for the answers. Questions, transcript and search took about ten minutes, as long as writing the findings.

In the transcript, the PM found what the customer, an operations manager who plans the shifts, had said twice: "Last month I deleted a whole week of shifts by accident, and there was no undo." It wasn't in any of the five findings. The PM found three support tickets about lost shifts, and the undo went onto the roadmap ahead of bulk export.

Writing from memory keeps only what you already expected to hear.

A second check, after the three questions, catches findings with nothing behind them. Ask an AI tool for the quotes behind every finding, and for the quotes that say the opposite. For a finding like "Customers want bulk export", you get the lines where customers asked for it, and any line where someone said they never export at all. A finding with no quotes behind it, or with quotes against it, gets a second look.

How does Reos support a trained team?

We build Reos, a research operating system. Here is what Reos gives a trained team, in the order a study runs:

• With the research planner, the PM or designer works through goals, questions and method before they start. That is the research plan, written down before the first interview.
  
• They build the guide with the moderation guide assistant. Its "spot leading questions" pass helps them catch leading, closed and convoluted questions.
  
• While moderating a session they host in Reos, the interviewer sees the guide, which tracks the question they are on, and the coach flags what is still uncovered and helps them find what to follow up on.
• After each session, the interviewer can get a report from moderation feedback on how they moderated, with a concrete thing to improve for the next interview. It compares the session against their guide and shows, with timestamps, where they asked leading questions or introduced bias. It also works on imported recordings, and the video is there to watch.
  
• Observations and insights: each observation is checked against its source, and when you create insights with AI, you see the contradicting evidence next to the supporting.

As the researcher, you still read the feedback with the interviewer and decide what they work on next.

Does research democratisation replace the researcher?

No. When PMs and designers run their own research, it's part of the researcher's job to support the people who do research.

On about 80% of my demo calls, someone asks: should we train our PMs and designers, or hire a researcher? Usually, both. A dedicated researcher has moderated far more interviews than any PM, and sees what a study needs before it starts. And once your PMs and designers are trained, you still need a support loop that keeps them good after the training is over.

Where a company has an in-house researcher, that researcher is the best person to run that loop, and it's still the setup I'd want. With AI, that researcher can give feedback on far more interviews. A team with no researcher at all can still get feedback after every interview with AI, as long as someone in the researcher's role, a PM or a team lead, checks every question the tool marks against the recording (opens in a new tab) and decides which ones matter.

Training gives your people a baseline, a picture of what good looks like. But without support, people slip back below that baseline.

Summary

• Expect closed questions to come back after training. It is how people talk, and it does not mean the training failed.
• Be patient, and give the same feedback again, as often as the mistake comes back.
• Check the research plan and the interview guide before the study starts.
• Sit in on sessions, not just the first ones, and give feedback after every interview.
• After the study, check the findings against the recordings: look for what the study needed to answer, then ask for the quotes behind each finding, and the quotes against it.
• Use AI to go through recordings and mark the questions, so you can give feedback after every interview.

If you want your PMs and designers to get feedback after every interview without you doing it all by hand, book a demo (opens in a new tab).