How many interviews should a product manager do for customer discovery?

If you are new to product management the answer may surprise you. The traditional thinking is that you have to do hundreds of interviews to quantitatively validate your hypothesis. Product discovery can often be conflated with surveys and empirical research methodologies, but in practice they are actually very different.

In order to not bury the headline, the answer is that you should do 8–15 customer interviews as you conduct discovery of a customer need or validate a solution. In some cases, even just 3–5 interviews can be sufficient.

Quantitative research vs qualitative product discovery

Surveys are great for building data around personas, demographics, customer satisfaction, and for tracking variations against a baseline overtime. A good survey requires thoughtful question creation, testing of the questions, and gathering a large enough sample size to be statistically significant (meaning the responses are representative ‘enough’ of the population to be meaningful). Here is a great sample size calculator from Qualtrics, but usually you don’t need more than about 385 responses, regardless of your population size.

Product discovery is not doing surveys. You may use a survey to help provide context or to give you a pool of people to interview, but surveys will not provide the user insights needed to truly understand their pain and needs. True product discovery is actually ethnographic research — defined as observing and interacting with people in their natural environment. It’s more of a social and behavioral science that a statistical/data science.

Qualitative Discovery Methodologies

Interviews: deep dive into everything surrounding the problem you are trying to understand and learn more about. This is a 15–60 minute interview with your potential user or customer. The power is in the follow up questions: why? tell me more? what’s an example of that?

Empathy cards: one of the gems of user-centered design is the empathy card research methodology. You create physical cards with potential attributes of a solution or aspects of the problem. Let’s use the example of a TV remote. You might have cards for:

• Battery dies
• Remote gets lost
• Remote doesn’t work when its too far from the TV
• Remote is too confusing
• Remote is too expensive

You then ask the person you’re interviewing to order the cards from the biggest pain (or most important) to least pain (or least important). You are looking for tensions, contradictions, emotions, and stories. You might ask, “Tell me about a time when you lost the remote? How did it make you feel?”

Shadowing: watching people do tasks in their normal environment can often be more enlightening than hundreds of interviews or surveys. It allows you to see actual behavior and not stated behavior, and then dig into the behavior. I’ll give you an example from when I was building software for dental offices. Our product team did lots of phone and Zoom interviews about the users workflows and task management, with many good insights. But then I went out to dental offices and sat behind the office admin. Immediately, I saw something that set off major fire alarms in my head. There were sticky notes surrounding the computer monitor. The notes had tasks to do, patient information, reminders, and more. As I dug into why the office admin had so many sticky notes, I realized that our product wasn’t solving the problem like we thought it was and it allowed me to understand the problem differently than I ever had before. Users vote with their behavior, not with what they say their behavior would be or even is.

Only 8–15 interviews? How is that possible?

The goal of qualitative product discovery is to deep dive into a problem and how well the solution solves the problem. Surprisingly, this rarely takes more than even just 5–10 interviews. Once you hear/see a trend repeat 3 or 4 times, then you know you have pretty solid footing. If the feedback in the interviews is different in every interview, then you need to keep talking to people until you see trends.

The goal in product development is to understand a need and then test your way into solving it. The real validation comes from getting a prototype or MVP into users hands and see how they use it. If you conduct hundreds of interviews before building a product, you are slowing the agile process down too much and potentially even losing sight of the core of the problem, which results in scope creep. Find trends, build a prototype, get feedback, and iterate.