How to Run a Problem Interview

Picture the moment the call connects: the founder has a willing subject, thirty minutes blocked, and every chance to learn something real. Then they open by explaining what they are building.

A problem interview is one specific conversation with one goal: find out whether this person has actually lived the problem you think they have, and what it cost them when they did. The format is thirty focused minutes spent entirely on their past, with your idea kept offscreen. Founders get this wrong because they cannot resist describing what they are building, and the interview becomes a polite audience nodding at a product that does not exist yet.

This guide is about the craft of that single conversation. Recruiting the right people and synthesizing across many calls are separate skills, covered elsewhere. Here the focus is narrow: how to open the call, how to sequence your questions, which questions quietly poison the result, how to take notes while listening, how to code the transcript afterward so quotes turn into counts, and when to stop. Do this part well and every other stage of validation gets easier.

A problem interview studies the past, not the future

The single rule that separates a useful interview from a wasted one comes from Rob Fitzpatrick’s The Mom Test: ask about specific past events, never about future intentions. “Would you use this?” and “Would you pay for that?” are questions about a hypothetical, and people are generous with hypotheticals because agreeing is free and pleasant. “Tell me about the last time this broke, and what you did next” is a question about a fact. Facts do not flatter you.

The reason this matters is that a problem interview has to be able to return a no. If your questions only allow enthusiastic answers, you have built a machine that confirms whatever you already believed. Steve Blank’s writing on customer development frames the whole exercise as leaving the building to test hypotheses against reality, and a hypothesis you cannot falsify is not being tested at all. The job of your script is to give the no every chance to show up.

Open the call without contaminating it

The first two minutes decide the quality of the next twenty-eight. The instant people know your idea, they start managing your feelings, and you lose access to what actually happened to them. So the opening has three moves and none of them mention your product.

1. Frame the call, not the product

   Tell them you are researching how developers currently deal with a rough area, and that there are no right answers.

2. Ask for stories, not opinions

   Say plainly that you want what actually happened, not what might.

3. Hand them a concrete anchor

   Name one wall you already know recurs in the public complaint stream, then ask them to take you back to the last time they hit it.

You are not fishing blind. That last move, anchoring on a real recurring problem instead of a vague prompt, is what makes the rest of the conversation flow.

Sequence questions from event to cost to workaround

A good problem interview climbs a ladder. You start by locating a real event, then dig into what the person did, then find out what it cost, and only near the end do you check whether they tried to fix it. Each rung returns a harder, more trustworthy kind of evidence than the one below it.

The bottom rungs are the traps, and the top rung is the only one worth climbing to, because it forces the person to reconstruct something that actually happened rather than imagine something that might.

Follow the story wherever it goes. When they mention they wrote a script to work around the problem, ask how long that took and whether they still run it. When they say it cost them a day, ask what the day would otherwise have been worth. You are trying to reach the moment where the cost of the problem becomes concrete enough to compare against the cost of a fix.

Take notes that preserve the evidence

Record the call if you have consent, but still take live notes, because the act of writing forces you to notice what matters in real time. Capture three things and nothing else: verbatim quotes, emotion, and workarounds. Do not write summaries during the interview. “User frustrated with CI” is your interpretation and it is worthless later. “I wrote a shell script to babysit the PR checks myself because I stopped trusting them” is the raw material you can code, count, and quote.

Emotion is a signal you can only catch live. When someone’s voice changes, when they laugh bitterly or sigh before answering, mark it, because that is where the expensive problems hide. And every workaround is gold, because a workaround is proof that the person is already spending time or money on the problem today. Talk is cheap; a workaround is a receipt.

Code the transcript so quotes become counts

One interview is an anecdote. The value appears when you turn several transcripts into a small table of counts. This is called coding: you read each transcript, tag every meaningful line with a short label, and then count how many separate people carry each label. The labels are simple, like built-a-workaround, copes-and-shrugs, or has-a-hard-deadline.

The count that matters most is how many people built a workaround versus how many just coped. If five of eight people built something to route around the problem, you have found a cost people are already paying, which is the strongest buying signal a problem interview can produce. If most people shrugged and moved on, you have found a real annoyance that is not a business, and learning that in a week is a win, not a failure.

Coding also protects you from your own memory. Left to recall, you will remember the two most vivid calls and forget the six quiet ones. The tally does not play favorites.

Ground the interviews in what already recurs

Problem interviews are far sharper when you walk in already knowing which problems are common, because then each conversation tests a pattern instead of a guess. The public complaint stream gives you that starting map.

There is no shortage of recurring walls to anchor an interview on. A few clusters from that feed show the kind worth testing, and the difference between them is breadth rather than score.

The first is exactly the sort you would open an interview by naming. The second flags a problem gathering heat that is worth understanding before rivals do. The third is a real bug, but an interview would likely confirm people cope rather than pay.

The interviews then tell you which of these is a business. “Automated Security Scans for Dependencies” showed up as a feature request with a pain score of 55.5 across 10 owners, and a feature request is precisely the kind of stated wish that needs a past-behavior check, because wanting a scanner in the abstract is not the same as having already bolted one on after a scare. “Malformed Tool Calls in Long Sessions”, a pain score of 57.5 across a volume of 23 mentions, is the same story: real in the data, but only an interview reveals whether anyone paid a cost worth removing.

Know how many to run and when to stop

You do not need thirty interviews, and you should not run three. The right number is the one where new conversations stop teaching you anything, a point researchers call saturation. In practice, for a tightly defined group, insight rises fast through the first handful of calls and then flattens, usually somewhere around six to eight, once the same codes keep reappearing and no new ones show up.

Stop when the curve flattens, not when you hit a round number. If your fifth, sixth, and seventh interviews produce no new codes and the workaround count is already lopsided in one direction, you have your answer. Framing the problem as a job people are trying to get done, as in Intercom’s guide to jobs to be done, helps you recognize saturation, because you are watching for the same underlying job to recur, not the same surface words.

The interview is yours; the reading can be automated

Running the interviews is craft you cannot outsource, and it is where the real learning happens. The part you can automate is the reading step that feeds it: figuring out which problems recur across thousands of public complaints so you know what to anchor each conversation on. That is what EchoSift does. It ingests complaints from GitHub, Stack Overflow, Hacker News, and Bluesky, clusters them into recurring pain patterns, and ranks them by volume, distinct-owner count, cross-vendor diversity, growth, and recency, so you walk into every interview already knowing which walls are real and how many independent people keep hitting them.

For where this fits in the wider workflow, see the pillar guide on how to validate a SaaS idea, and for the surrounding stages, how to do customer discovery when you can read the complaints, what the developer pain-point data actually says, and the signals of product-market fit you are ultimately testing for.

This article was drafted with AI assistance and reviewed against EchoSift’s proprietary signal data before publishing.