Real research is AI-proof.
Here's why, and where AI still makes it better

Real research is AI-proof because most research exists to find out something nobody knows yet, and AI can only work from what is already known. AI now takes time, cost and friction out of research, which makes it essential. But judgement, the unexpected and real people can't be replaced, and the best results come from using AI alongside them.

That's the argument of a session I'm giving at Insight Platforms' Qual Insights Summit, Real Research Is "AI-Proof": Three Real-World Examples That Prove It… While Still Using AI. The title is meant to provoke a little, because every example in it uses AI. This is the written version, with the sources a 25-minute talk can't fit.

By "real research" I mean research that sets out to find something we don't already know. That means preparing for the unexpected and getting to insights that weren't obvious at the start. And, to some extent, it means talking to real people.

Two things can be true at once

Most researchers have now used AI enough to feel two things at the same time. On one hand, work that used to be slow and laborious has become close to trivial, which makes AI essential. On the other, there are things a researcher does, and things research needs, that it can't fully replace. And the more AI does, the more natural it is to wonder whether it will eventually do the whole job.

I think both are true. The useful question isn't which side you're on. It's where each one belongs.

Why can't AI just do it all?

AI is clearly capable, so it's easy to feel we're close to the point where a model, perhaps with synthetic participants (AI-generated stand-ins for real people), gives answers good enough that real research isn't worth the time or the money. I don't think we are, for three reasons, and none of them is about the technology not being clever enough yet.

1. It gives you the most likely answer, and the most likely answer is the average

A language model doesn't work out the best answer to a question. It predicts the most likely one, and the most likely answer is, more or less by definition, the average.

There's a good illustration from earlier this year. Researchers from Esade, the University of Sydney and NYU Stern, writing in Harvard Business Review, put classic strategy dilemmas to the big AI models about 15,000 times, each set in a specific business. Whatever the business, the models kept choosing the same fashionable answer, and more detail barely changed it. They called the result "trendslop".

That isn't a fault that will be patched out. It's how the models work. If every agency asks the same AI to write the discussion guide, play the participants and find the themes, every client gets the same research. No agency decides to be average. It gets there one sensible shortcut at a time.

Nobody commissions research to find out what the average is.

2. It answers with complete confidence, whatever that's based on

Here's a deliberately silly example. Say we want to find out who's most likely to like a new product: a Marmite-flavoured Coke. Marmite, if you don't know it, is a very strong yeast spread from the UK. People either love it or hate it, and the brand has built 30 years of advertising on saying so.

Ask any person about Marmite-flavoured Coke and they'll tell you straight away that it's a bad idea. So I asked one of the latest AI models whether men or women, and older or younger people, would prefer it.

Any human's answerBad idea.

It came back with a starting hypothesis: men more willing than women, younger adults more receptive than older ones. It even gave me a table, with younger men at the top and older women at the bottom.

A different prompt might well get a different answer, but that isn't the point. It was perfectly willing to answer with complete confidence, with no way of knowing what that was based on. And it wasn't even useful. The most helpful thing it could have said was: don't make it.

3. It works from what's already known

Models learn from an enormous amount of what has already been said and written. If nobody knows something yet, it isn't in the data.

A lot of research exists to find out what nobody knows yet.

The obvious comeback is mathematics. Over the past year there have been headlines about AI solving "Erdős problems", a famous collection of puzzles left by the mathematician Paul Erdős, some of which had defeated mathematicians for decades. If AI can do that, surely it can find out new things too? Looked at closely, the first big claim turned out to be problems already solved in old papers: a very good literature search, not new mathematics. The genuinely new results since are impressive, but as Quanta and mathematicians such as Terence Tao have described, they mostly combine known techniques on problems nobody had got round to.

New answers, then, but not new ways of thinking. Real research is often looking for exactly that kind of new answer.

New answers come from real people.

Where AI fits

There's an old rule you'll know: good, fast, cheap, pick two. If you want it good and fast, it won't be cheap. If you want it fast and cheap, it won't be good.

Some research has leaned on AI to be fast, cheap and "good enough", in two broad flavours. Qual at scale runs quick, largely automated studies with hundreds of participants, often from quant panels, who give shorter answers with little or no human oversight. Synthetic data removes people altogether. Both have their place: synthetic respondents can help pilot a questionnaire or test a theory on well-mapped ground, because that's what they're built from.

Real research with AI works differently. It doesn't chase the extremes of speed and cost, so it gets faster and cheaper without dropping to good enough, and the quality goes up.

The most practical way I know to decide where AI belongs in a study is this:

Use AI for the things that have a right answer, and keep judgement with the researcher.

Did the photo show what was asked for? Did the video answer all three parts of the question? Those have right answers, and AI is very good at them. What a finding means, and what the client should do about it, are yours.

Three places real research is AI-proof, and AI still helps

Here are the three examples from the session, each with something you can use whatever platform you work on.

1. Long-term communities

We've run communities from a few hundred people to well over a hundred thousand, and the honest truth is that running a really good one is hard. You have to recruit people, keep them engaged and manage them. Without that, you end up with members who never answer and a few who are after the rewards for the least effort. The old answer was to make the community big enough that it didn't matter.

What clients ask for now is different: smaller communities, with people who actually take part. And that's the AI-proof part. Members stay for months or years, so you can be much more confident they're real and answering for themselves, not getting AI to answer for them.

So it's worth recruiting good people and looking after them:

• Have a researcher write the screening and profile questions. That's where the quality of a community is decided.
• Weed out regularly. People who never reply, who put in the least effort, or who use AI to write their answers dilute what the others give you.
• Spot people drifting away. Rather than writing rigid rules, describe what fading looks like ("used to post photos, now only answers polls") and let AI find the members who match.
• Make the everyday parts worth taking part in. A place to have your say, quick polls and a clear reward system keep people coming back.
• See the community before it's built. AI makes it quick to prototype a whole community, so the client and the researcher can click through it and settle the details first.

There's more on how communities work in What is an MROC? and on Qualzy Community.

2. Live interviews and focus groups

This is the clearest case: people talking to people. So it's worth asking fairly whether AI-moderated interviews, where an AI asks the questions and the follow-ups instead of a moderator, will replace researcher-led ones.

There's a lot to like about them. You can run hundreds at once, in any time zone, with results in hours and a lower cost per interview. Everyone gets the same probes, in many languages, and some people say more to a machine than to a person. For structured questions and product feedback, they do the job well.

But when the usability research firm Nielsen Norman Group tested two AI interviewers this year, the AI stuck to the script, couldn't tell when a topic had been covered and missed everything a face would show. Only three of their ten participants said the conversation felt natural. Their conclusion was to use AI interviews alongside human moderation, not instead of it.

The technology will keep moving, but two things hold. Talking to a person feels different from talking to an AI: some people open up more to a machine, and there's often less connection and less gets picked up. And a group needs a moderator. What gets called an AI focus group is really a set of individual interviews running at the same time.

So researchers still matter here, but researcher-led sessions have to be run efficiently. A general video-meeting tool is cheap, but it doesn't make a session any quicker to run or any better to analyse, and time costs money.

A false economy.

Wherever you run your sessions, this is where the hours go, and what helps:

• Booking. Let participants book themselves from the moderator's availability, with a "none of these work for me" option.
• Tech checks. Run them when people book, not five minutes before the session.
• Stimulus. Keep the discussion guide beside you as you moderate, with the images, videos and exercises ready to show participants in one click, so there's no fumbling with screen sharing.
• Marking moments. When something good happens, mark it and add a few words about why. Later, AI can turn each marked moment into a video clip, and your note tells it what you cared about.
• The last ten minutes. Check what's still open in the guide, and what someone hinted at that you never followed up.

None of that replaces the interviewer. It makes you better at the part only you can do. Our guide on how to run an online focus group goes further, and Qualzy Live is how we've built this.

3. AI-moderated chat

An AI-moderated chat is an online study that participants complete as a chat, with an AI moderator asking the follow-up questions. They seem to be everywhere, promising the depth of qual at the speed and scale of a survey, and one tool instead of a recruiter, a moderator, a transcriber and an analyst.

The concerns are real too. A researcher recently wrote on LinkedIn about taking part in one as a participant. By the third question they felt nobody was out there, their effort dropped, and the AI happily accepted superficial answers. The comments kept coming back to tools designed by developers rather than researchers, and to trading quality for cost.

That's fast, cheap and good enough again. You could simply not use these tools for deeper online qual, but some clients now expect to see AI used somewhere, and assume you're being inefficient if it isn't.

The answer, I think, is an AI-moderated chat that's led by the researcher:

• The design doesn't change. It's still a structured qual project, with the order, the activities and the questions you choose. Your skill as a researcher goes where it always went.
• The AI works to your goals. For each question you say what a complete answer looks like. If a question has no goal, the AI doesn't step in.

Writing good goals is the new skill, and a quick one to learn. Compare these two for a question about coffee at home:

Say what a complete answer contains and when to stop, and keep to one goal per question. And before fieldwork, take the study yourself as a participant. That was the best advice in the LinkedIn thread, whichever tool you use.

I've written more on when AI-moderated interviews work, and when they don't. Qualzy's own version is the AI-moderated chat.

A five-question check for your next study

Before you hand any part of a study to AI, ask:

The bottom line

If you'd like to see the three examples working, the session recording stays on Insight Platforms for a year after it airs on 11 November.