Chatbot User Testing: A Practical Guide for French Teams

To user-test a chatbot, give 5 to 10 people from your real target audience realistic tasks, watch whether they complete them without help, and score what you see. Do it on a prototype, again before launch, and then keep measuring in production. This guide covers the preparation, the session, the analysis and the personal-data rules that apply in France, so your first test produces decisions rather than opinions.

TL;DR

• Test before launch on a pre-production version, then monitor continuously in production. One test is never enough.
• Define objectives and realistic tasks first. Recruit people who match your real users: 5 to 10 per segment is a common range for a qualitative test.
• Run sessions with two people (one leads, one takes notes), ask participants to think aloud, and record only with consent.
• Measure task completion, a usability score such as SUS, and product KPIs like abandonment. Share a short report ending in decisions.
• Treat transcripts as personal data under GDPR: limited purpose, limited retention, and a purge plan for sensitive data nobody asked for.

Table of Contents

• Why test a chatbot with real users
• When to test: prototype, pre-production, production
• Preparing the test
• How many participants, and who
• Running the session
• Analyzing results and sharing them
• Protecting participants’ data: GDPR and the CNIL
• Common mistakes that spoil a chatbot test
• Our point of view: test early, test with real cases
• Run your tests on a real pilot
• FAQ
• Sources

Why test a chatbot with real users

A chatbot can pass every technical check and still fail in front of a customer. It answers a question nobody asked, loops on a vague message, or hides the way to a human. Only people unfamiliar with the build will reveal that.

Research on evaluating chatbots frames the issue as trust. Businesses should test chatbots to address trust issues around service performance, user satisfaction and long-term unintended consequences. For business chatbots, the same paper points to accurate, up-to-date information, reliability and human interaction as what customers value most.

So a good test answers four questions:

1. Can users reach their goal (track an order, book a slot, get a quote) without leaving the chat?
2. Are the answers correct and current?
3. Does the tone and flow feel natural, including when the bot does not understand?
4. Can the user get to a human when they want to?

If you are still shaping the conversation itself, our piece on production principles for chatbot conversation design helps you decide what to put in front of testers.

When to test: prototype, pre-production, production

The French government’s guide for building a RAG-based conversational assistant is clear on the rhythm: a chatbot should be evaluated in pre-production before deployment, then monitored continuously in production to follow its key metrics and user satisfaction. Before release, technical changes go through automated tests on a reference dataset, plus human validation on a critical sample.

In practice, that gives you four moments:

Re-test after any significant change: a new knowledge base, a new model, a rewritten prompt, a new channel. The same state guide notes that monitoring and re-testing in production help detect performance regressions, and that real cases can be added to your reference dataset over time.

Build a feedback mechanism in from the first prototype. The state guide recommends a simple rating or comment mechanism in the prototype, because without it you have nothing to feed the next iteration.

Preparing the test

French public-sector guidance on integrating user tests lists the preparation steps: define objectives, target the users of the tool, and write a protocol with realistic tasks. Here is how that looks for a chatbot.

Set one to three objectives

Vague goals produce vague findings. Compare:

• Weak: “See whether people like the bot.”
• Strong: “Check that a first-time visitor can find a delivery status in under two minutes, and that the bot hands over to a person when it cannot answer.”

Each objective should map to something you can observe: a task completed, a wrong answer, a moment of hesitation, an escape to the phone number.

Write realistic tasks and scenarios

A task describes a situation, not an instruction. Do not write “ask the chatbot about returns.” Write: “You bought a jacket last week, it is the wrong size, and you want your money back. See what you can do.” The participant then chooses their own words, which is exactly what real customers do.

A good protocol for a customer-service bot includes:

• Easy tasks the bot must nail (opening hours, delivery times).
• Realistic but messy tasks (a message with a typo, two questions in one sentence, a complaint with frustration).
• Out-of-scope requests, to see how the bot declines and whether the exit to a human is visible.
• One trap, such as a question whose answer changed recently, to check the knowledge base is current.

Keep to five or six tasks. Participants tire quickly, and each task should be completable in a few minutes.

Decide what success looks like

Before the first session, write down what counts as success for each task: completed alone, completed with a hint, abandoned, wrong answer accepted as right. That last category matters most with generative bots, where a confident wrong answer is worse than “I don’t know.”

How many participants, and who

For a qualitative test, a widely used rule is that five users uncover roughly 85% of major problems, so plan 5 to 10 participants per segment. Treat it as a rule of thumb, not a guarantee: if your chatbot serves very different audiences, each audience is its own segment.

Examples of segments:

• New visitors versus existing customers.
• Consumers versus professional buyers on a B2B site.
• Customers who use WhatsApp daily versus those who prefer email.

The same source stresses recruiting participants who match the real target of the product. Colleagues are fine for a dry run to fix your script, but they know the product and the vocabulary, so their results will flatter the bot. Recruit from your customer list, your newsletter, your support tickets or an outside panel. Mix ages and comfort levels with digital tools.

Running the session

The public-sector guidance describes sessions that can be face to face or remote, ideally with two people: one leads, the other takes notes. The protocol can include think-aloud, and sessions are recorded with the user’s consent.

A simple running order for a 30 to 45 minute session:

1. Welcome (3 minutes). Explain that you are testing the chatbot, not the person. There are no wrong answers.
2. Consent and context (3 minutes). Ask for permission to record. Explain how the recording will be used and when it will be deleted. Ask two or three questions about habits with chatbots.
3. Tasks (20 to 30 minutes). Read each scenario aloud. Ask the participant to say what they are thinking. Stay quiet. If they get stuck, wait before offering a hint, and note when you did.
4. Debrief (5 to 10 minutes). Ask what felt easy, what felt strange, whether they trusted the answers, and whether they would use the bot instead of calling.
5. Short questionnaire. Use a standard scale (see below) so scores compare across rounds.

What the note-taker records

• The exact words the participant typed, including failures.
• Where the bot misread the intent.
• Moments of hesitation, rephrasing or irritation.
• Whether the participant noticed the handover option, and used it.
• Any answer the participant believed but that was wrong.

Do not coach. If a participant says “I’d just call them,” that is a result, not a problem with the participant.

Analyzing results and sharing them

The same French guidance suggests that teams classify and quantify findings, use affinity diagrams, and share a report covering objectives, target users, hypotheses, key learnings, decisions and next steps. It also suggests SUS, the System Usability Scale, as a measure.

From notes to priorities

1. Put every observation on its own sticky note (physical or digital).
2. Group notes by theme, such as “misunderstood intent,” “wrong answer,” “no way out,” “tone.” That is an affinity diagram.
3. Count how many participants hit each issue, and rate its severity: blocks the task, slows the task, or merely annoys.
4. Rank by severity times frequency. Fix the top five before anything else.

What to measure

Combine what you observe in sessions with what you track in production. The French state guidance on continuous evaluation says effective monitoring tracks both AI-specific metrics such as retrieval accuracy and product KPIs such as adoption and session abandonment.

For the production side, our guide to chatbot KPIs that predict success goes deeper on which numbers to put on a dashboard.

Freeze a reference set

Turn the real questions from your sessions into a fixed list of test conversations with expected outcomes. Replay it against every new version before release. That way a fix for one problem does not quietly create another, and your human tests can focus on new situations instead of re-checking old ones.

Write the report on one page

Open with the decision you need. Then: objectives, who you tested, the five biggest findings with a quote or clip each, what you will change, what you will re-test, and when. Reports that end with owners and dates get acted on.

Protecting participants’ data: GDPR and the CNIL

A test session produces personal data: voice, video, chat transcripts, sometimes details a participant shares unprompted. In France, the CNIL’s guidance on chatbots sets the frame. Personal data should be kept only as long as needed for the purpose of the processing. If sensitive data is collected unexpectedly, the CNIL advises warning users beforehand and purging it immediately or at regular intervals.

Translated into test practice:

• State the purpose. Tell participants the recording is used to improve the chatbot, and for nothing else.
• Get consent for recording, separately from consent to take part.
• Set a deletion date for recordings and raw transcripts, and keep only anonymized findings after that.
• Warn testers not to share real health, financial or identity data. Give them fictional personas and test accounts.
• Purge anything sensitive that slips into a transcript, rather than letting it sit in a shared folder.
• Limit access to the people analyzing the results.

If you plan to reuse real conversations to train or tune a model, that is a separate processing with its own legal basis. Our CNIL checklist for GDPR and AI walks through what to settle before any training run. For regulatory questions specific to your case, involve your data protection officer.

Common mistakes that spoil a chatbot test

• Testing only the happy path. Real users ramble, misspell and get angry. Include messy tasks.
• Leading the participant. Telling them which button to press hides the real problem.
• Testing with colleagues only. They know too much.
• Treating a single round as proof. The bot changes, so the test must repeat.
• Ignoring the exit. A bot with no clear route to a person fails exactly when stakes are highest. Our 30 to 60 day handoff pilot guide covers how to design and test that route.
• Fixing symptoms. If five people misunderstand the same answer, rewrite the answer, not the instructions for users.

More patterns are in our review of common chatbot mistakes in customer support.

Our point of view: test early, test with real cases

We build chatbots, agents and automations for businesses, and the pattern we see is consistent: the teams that do well treat user testing as part of delivery, not as a final check. That is why our approach starts with a proof of concept, which our site describes as often deliverable within 14 days, so real users can react to something concrete early. A short PoC is the cheapest way to learn which questions your customers really ask.

Our process is built around that learning: a free initial audit, no long-term contract required, and production tools connected to the systems your team already uses, such as CRMs, e-commerce stores and WhatsApp. Testing with real conversations is how we turn a prototype into a tool that is designed to be measurable in production. We also deploy with GDPR in mind, which is why we ask clients to settle consent, retention and purge rules before the first participant sits down.

Our advice: start with a narrow scope, test with five to ten real customers, fix the top issues, and widen the scope only when completion rates and answer accuracy hold.

— Botiqueai

Run your tests on a real pilot

If you would rather test a working bot than a mock-up, we can build a custom chatbot for your website, store or WhatsApp channel, and set up the feedback loop with you. You can see what we offer on the BotiqueAI homepage, or look at Aria, our AI chatbot for websites and e-commerce. For Shopify stores, our app plans start at $19 per month (Starter) and $49 per month (Pro) on the Shopify app page.

FAQ

How many people do I need for a chatbot user test?

For a qualitative test, 5 to 10 participants per user segment is a common range, and around five users are said to surface most major problems. If you serve distinct audiences, test each separately. Add rounds rather than participants: five people, fix, then five more.

Can I test a chatbot remotely?

Yes. Public-sector guidance on user testing accepts face-to-face or remote sessions. Share a screen, ask people to think aloud, and record with their consent. Remote sessions make it easier to reach customers from different regions of France.

Should I use automated tests or human testers?

Both. Automated checks on a reference dataset catch regressions before each release, while human validation covers critical cases and the things scripts cannot judge, like tone or trust. The state guidance describes both as part of pre-production evaluation.

How long can I keep test recordings?

Only as long as your stated purpose requires. The CNIL says personal data should be retained no longer than needed for the processing purpose. Set a deletion date before recording, tell participants, and keep only anonymized findings afterward.

Which usability measure should I start with?

Start with task completion, then add a standard questionnaire such as SUS so you can compare rounds. Once live, add abandonment and handover outcomes to your dashboard.

Sources

• Evaluating Chatbots to Promote Users’ Trust (arXiv)
• Evaluations continues, Guides IA dans l’État
• Prototypage, Guides IA dans l’État
• Comment intégrer concrètement les tests utilisateurs, webinaire 2026
• Tests utilisateurs: cadrer ses tests, laConsole
• Chatbots : les conseils de la CNIL pour respecter les droits des personnes

Recommended

• GA4 Chatbot Integration: A Practical Tracking Guide for France