How to Test a Chatbot Before Launch: A 30-Minute Checklist

Most chatbot disappointments could have been caught in half an hour of deliberate testing. You do not need a technical team, just a list of questions and a willingness to be a difficult customer.

1. Set Up Your Test Sheet

Use a simple table with four columns: the question you asked, what the bot said, whether it was right, and what you will change. Ask someone who did not write the answers to help; they will phrase things differently from you.

Fill in the "what I will change" column as you go, rather than saving it for later. A note such as "add refund window to the Q&A pair" or "write a new answer for international shipping" turns a vague sense that something is off into a list of jobs. Keep the sheet in a shared document so that whoever edits the content can work from it directly.

2. Test Known Questions

Take your top 15 to 20 real customer questions and ask each one, word for word and then in your own words. For every answer, check three things: is it correct, is it complete including exceptions, and is it current.

Do not only check that the answer is right; check that it is the right level of detail. An answer that is technically correct but leaves out an important exception, such as "free shipping" without mentioning the minimum order, can cause as much trouble as a wrong one. Asking the same question in two phrasings also shows whether the bot finds the content reliably or only when the wording matches.

3. Test Questions the Bot Should Not Know

This is the test that matters most. Ask about things that are not in your content: a product you do not sell, a policy you have never written, a competitor's prices. The right behaviour is to say it has no confirmed information and offer a way to reach you, not to produce a confident guess. Why this matters is covered in why bots make up answers.

Run at least ten of these. Include some that are close to your topics, such as a service you almost offer, because near-misses are where bots are most tempted to blend two answers into a new one. If any reply sounds confident but is not in your content, note it as a high-priority fix. A bot that guesses on out-of-scope questions will guess in front of real customers too.

4. Test Different Wording

Real visitors do not write the way you do. Try each of these styles of message and note how the bot copes.

• Short and messy: "refund??", "opening hrs", "do u deliver".
• Long and rambling: a paragraph with the real question buried in it.
• Typos and slang.
• Two questions in one message.
• Wrong assumptions: "Since you're closed on Sundays, can I..." when you are not.

Real visitors rarely write in complete, tidy sentences. They abbreviate, misspell, ask two things at once and sometimes assume something false. The goal of this step is to confirm that the bot handles ordinary messiness gracefully: it should find the right answer when the wording is clumsy, and politely correct or clarify a wrong assumption rather than agreeing with it.

5. Test the Handoff

Follow this sequence from the visitor's side and then from your team's side. Each step should work without help.

1. 1Ask something the bot cannot answer and follow the path to a person.
2. 2Type "I want to speak to a human" and confirm it responds immediately.
3. 3Submit the contact form with your own details and check the notification and auto-reply arrive.
4. 4Reply from your inbox and confirm the message shows up in the chat window.
5. 5Check that the bot stops replying once a teammate joins.

For design ideas see when a chatbot should hand over.

Do this on a phone as well as a computer, and from the visitor's side and your team's side. Many handoff problems are invisible until you try the whole round trip: the visitor sends details, a teammate receives them, replies, and the visitor sees the answer. If any link in the chain is slow or confusing, fix it now, when it costs nothing.

6. Try to Break It

Be adversarial for ten minutes. Tell it to ignore its instructions, ask it to reveal its setup, change the subject to something unrelated, paste a long block of odd text, ask for something it should refuse. A well-constrained bot stays on topic and hands over. For context see our prompt injection guide.

You are not trying to prove the bot is perfect, only to find out how it behaves under pressure. The acceptable outcomes are a polite refusal, a return to the topic or a handoff. The unacceptable outcomes are following the odd instruction, revealing internal setup details or answering outside its scope with confidence. If you see any of those, tighten the bot's instructions and content limits and retest.

7. Check the Experience

Finally, look at the overall experience. These checks catch problems that content testing alone will not reveal.

• Mobile: open the site on a phone and use the chat one-handed.
• Tone: does it sound like your business?
• Disclosure: does it say it is automated? See our disclosure guide.
• Speed: do answers appear quickly enough to feel natural?
• Accessibility: can you open, read and close the widget with a keyboard?

Finally, step back and read a few full conversations as a customer would. Is the greeting clear about what the bot can do? Are the replies short enough to read on a phone? Does the tone match the rest of your site? Small details of this kind decide whether visitors keep using the chat or close it after one message.

When you are done, decide on a launch rule. For example: no wrong answers on your top twenty questions, a working handoff, and no confident replies on out-of-scope tests. If the bot meets the rule, launch; if not, fix and retest. Having the rule written down stops launch from becoming a matter of hope.

Summary

Testing a chatbot is mostly about asking awkward questions. Cover known answers, unknown questions, different wording, the handoff and deliberate attempts to break it, then fix the content and run it again. Thirty focused minutes before launch saves a lot of public embarrassment.