Research insights
Customer Service Chatbots: Test Resolution and Human Handover
A customer-service chatbot should be judged by whether it helps someone resolve a defined task or reach the right person. A fast reply is not a successful outcome when the answer is wrong, incomplete or sends the customer through repeated questions. Service research needs to follow the whole journey.

On this page
Start with a limited scope: product information, an order-status enquiry or a request for contact. These tasks have different data and system requirements. Describe what the chatbot is authorised to do and where human assistance remains necessary.

Choose tasks before choosing the interface
Review recent support requests and group them by purpose, complexity and consequence of error. A question about opening hours is different from a request involving a disputed payment. Volume alone should not determine which tasks enter the pilot.
Separate providing information from completing an action. Explaining how to cancel an order is not the same as cancelling it. The interface should make that distinction clear and confirm what has actually happened. Test the scope with service staff as well as customers.
Build realistic conversation tests
Use representative scenarios with fictitious account and order details. Include short questions, misspellings, incomplete descriptions and requests that fall outside scope. For Vietnamese conversations, test everyday wording and the language actually used by intended customers rather than relying only on formal scripts.
A hypothetical delivery enquiry can reveal several problems: an unavailable status, a customer who cannot find an order reference, or a request to change the address. Define the correct response and escalation for each scenario before running the test. Do not reward a confident answer when the system lacks the information.
Check answer quality separately from tone
Assess accuracy against approved service information, completeness for the task and clarity about uncertainty. Then ask whether customers understand the answer and know what to do next. Friendly wording can improve readability without resolving a missing fact.
For AI-enabled designs, include systematic evaluation rather than assuming answers improve automatically with use. The voluntary NIST AI Risk Management Framework provides a reference for considering trustworthiness in AI design, use and evaluation. It is not evidence that a particular chatbot is reliable.
Test the handover under difficult conditions
Make the route to a person visible when the task requires it. Check whether the receiving staff member gets the relevant conversation context so the customer does not have to start again. Explain expected response timing when immediate human support is unavailable.
Include failed authentication, repeated misunderstanding and unavailable systems in the test plan. Ask participants whether they can stop the conversation or choose another support route. Avoid collecting personal information merely because the conversational format makes it easy to ask.
Measure resolution, effort and total cost
- Task completion: Was the requested information or action actually delivered?
- Answer quality: Was the response correct and sufficient?
- Customer effort: How many steps and explanations were needed?
- Repeat contact: Did the same unresolved issue return?
- Handover: Did the customer reach suitable support with context intact?
Do not equate a closed chat with a resolved issue. Check outcomes through suitable follow-up and service records. Include maintenance, content review, staff intervention and escalation work when evaluating costs; reduced chat handling does not automatically mean lower total service cost.
Expand only after learning from a pilot
Compare the pilot with an appropriate existing support route and review results by task. A good result for simple information requests does not justify expansion to more consequential actions. Update approved information and retest when policies or system connections change.
Keep resolution and loyalty separate
Resolution, immediate satisfaction, intention to return and observed repeat purchasing are different outcomes. A customer may value the chatbot yet choose another brand because of price or availability. Follow a suitable customer cohort over time and use an appropriate baseline before attributing a retention change to automation. Record other product, price and service changes during that period.
Explore market research services or discuss a customer-experience study centred on realistic tasks, resolution and the handover to service staff.
Related reading
Latest posts
- Effective M&A (Mergers & Acquisitions) Opportunity Research
- Market Research: Building a Market Research Report for Investment Funds
Survey Sample Size: How Many Respondents Are Enough?
Survey Data Collection Methods in Vietnam: How to Choose- Technology Adoption in Enterprises: How to Research Employee Behavior


