AI Chatbot QA Checklist: 25 Checks Before and After Launch
By Muhammad Usman · · 6 min read
Knowledge base
- Every important topic has one current article (pricing, delivery, refunds, cancellations, opening hours).
- No two articles contradict each other.
- Old promotions and discontinued products are removed.
- Articles state limits clearly ("we don't ship to…", "not covered by warranty").
Accuracy
- The bot answers your 20 most common questions correctly.
- Prices, dates and numbers match your articles exactly.
- The bot says "I don't know" instead of guessing when the articles don't cover a question.
- Answers don't invent links, phone numbers or email addresses.
- Required statements appear where needed (for example "prices include VAT").
Escalation to a human
- Complaints are handed to a human.
- Legal threats and chargebacks are handed to a human.
- Refund disputes are handed to a human.
- Angry customers (all caps, repeated questions) are handed over quickly.
- The hand-over actually works and someone receives it.
Safety and policy
- The bot never mentions competitors or makes guarantees you don't offer.
- It refuses to agree to prices, discounts or contracts.
- It resists "ignore your instructions" style manipulation.
- It doesn't give medical, legal or financial advice unless you allow it.
Privacy
- Customers are told they are talking to an AI.
- Personal data (emails, phone numbers, card numbers) is masked in logs and reports.
- Conversation data is deleted after a set period.
Performance
- Replies arrive within a few seconds.
- The bot replies to every message (no silent drop-outs).
After launch
- Every answer is checked against your docs, with alerts for made-up answers.
- Your key questions are re-tested every night and after every change.
- A weekly summary shows accuracy, escalations and the articles to fix.
Automate the checklist
Many of these checks can run automatically. ProofMyAI grades answers against your docs, enforces your own rules ("never say…", "always hand over when…", "must say…"), detects frustrated customers and missing replies, masks personal data, and alerts you. Try it free.
✅ Key takeaways
- Most chatbot quality problems start in the knowledge base.
- Test your top 20 questions, escalation paths and 'never say' rules before launch.
- Protect personal data and tell customers they're talking to an AI.
- After launch, check every answer and re-test after every change.
❓ Frequently asked questions
What should I test before launching an AI chatbot?
Your most common questions, pricing and policy answers, hand-over to a human for complaints and legal topics, 'never say' rules, resistance to manipulation, and personal-data masking.
How often should a chatbot be tested?
Before launch, after every change to its prompt, model or knowledge base, every night for the most important questions, and continuously through live monitoring.
Who should own chatbot quality?
Usually the support or customer-experience lead, with the developer who maintains the bot. A weekly quality summary keeps both informed.