How to test an AI website support assistant

Build a balanced question set

Test the common questions the assistant should answer, but also include vague wording, follow-up questions, outdated assumptions and questions that must go to a person. Use real language from enquiries where you can do so without exposing personal information.

• Direct questions with one clear approved answer.
• Questions whose answer depends on a stated condition.
• Natural follow-ups that rely on the previous message.
• Questions outside the approved knowledge that should fall back.

Check the source as well as the wording

A polished answer is not enough. Confirm that the assistant used the current approved source, preserved important conditions and did not blend two policies incorrectly. If the source is ambiguous, fix the source before rewriting the test question.

Test the surrounding experience

Open the assistant on mobile and desktop, navigate between pages, use keyboard controls and test slow or failed network conditions. Confirm that the fallback contact route works and that the widget does not cover important navigation or consent controls.

Define a review rhythm

Review the first conversations closely, then move to a regular cadence based on volume and risk. Re-test after important policy, pricing or website changes. Keep a small regression set of critical questions so improvements in one area do not quietly break another.