A question set for evaluating an assistant
Build a set of ordinary questions, questions with missing information and out-of-scope requests. Define an acceptable answer and human handoff point for each. If the source has no price or condition, the assistant must say so rather than invent a value.
- Track unsupported answers and human handoffs separately.
- Measure cost and response time on the same question set.
Start with the task, not the technology
"How can we use AI?" is too broad a question. It's more useful to find where your team regularly spends time on repetitive text work: answering similar questions, searching documents, copying data from emails into spreadsheets.
Good tasks to start with
- Draft replies to customers based on your knowledge base
- Search across internal documents and instructions
- Extracting data from emails, enquiries and documents into the right fields
- Classifying and routing incoming requests
- Short summaries of long conversations and documents
What a first project looks like
A good first step is a pilot on a single process, for example draft answers to frequent questions based on your knowledge base. Your team still replies themselves, but starts from a ready draft. Over time it becomes clear where the assistant saves time, where it makes mistakes and which materials need improving.
Prepare your materials
An assistant is only as good as its sources. Gather up-to-date documents, answers to frequent questions and rules. Fix outdated or contradictory materials before rollout, or the model will confidently repeat the mistakes.
Quality checks
Before launch, collect test cases: real questions and documents with correct answers. The assistant is checked against them, and you return to them after every change. That way quality is judged on facts, not impressions.
People stay in the loop
Models can make mistakes. Important actions need a clear process: the assistant proposes a draft and a team member decides; complex cases go to a person. It's best to start with an assistant role rather than full replacement.
Costs and data
Model usage via API is usually billed by the amount of text processed, so estimate costs on real examples before launch. Separately decide which data may be sent to an external service and which may not.
Common implementation mistakes
- Starting with the hardest task instead of a narrow scenario
- Connecting outdated or contradictory materials
- Judging quality by eye without test cases
- Removing people entirely from important decisions
- Not estimating model costs before launch