I was just thinking about how I'd explain some of what I've learned from building AI to someone who's never done it. So yeah, here's one.
So, you've built your assistant and it sounds good. Then you send it out into the wilderness, real users find it, and it doesn't work as well as you thought it would.
And sure, you did test it. You sent it to people you trust, you tightened the loose ends from their feedback, and by the end it sounded like perfection. Now it's out there and it's struggling, and there are a couple of usual suspects.
The first is that testers are nice. They tend to happy-path it. Nobody wants to be weird or horrible to an assistant their friend built, so the testing never really stresses it.
The second is that everything you tested came out of your own head. Even the hostile questions you wrote and fixed, you thought of those. Real users ask things you wouldn't think to ask.
There's a name for deliberately trying to break your assistant, which is red teaming. It's the first half of what I do for businesses. The second half is mending what breaks. If you'd like me to do both for yours, send me your site.
Back to the Yummy Pie Shop. Say a customer walks in and asks for a gluten-free vegetarian pie.
Fail.
If you read that looking at tone, nothing's wrong. It's polite, it's fast, it got the vegetarian part right. If you read it looking at what she actually asked, she was offered something that could make her seriously ill.
Assistants fail like this. They rarely produce gibberish or an error message. More often they give a confident, friendly, well-spelled answer to a question nobody asked, which is why a few rounds of friendly testing can't tell you whether yours is any good.
ok, so what's an eval
A set of test messages for your assistant, typed or spoken, each with a note written beforehand saying what a good result looks like. Sometimes that's a particular answer. Sometimes it's a list of things the answer has to do. You send the messages, then check what came back against your notes.
“I need a gluten-free vegetarian option” is a test. So is “do you deliver to Berkhamsted”, “my last order was cold” and “what time does the train go”. Some tests are a whole back and forth instead of one message.
your customers won't say “vege medley”
If you write the tests yourself, you'll ask about what you built it to do, in the words on your own product list. Customers say “the veggie one”, or “something without meat”, or “is the veg one okay for my daughter, she's coeliac”. They ask three things in one sentence, and they ask about the train because you're on the high street and you look like someone who'd know.
So when a real person says something your assistant fumbles, write down their exact words. Not a summary. The words.
so what counts as a pass
A customer wants a chicken pie at half eleven and there aren't any. The assistant offers the steak. Good answer.
Now the gluten customer gets offered the shortcrust. The assistant behaved identically both times, so what separates them is what happens if she says yes. Say yes to the steak and you've got a steak pie you weren't planning on. Say yes to the shortcrust and you're ill by teatime.
Writing that down is harder than it sounds, because you have to decide what your business actually believes. The first attempt is usually “don't suggest substitutes when someone mentions an allergy”, which falls over the moment you notice she never said allergy. She said gluten free.
The written-down version of a pass is called a rubric. It should be specific enough that two people scoring the same reply would agree. “Handled it well” isn't a rubric. “Says what's safe and what isn't before recommending a pie” is.
run them more than once
Depending on how the assistant is set up, it can give the good answer first time and the shortcrust answer the second. So run each test more than once, on different days if you can. Mark each run pass or fail, with a note on what went wrong.
then the annoying part
You fix it. The assistant is told to say what's safe before it names any pie, and when you rerun the test, it does.
Safe, but not your shop.
The warm assistant who offered the steak now sounds like a compliance notice. This is a regression, where a change you made fixes one thing and breaks another. It's the check people skip, because after a fix you retest the thing you fixed.
So one line per test isn't enough. “Still sounds friendly” is too vague to score, so I'd split it into four checks.
- Safety. It says what is and isn't gluten free before recommending anything, never offers a pie with gluten in it, and says so when it isn't sure.
- Grounding. Everything it offers is something the shop really does.
- Helpfulness. It gives her a real next step.
- Voice. First person, everyday words, none of “cannot recommend” or “unable to”, and it says “gluten free” because that's what she said.
The compliance notice passes safety and grounding and fails the other two. Now try this one, assuming the Thursday pie is real.
Safe, and still the shop.
It passes all four. Whether the pie is actually okay for someone coeliac isn't the assistant's call, so it hands the kitchen question to a person.