Course contents · Get better answers from your AI assistant

Lesson 5 of 5

Is it actually working?

"It feels faster" isn't a number. A baseline, an honest count of time saved, a test set of real tasks, and a decision: expand, improve, pause or retire.

Joe LeungJoe LeungCTO and co-founder, LaunchLemonade8 minReading time

You wouldn't judge a new hire on how confident they sound. You'd look at the work: how much they got through, how much you had to fix, and whether anything went wrong. Judge an assistant the same way.

Start before you launch

Before the assistant takes on the job, write down how the job goes today. How long one takes, how many get rewritten, how often something goes wrong. That's your baseline.

Without it, you're comparing against a memory, and memories are generous to whatever is new. A week or two of rough notes is enough.

Count time saved honestly

Time saved isn't the manual time minus the assistant's time. Someone has to read the draft and fix what's wrong.

Time savedExample
Manual time
− the assistant's time
− time spent reviewing
− time spent on rework

Leave out review and every estimate looks better than it is. If reviewing takes longer than doing the job by hand, that's worth knowing early.

A test set of real tasks

Course one's evals lesson made the case. Here's a practical version: ten to twenty real tasks you can run again after every change. Include the awkward ones, because easy cases always pass.

  • A normal request.
  • One with a detail missing.
  • Two documents that disagree.
  • A vague request that needs a question back.
  • Something it should refuse, or pass to a person.

For each one, write down what a pass looks like before you run it, so you don't move the goalposts after a good-looking demo.

Name the failure

"Bad answer" tells you nothing. Label each failure with what went wrong, and the label tells you where to look.

Missed a conflict
Look at the instructions: is there a rule to flag conflicts?
Used an old document
Look at the knowledge: what's in there that shouldn't be?
Ignored the format
Look at the system prompt: is the format clear, with an example?
Didn't hand it on
Look at the guardrails: does it know when to escalate?

Change one thing at a time

A clearer rule, a better document, or a different model. Change one, then run the same test set. If you change three things at once, you won't know which one helped, or which one broke something else.

Try the instructions and documents before you swap the model. They're cheaper to change, and they're the most common cause.

Then decide

Expand
The work is good and people use it.
Improve
It's useful but has gaps you can name.
Pause
It's unreliable or unsafe. Stop until it's fixed.
Retire
Nobody needs it. That's a result too.

A polished answer isn't proof of value. A few assistants you trust beat a big library nobody uses.

Running example

The running job

Chasing last month's unpaid invoices, one lesson at a time.

Before switching on the tenant helper, Sam's team noted a fortnight of replies: how long each took, and how many were rewritten before sending.

A month in, Sam runs a test set of fifteen real tenant messages, including a fee question, a message with no address, and a deposit dispute that should come straight to Sam. Two fail. One is labelled "used an old document": an old copy of the repairs process had crept back into the knowledge. The other is "didn't hand it on". Sam removes the old copy, adds one escalation rule for deposit disputes, and runs the same fifteen again.

Counting review time, replies are faster than before. Sam's decision: improve, then expand to landlord enquiries.

Practice

Try it yourself

About five minutes, in any AI chat app

Build a five-task test set

  1. Pick one job you use an AI tool for. Write down roughly how long it takes you by hand.

  2. Write five real test tasks for it: a normal one, one with a missing detail, one with conflicting information, a vague one, and one it should refuse or hand on.

  3. Write what a pass looks like for each, then run them and label every failure.

  4. Make one change, run all five again, and compare.

What to look for

Whether your change fixed the failure without breaking a task that used to pass. That's why you re-run the whole set, not just the one that failed.

Common mistakes

  • Judging it on how it feels.

    Write a baseline before launch and compare against it.

  • Counting time saved without review and rework.

    Subtract the time spent checking and fixing, or the number flatters it.

  • Swapping the model at the first failure.

    Label the failure, then fix the instructions or documents first, one change at a time.

Quick check

3 questions to see what stuck. Nothing is scored or saved.

  1. 1. What's missing from "manual time minus the assistant's time"?

  2. 2. Why write down a baseline before launch?

  3. 3. A test fails with "used an old document". Where do you look first?

Remember

Measure it like a new hire: a baseline first, time saved after review and rework, a test set with the awkward cases, one change at a time, then expand, improve, pause or retire.

Sources and further reading

  1. 1.The blog post this lesson grew out of: How to measure AI agent performance without guesswork

Finished this lesson?

0 of 5 done in this course. Progress stays in this browser, no account needed.