Course contents · How AI agents actually work

Lesson 13 of 13

The whole picture.

Every part of an agent on one page, with one job running through it, and the questions to ask before you build or buy one.

Joe LeungJoe LeungCTO and co-founder, LaunchLemonade8 minReading time

Look at a diagram of a working agent and find the model. It's one small box. Its whole job is to look at what's on the desk and pick the next step. Everything else on the page is the system around it.

The model is the small box

Twelve lessons ago we started with "what is an LLM?". The answer was: something that predicts the next piece of text. That's still all the model does, even inside the most capable agent. At each turn of the loop it looks at what's in front of it and picks the next step.

Everything else is the system around it:

The goal
the brief: what the job is and what good looks like (prompts)
What it can see
the desk and the filing cabinet (context, memory, RAG)
What it can do
its tools, and the servers it can reach (tools, MCP)
How it works
the loop, the route and who it hands work to (agents, workflows, subagents)
What it may do
what it's allowed to send or change without asking (permissions)
When it stops
the stop condition, the budget and the checks on the result (agents, evals)

The part you control

My estimate is that the system is maybe 80% of the picture. Whatever the exact number, it's the part you control.

Frontier models improve on someone else's schedule. Very few of us will ever build one, and most of us can use roughly the same ones. The system around the model is different. You can improve it yourself, this week: a clearer brief, a tidier desk, one fewer tool, a better stop condition, a test set that catches mistakes.

That's also why two products "powered by" the same model can feel worlds apart. The difference is everything outside the small box.

Before you build or buy an agent

Whether you're building one yourself or choosing a product, these questions come straight from the course:

  1. 01What exactly is the job, and what does a good result look like?
  2. 02Could this be a workflow, with the steps written down in advance? If not, which step needs judgement?
  3. 03What does it need to see at each step, and what can it safely ignore?
  4. 04Which tools does it need, and which can it do without?
  5. 05What should it remember between runs, and what should it forget?
  6. 06Which actions are read-only, which can be undone, and which need a person to approve?
  7. 07Does it have all three parts of the lethal trifecta: private data, untrusted content, and a way to send things out?
  8. 08How does it know when it's done, and what stops it if it isn't?
  9. 09How will you test it, more than once, before you trust it with real work?
  10. 10When something goes wrong, how will you find out, and how will that case join your tests?

Where to go from here

You don't need to remember all thirteen lessons. Remember the shape: a small model in the middle, a system around it, and every part of that system a choice you make.

The best next step is to pick one real, boring, repeatable job of your own and sketch it the way we've sketched Priya's. Start at the workflow end, keep the tools few, put a person before anything irreversible, and test it more than once.

Running example

The running job

Chasing last month's unpaid invoices, one lesson at a time.

Here is Priya's invoice job, assembled from every lesson in the course:

Chasing overdue invoices, end to end10 steps
  1. First working day of the month: the schedule starts the run (workflows)
  2. A brief says what the job is and what a good reminder looks like (prompts)
  3. Read-only tools pull the overdue invoices from the accounting software (tools, MCP)
  4. Only the facts this step needs go on the desk: client, amount, due date, recent emails (context, RAG)
  5. Memory recalls each client's history, such as "always pays late but always pays" (memory)
  6. Decision point: paid since, or disputing? Then skip this client (agents, workflows)
  7. The model drafts a reminder in Priya's tone (what is an LLM)
  8. Priya approves each email before it is sent (permissions)
  9. The spreadsheet gets a new row, and the run stops when every client is done (agents)
  10. Every mistake Priya spots becomes a test case for next month (evals)

In the first lesson, a bare model could write a polite email and nothing else. Now Priya has a job that runs itself every month: it chases only the clients who owe money, in her voice, never sends anything she hasn't seen, keeps a record, and every mistake she corrects becomes a test for the next run. The model hasn't changed. The system around it has.

Practice

Try it yourself

About five minutes, in any AI chat app

Sketch your own agent

  1. Pick one repeatable job from your own work, something boring that comes round every week or month.

  2. Ask any AI chat app: "Help me sketch an AI agent for this job: [describe it]. For each part, tell me what it needs: the brief, what it should see, which tools, what to remember, which actions need my approval, how it knows it's done, and how I'd test it."

  3. Read the answer against the ten questions above, and push back wherever it gives the agent more tools or freedom than the job needs.

What to look for

Whether the sketch starts at the workflow end, keeps the tools few, and puts you in the loop before anything irreversible. If it doesn't, you now know enough to say why.

Common mistakes

  • Waiting for a better model to fix a disappointing agent.

    Look at the system first: the brief, the context, the tools, the stop condition. That's where most of the gains are.

  • Starting with the most ambitious version.

    Pick one boring, repeatable job, build the workflow version, and turn up autonomy only where it fails.

  • Judging an agent product by the model it uses.

    Ask the ten questions above. Two products on the same model can behave very differently.

Quick check

3 questions to see what stuck. Nothing is scored or saved.

  1. 1. Inside a working agent, what does the model itself do?

  2. 2. Why can two products using the same model feel so different?

  3. 3. In Priya's finished job, what happens before a reminder is sent?

Remember

The model is the small box. The system around it is the part you control, and where most of the difference is made.

Sources and further reading

  1. 1.The LinkedIn post this lesson grew out of: The whole picture

Optional

Try it in LaunchLemonade

When you're ready, build the invoice chaser as a scheduled workflow. When it runs by itself, it asks you first before sending anything. Everything above works in any AI tool.

Start building

Finished this lesson?

0 of 13 done in this course. Progress stays in this browser, no account needed.