Course contents · How AI agents actually work

Lesson 5 of 13

Memory: the filing cabinet.

How an agent carries useful facts from one conversation to the next, and why remembering everything is as bad as remembering nothing.

Joe LeungJoe LeungCTO and co-founder, LaunchLemonade7 minReading time

Think of an office where you hot desk. Every morning you sit at a clean desk with nothing on it. That desk is the model's context: each new conversation starts empty. But the filing cabinet in the corner stays put, and you always know where it is. That cabinet is memory.

Memory is a system, not a recording

An LLM forgets everything between calls. Each new conversation is a new desk with nothing on it. If an agent seems to remember you, something around the model is saving facts and putting them back on the desk at the right moment.

So memory for an agent isn't a single feature. It is a system for deciding what to keep, how to find it again, and when to let it go. Like context, it is harder to get right than it sounds.

Context
the desk: what the model can see right now, for this one decision
Memory
the filing cabinet: what survives after the desk is cleared

Three jobs: write, retrieve, maintain

The memory cycle4 steps
  1. Write: decide what's worth keeping
  2. Retrieve: find the few memories that help with this turn
  3. Put them on the desk (into the context)
  4. Maintain: update, merge and forget as things change

Write. Keep the salient facts, not the transcript. "Prefers a phone call to email" is worth filing. The whole conversation where you mentioned it is not.

Retrieve. When a new task starts, fetch only what helps with it. This is the same idea as RAG, which has its own lesson later, pointed at the agent's own history instead of a library of documents.

Maintain. Facts go out of date, and old facts can contradict new ones. Memory that only ever grows slowly rots. Good systems update, merge and forget. Some people call this tidying-up step "dreaming".

How memory goes wrong

The failure modes are fairly predictable once you see memory as a filing cabinet:

Remember everything
expensive and noisy, and finding the right thing gets harder
Forget everything
no continuity: every session starts cold
Retrieve the wrong thing
confidently wrong, which is worse than uncertain

The last one is the sneaky one. A memory that was true in March, brought back in October, looks just as authoritative as a fresh fact. The model has no way of knowing it has gone stale unless the system tells it.

Memory and privacy

Anything an agent remembers can come back later, in a different conversation, possibly in front of a different person. That is useful when it is your own preferences. It is a problem when it is a client's private details, or advice based on something that is no longer true.

So the questions worth asking of any AI tool with memory are simple: what does it save, can I see it, can I edit or delete it, and who else can it surface it to?

Running example

The running job

Chasing last month's unpaid invoices, one lesson at a time.

Priya's invoice chaser runs once a month. Each run starts at a clean desk. Without memory, it would treat every client as a stranger every month.

Worth filing
this client disputed an invoice in May; this one prefers a phone call; this one always pays a week late but always pays
Not worth filing
every reminder email ever sent, word for word
Needs maintaining
the dispute was settled in June, so that note should be updated, not left to block future reminders

At the start of each run, the chaser retrieves only the notes for the clients it is about to chase. The rest stays in the cabinet.

Practice

Try it yourself

About five minutes, in any AI chat app

Be the filing cabinet

  1. Open any AI chat app. Paste in a short, messy note about a client, the kind you'd jot after a call. Include a few useful facts and a lot of chatter.

  2. Ask: "If you could only remember three facts from this for next month, which would they be, and why?"

  3. Then ask: "Which of these facts might be out of date in six months, and how would you check?"

What to look for

A good answer keeps durable facts (preferences, standing arrangements) and drops the chatter. Notice that it's the system around the model, not the model, that would have to actually save those three facts and bring them back next month.

Common mistakes

  • Saving whole conversations and calling it memory.

    Save short, specific facts. A transcript archive is the messy desk in a filing cabinet.

  • Never letting anything go.

    Give memories a way to be updated or removed. Old facts that contradict new ones cause confident mistakes.

  • Forgetting that memory can surface later.

    Decide up front what must never be remembered, such as client personal details, and check what your tool actually stores.

Quick check

3 questions to see what stuck. Nothing is scored or saved.

  1. 1. In the desk and filing cabinet picture, what is memory?

  2. 2. Which of these is the best thing to write to memory?

  3. 3. Why is retrieving the wrong memory worse than having none?

Remember

Memory is a filing cabinet with three jobs: write what matters, retrieve what helps, and maintain it so it doesn't rot.

Sources and further reading

  1. 1.The LinkedIn post this lesson grew out of: Memory

Finished this lesson?

0 of 13 done in this course. Progress stays in this browser, no account needed.