Course contents · How AI agents actually work

Lesson 9 of 13

RAG: finding the right page.

How an agent answers from your documents without reading all of them, and the three ways it goes wrong.

Joe LeungJoe LeungCTO and co-founder, LaunchLemonade7 minReading time

Think of an open-book exam. You don't photocopy the whole library onto your desk. You bring the one textbook, and you use the index to find the three pages that matter for this question.

The problem RAG solves

RAG stands for retrieval-augmented generation. It answers a simple problem: your documents don't fit in the context window. And even if they did, you wouldn't want them all there. Remember the messy desk from the context lesson: more isn't better if the one page that matters gets lost.

So instead of handing the model everything, you hand it the few passages that matter for this question, at the moment it needs them.

Retrieve, augment, generate

RAG, step by step4 steps
  1. A question arrives
  2. Retrieve: find the handful of passages relevant to it
  3. Augment: put those passages into the model's context
  4. Generate: the model answers using what it has just been shown

The generation part is the easy bit now. Models are good at answering from what's in front of them. The quality of a RAG system is mostly the quality of what it finds.

Searching by meaning

The clever part is the finding. Most RAG systems use embeddings: the meaning of a chunk of text turned into a list of numbers. Chunks with similar meanings get similar numbers, so search works by similarity of meaning, not just by matching keywords.

A question and the passage it findsExample
Question: "Can I bring my dog to the office?"
Found: the pet policy, even though the word "dog" never appears in it.

How well that works depends on choices you can control: how you split documents into chunks, what you embed, how many results you return, and whether you re-rank them before they reach the model.

How it goes wrong

Too little
The model guesses, confidently.
Too much
You're back to the messy desk, and the important detail gets missed.
The wrong thing
The model faithfully answers from the wrong page. And cites it.

For most business uses, like your policies, your product documents and your knowledge base, showing the model the right page at the right moment beats retraining the model itself. And agent memory works the same way: it is RAG pointed at the agent's own past instead of a document library. Same mechanism, different filing cabinet.

Running example

The running job

Chasing last month's unpaid invoices, one lesson at a time.

To chase one client, Priya's agent needs two things it can't hold in its head: the invoice, and the studio's payment terms. The invoice is a record, so it comes straight from the read invoices tool, looked up by client. No searching by meaning needed. The terms are a document, and that's where RAG comes in. It retrieves just the paragraph that says when payment is due and how to pay, then drafts the reminder from what it has just been shown.

The failure to watch for is retrieving the wrong thing. The studio changed banks last year, and the old terms are still in the folder. If the agent pulls the old paragraph, the email is polite, fluent and accurate to the wrong page: it gives the old bank details. That's why the old terms should come out of the folder, and one more reason Priya reads every email before it goes.

Practice

Try it yourself

About five minutes, in any AI chat app

Be the retriever

  1. Copy three or four short sections from a policy or handbook you know (for example holidays, expenses and working from home) into any AI chat app.

  2. Ask it a question that uses different words from the text, such as "Can I claim for a taxi home after a late meeting?" when the policy talks about "travel outside office hours".

  3. Now ask a question the sections don't answer at all, and see what it does.

What to look for

It should find the right section even when the words don't match. That's searching by meaning. On the unanswerable question, notice whether it says it doesn't know or fills the gap with something plausible. That gap-filling is what "retrieve too little" looks like.

Common mistakes

  • Pasting every document in and hoping for the best.

    Give the model the few passages that matter. More text makes the important part harder to find.

  • Blaming the model when the answer is wrong.

    Check what was retrieved first. Most RAG problems are finding problems, not writing problems.

  • Trusting a citation because it's there.

    A citation only shows where the answer came from, not that it was the right page. Spot-check it.

Quick check

3 questions to see what stuck. Nothing is scored or saved.

  1. 1. What does the "retrieve" step do?

  2. 2. Why can an embedding search find the pet policy when you ask about a dog?

  3. 3. An agent answers fluently, cites a source, and is wrong. What's the most likely cause?

Remember

RAG is an open-book exam for an agent. Its answers are only as good as the pages it finds.

Sources and further reading

  1. 1.The LinkedIn post this lesson grew out of: RAG
  2. 2.The original RAG paper, Lewis et al. (2020)

Optional

Try it in LaunchLemonade

Upload documents, web pages or connected sources to Knowledge, and the assistant answers from them. Everything above works in any AI tool.

Add documents

Finished this lesson?

0 of 13 done in this course. Progress stays in this browser, no account needed.