Lesson 19.1619.16 Measuring retrieval

In everyday life. A new pharmacist is tested with a fixed set of prescriptions before she works alone, the same set for every new hire, each with the right answer written down by a senior pharmacist. If the training changes, the same test shows whether it got better or worse.

What you will learn. How a fixed list of questions, written with the expert, measures retrieval on every build, and how a person checks the answers that matter most.

Why it matters. A change that makes retrieval worse can look harmless in review. A fixed set of questions fails the build before any user sees the wrong citation.

One request, start to finish. A builder wants bigger passages, "so the model gets more context". The change is one setting: whole documents instead of one rule each. Before it merges, the build runs a fixed set of questions:

1. Twelve questions, six per company, written with Ana and Luis: for each company, five that the policies answer and one off-topic question they do not. Each has the passages that should come back first, or "abstain". So 10 questions have an answer and 2 should abstain.
2. With one passage per rule, every question that has an answer finds its passage in the top three, and every off-topic question abstains.
3. With whole documents, Cedar Ridge's three documents become three passages, and every citation points at a document's first rule instead of the rule that answers. Three of Cedar Ridge's five answerable questions now fail: found in the top three drops from 10 of 10 to 7 of 10, and the build fails.

The bigger-passage change never merges. The questions said no, before any user saw it.

What is a question set?

A question set is a short, fixed list of questions, each with its expected answer: the file and line that should come back, or "abstain". It is written with the expert, because only the expert knows the right answer. Two measures, in plain words:

• Found in the top three: for questions that have an answer, how often the right passage is among the first three returned.
• Abstained when it should: for questions the policies do not answer, how often the tool abstained.

The set runs in the build, with a pass mark (in the lab, every question, with the test model), so a change to chunking, the model or the threshold that makes retrieval worse fails like any other test.

Is the right passage enough?

Finding the right passage is not the same as giving the right answer. The question set checks retrieval: did the right passages come back? It does not check what the model then says. A model can be handed both purchasing lines and still answer Luis's $400 question wrongly, or let AP pay an invoice that is more than its purchase order, forgetting the rule that holds it. So the lab adds a small answer check, written with the AP expert, as in Chapter 6's review contract. Each row has three parts. A citation gives the line in the file, counting the title, the owner's line and the blank lines: in approvals.md, rule 1 sits on line 5 and rule 4 on line 8.

The AP expert reads each answer and marks it pass or fail. A model's wording changes from run to run, so this check is read by a person, at each model or prompt change, not run on every build. The question set guards retrieval on every build. The answer check guards the answers that matter most.

Figure 19.27. Two checks. The question set guards retrieval on every build. The answer check guards the answer.

What happens when the model changes?

Changing the model means embedding again. Embeddings from two different models are not comparable: their numbers mean different things. So every stored embedding carries its model's name, and the search reads only rows from the model it is using. If the connector's model and the stored rows disagree, the tool says so plainly ("The policies were embedded with X, not Y. Tell the builder to run npm run embed.") instead of comparing numbers that mean different things. npm run embed embeds a company's passages again with the current model.

At work. In Lab 19, Task 14, OpenCode writes scripts/retrieval-eval.ts, which loads the question set, asks each question as its company, and prints a table and a pass or fail. Then it loads Cedar Ridge again with whole-document passages, runs the set, shows it failing, and loads it back.

The operating rule: write a question set with the expert, run it in the build with a pass mark, and store the model's name with every embedding. Then check a few answers with the expert too: the decision, the citation, and the claim the answer must not make.

Figure 19.28. The question set, run on every build. A change that makes retrieval worse fails it.