Lesson 19.1519.15 Answers that cite, or abstain

In everyday life. Ask a careful lawyer a question her firm's contracts do not cover, and she says, "Our contracts don't address that. Let me check." She does not quote the nearest clause as if it answered you. And when she does answer, she tells you which clause, so you can read it yourself.

What you will learn. How a policy search cites what it found, says so when nothing is close enough, and lets the AP expert decide which documents count.

Why it matters. There is always a nearest passage, even for a question the policies do not cover. Without a lowest score it accepts, the tool hands the model the closest wrong rule.

One request, start to finish. Two questions reach search_policy, each from Cedar Ridge.

1. "Who approves an invoice over $3,000?" The nearest passage is approvals, line 5: "An invoice of $2,500.00 or more needs the general manager's approval before payment," with a high similarity. The tool returns it, cited by document, file and line, and the model answers: "The general manager (Invoice approvals, line 5)."
2. "What is the capital of France?" Every passage is far away, with a similarity near zero. The tool returns no passages, abstained: true, and a message: "No policy passage is close enough to this question. Say that the policies do not answer it, and ask the AP lead." The model says the policies do not cover it.

When should a search say it found nothing?

Retrieval always returns something: there is always a nearest passage. So a good retrieval tool needs a threshold, a lowest similarity it accepts. Below it, the tool abstains: it says it found nothing close enough, rather than offering the closest wrong passage. Above it, the tool returns each passage with a citation: the document's title, its file and its line, so a person can check it in one look.

Who decides which documents count?

Four decisions belong to the expert, not the builder, and the lab records each in docs/data-spec.md:

The first decision is enforced by the loader, not by the model. Each company's policy folder has a short list of its documents, with a column saying whether each one counts. The loader loads only those that do. The old email never reaches the database, so no search can ever find it.

Can a passage give the model instructions?

Passages are untrusted text. A policy passage is text written by people, read by the model as data, never as instructions to it (17.7). The tool's description says so, and the model must cite what the tool returned, not what it remembers. If the tool abstained, the model says so. It never fills the gap from its own knowledge.

At work. In Lab 19, Task 12, Claude Code interviews the builder, who plays Luis, about the four decisions. When it asks what to do with old-email-approvals-2024.md, the builder answers as Luis: "It doesn't count. Our rule is $2,500, in the approvals policy." Claude Code keeps it no in the list, and the loader's report confirms: not loaded (the AP lead said they do not count): old-email-approvals-2024.md.

The operating rule: return passages with citations a person can check, abstain below a threshold, and let the AP expert decide which documents count. Enforce that decision in the loader, where the model cannot override it.

Figure 19.26. Above the threshold, cite. Below it, abstain. A document that does not count is never loaded at all.