Lesson 19.1319.13 Retrieval from zero

In everyday life. Ask a good librarian for "something about getting the van fixed cheaply" and she brings you a book called Small Vehicle Maintenance on a Budget. None of your words is in its title. She matched what you meant, not what you said. A card catalog that matches only exact words would have found nothing.

What you will learn. How a search by meaning works, from zero, part by part, with every search kept inside one company.

Why it matters. A search by shared words cannot tell that "PO" and "purchase order" mean the same thing. A search by meaning can, and the answer can cite where it came from.

One request, start to finish. Luis asks: "Do we need a PO for a $400 repair?" Cedar Ridge's purchasing policy has the answer in two lines:

• Line 5: "A purchase order is required for any purchase above $500.00."
• Line 6: "Purchases of $500.00 or less, such as a small truck repair, may be paid without a purchase order."

Chapter 17's search_policy looked for lines that share words of three letters or more with the question. It would find line 6, which shares "repair". Line 5, the rule itself, would score no higher than any other line with "for" in it, because Luis says "PO" and the rule says "purchase order". Now:

1. The question is turned into numbers by a model made for this: a list of 1,024 numbers.
2. The database finds the policy lines whose numbers are nearest, among Cedar Ridge's lines only, under the same row-level security as every other table.
3. The two purchasing lines come back first, each with its document and line number, and the model answers from them, citing both.

What are the four parts of retrieval?

Retrieval has four parts. Each is simple on its own.

• Passages. A long document is split into short pieces, each stored as one row with the document it came from and its line numbers, so every answer can cite exactly where it came from. Splitting is called chunking. The lab makes one passage per rule, because Brightline's and Cedar Ridge's policies are short numbered rules, and a rule is what an answer cites.
• Embeddings. An embeddings model reads a piece of text and returns a list of numbers, its embedding, placed so that texts with similar meaning get similar numbers. "PO" and "purchase order" land near each other. "Repair" lands near "service visit". The length of the list is the model's number of dimensions: the lab's model makes 1,024. Another common model makes 1,536. The column's type must match the model: vector(1024). A list of numbers like this is called a vector. With two numbers you could draw it as an arrow on paper, from the corner to the point the two numbers name. Nobody can draw 1,024 directions, but the idea is the same: each embedding is an arrow, and texts with similar meaning have arrows that point in similar directions.
• Similarity search. The distance between two embeddings measures how different their meanings are. The lab uses cosine distance, written <=> in pgvector. It compares the directions of two arrows and ignores their lengths: 0 means "pointing the same way" and larger means further apart. The search orders the company's passages by distance to the question and takes the nearest few:

SELECT p.document_id, d.title, d.file, p.line_start, p.content,
       1 - (p.embedding <=> $2::vector) AS similarity
  FROM policy_passages p
  JOIN policy_documents d ON d.tenant_id = p.tenant_id AND d.document_id = p.document_id
 WHERE p.tenant_id = $1 AND p.embedding_model = $3
 ORDER BY p.embedding <=> $2::vector
 LIMIT $4;

In plain words: $1 is the company, $2 the question's embedding, $3 the model's name and $4 how many to take. Among this company's passages made by this model, order them by distance to the question's embedding ($2), nearest first, take the first few, and give each its document's title. 1 - distance turns the distance into a similarity score, where higher means closer. Two small things in the query: p and d are short names for the two tables, given in the FROM and JOIN lines, so p.content is the passage's text and d.title its document's title. And $2::vector tells the database to read the question's embedding, which arrives as text such as [0.12,-0.03,...], as a vector.

Figure 19.22. An embedding is an arrow. The search finds the passages pointing most nearly the same way as the question.

• A vector index. Ordering every row by distance is fast for a few hundred rows and slow for millions. A vector index finds near neighbors without measuring every row. pgvector's HNSW index (Hierarchical Navigable Small World, a layered map of which vectors are near which) is approximate: it is very fast, and it can, rarely, miss a close neighbor. The lab builds one:

CREATE INDEX policy_passages_embedding ON policy_passages USING hnsw (embedding vector_cosine_ops);

In plain words: build an HNSW index on the embedding column. vector_cosine_ops tells the index to measure by cosine distance, the same <=> the query uses. If the two did not match, the database could not use the index. At the lab's size the database does not use it yet (below). It is built now so that it is already there when a company grows, because building one later, on a table of millions of rows, is slow and easy to forget.

Why do the company and the vector share a row?

The tempting design, the one that looks most like 3NF (19.5), moves embeddings into their own table, passage_embeddings (passage_id, embedding), and joins it to the passages for the company. Both layouts were tried on the same database, with 10,000 passages for 20 companies, so 500 each, and the same vector index:

Both ran in a few milliseconds. The difference was not speed. It was the answer: the split layout returned fewer passages than asked, with no error, and could miss the one rule that answers the question. The rule: keep the column you filter by (the company) and the column you search by (the embedding) on the same row. That layout is what makes a correct search possible. It is not a promise on its own, so the question set (19.16) checks, on every build, that the right passages really come back. A join after the search is fine: the lab joins each of the five results to policy_documents for its title, five lookups by key. Why the index behaves this way, and the settings for very large companies, are in field guide F4.

Figure 19.23. Split the vector from the company and the search can come back short. Keep them on one row.

When does the database use the vector index?

At Brightline's size, about a dozen passages per company, the database reads the company's rows through its key and orders them all by distance: an exact search that does not use the vector index at all. A company with many thousands of passages will make the database use the index. Then turn on pgvector's iterative index scan, which keeps searching until enough of the company's rows pass the filter (field guide F4 shows the setting), and run the question set again to check.

What is a vector database?

A database that stores embeddings and finds the nearest ones quickly is often called a vector database. Some products are nothing else. This chapter does not add a separate one: pgvector, an add-on for Postgres, gives Postgres the vector column type, the distance operators and the vector index, so the passages sit in ordinary tables beside the invoices. One database, one set of rules, one backup.

Can a search reach another company's passages?

Retrieval is per company. The passages sit in the same database as the invoices, under the same policies, read in the same transaction. A Cedar Ridge question can only ever find Cedar Ridge's passages. The lab tests it: every passage in every Cedar Ridge answer comes from data-cedar/policies/.

At work. In Lab 19, the passages table and its index arrive with the other tables in Task 2, the loader splits and embeds each company's policies in Task 4, with a stand-in test model that needs no account, and in Task 13 Claude Code embeds them again with the real model. The builder asks Luis's question. The top two passages are lines 5 and 6 of Cedar Ridge's purchasing policy. Then Ana asks the same question. Her answer cites Brightline's purchase order matching policy, line 5: "Every invoice must match a purchase order (PO) before it is paid." Same words, different companies, different rules, each cited.

The operating rule: split documents into passages you can cite, embed passages and questions with the same model, search by distance within one company, and keep the company and the vector on the same row.

Figure 19.24. Retrieval by meaning. The nearest passages are the candidates the answer may cite.