Skip to content
The AI Agent Factory

0.1.1 The machine

Approved by
Panaversity ·
16 min read

In everyday life. A friend starts to sing "Happy birthday to", and you know the next word is "you". You did not look it up. You continued a pattern you learned long ago. Now try to finish a line from a song you have never heard. You still think of a word that fits. It sounds just as right, but it is a guess.

What you will learn. Where the AI's answers come from, and why the same question can get a different answer. Why it seems to remember you, and why it makes things up.

Why it matters. The same tidy answer can come from a fact or from a guess. When you know what the AI works from, you know which one to suspect and what to look at.

Prediction, not lookup

Every AI chat has a model inside it. The model is the trained engine that turns the text in front of it into a reply. A large language model (LLM) is the kind of model inside today's AI chats. When you send a message, the model does one thing. It continues the text in front of it with the most likely next pieces of text, one small piece at a time.

The model works from exactly two things. The first is what it learned in training, the long process that made it before it was released. What it learned is stored inside it as a huge set of numbers, called its weights. Here a weight is simply a number the model learned, not a weight in kilos. The second is the context window, everything the app puts in front of it for this one reply. Nothing else exists for the model.

Does the AI look facts up?

No. It is natural to imagine a librarian who finds the fact in a huge book and reads it back, but that picture is wrong. A lookup points to a source. A prediction has no source to point to. Ask for the capital of France, and "Paris" comes back because those words appeared together very often in the text the model learned from. Ask about a small business it never read about, and a reply still comes back, the one that sounds most likely. When neither of its two sources holds the fact, the answer is a guess. So the more often a fact appeared in its training text, the more reliable the prediction. A rare topic gets a guess in the same fluent words.

Some apps also search the web, and a search does look things up. The model still only predicts, now from what the search put in its window, as 0.1.3 shows.

Why does the same question get a different answer each time?

The model does not pick one fixed next piece. It scores many likely pieces, then picks one of them, with some chance in each pick.1 So the same question can come back in different words, or even with a different answer. A different answer does not mean the model changed its mind.

Watch it write, one piece at a time

The model continues your text with a likely next piece. Press the button and watch it pick. Then pick again.

Happy birthday to ▍

  1. you96%
  2. me2%
  3. your1%
  4. everyone1%

Pick a few times and watch where the picks land.

The chances here are made up to show the idea. A real model scores many thousands of pieces at every step.

It continues, it never looks up: the model predicts one piece at a time. Four boxes run down the page. The first holds "Your message + everything in the context window". An arrow leads to "Frozen weights predict the next piece", then to "One piece is chosen and added to the answer", then to "That piece is fed back in with everything else". An arrow labeled "repeat, piece by piece" runs from the last box back up to the second. A banner reads: "No database is ever opened. It predicts a likely next piece, then does it again."

Figure 0.1.1. The model writes its answer one piece at a time, and each piece is picked from the likely ones.

Trained once, then frozen

How was the model made, and why does it answer you instead of just continuing your text?

Before release, the AI vendor, the company that makes the model, trained it in three stages.1 First, the model read a huge amount of text, such as public web pages, books and code. Again and again, it guessed the next piece of a passage, and it was adjusted to guess a little better. Next, it learned from examples of instructions paired with good answers, so the most likely text after a question became an answer. Last, people rated its answers, which shaped its manner, as 0.1.2 shows.

When training ends, its weights are frozen, which means they never change again. The date when its training text stops is the model's knowledge cutoff. Nothing that happened after that date is in what it learned.

Why can't it learn from you as you talk?

Because the weights are frozen, and that is a choice, not a fault.1 Training takes months of computing and costs a great deal, while answering you is fast and cheap. A frozen model can be tested once, and then it behaves the same way for millions of people. A model that changed with every chat could drift, and people could push it off course on purpose. So the model keeps nothing of its own from one reply to the next.

If you correct it, it may reply "You're right." But it has only predicted the words that usually follow a correction. Inside this chat the correction helps, because it is now in the context window. In a new chat, the model starts again from the same frozen weights. A separate setting decides whether the AI vendor may use your chats to train a later model, and Chapter 0.2 shows where it is.

Brightline's AP policy is the company's own document, so it was never in any training text. With nothing about Brightline in the window, whatever the model says about Brightline's approval limit comes from what it learned about other companies.

Then why did yesterday's chat seem to remember you?

The app did the remembering, not the model. The app remembers in two ways.

  • Continuing a chat. The app stores each chat. When you continue one, it puts the conversation so far back into the context window before every reply.
  • A memory feature. Some apps save short notes about you from your chats. If the feature is on, the app puts those notes into the window of a new chat.

In both cases, the model learned nothing. Open a new chat with memory off, and the "memory" is gone. So keep two things apart: frozen weights, which no chat changes, and saved notes, which are text that the app puts back into a window.

What the model works from: two sources only, and nothing else exists for it. An arrow runs into "The model" from "What it learned in training", which is fixed at the knowledge cutoff. Another arrow runs in from "The context window", which is built for this one reply. The window holds six tiles: your message, standing instructions, files you attached, saved memory notes, tool results, and the conversation so far. One arrow runs out, to "The next pieces of text". A dashed arrow from "Your chats" toward training is crossed out with a red X and labeled "does not change what it learned". A banner reads: "When it seems to remember you, the app put your old text back in."

Figure 0.1.2. The model's two sources. Nothing you type goes back into what it learned.

No truth-checker

What is a hallucination, and why does it happen?

A hallucination is invented or false content presented as fact. It happens because nothing inside the model checks its answers. A careful person has a second voice in their head that asks, "Am I sure? Where did I learn that?" The model has no part like that.1 Nothing inside it compares a prediction with what it learned, with the context window or with the world. So when neither source holds the fact, the model fills the gap in the only way it can, by predicting. The guess comes out in the same tidy sentences as a fact.

Hallucination is a normal result of the model's design, not a fault in one app. One process continues "The capital of France is" with "Paris". The same process continues a question about Brightline with a limit that sounds right.

To see it, open a new chat, ask about Brightline's approval limit, and then ask where the answer came from. If the reply names a Brightline document, that document was never in front of the model, so the citation is invented.

The model does not check its own answers. Checking is your job. You compare the answer with the record, the document that really holds the fact. For Brightline's limit, the record is the policy itself. Chapter 6 teaches how to check.

A guess and a fact look the same: nothing inside the model checks the answer. Two answer cards are drawn the same way. The left card is labeled "from training: a guess" and says "Invoices over $10,000 need approval." The right card is labeled "from the context window: the policy" and says "Invoices over $5,000 need the controller's approval." A tag between them reads "nothing inside checks". A document panel below both cards shows AP policy version 3, approved September 1. It reads "over $5,000: the controller's approval", and a "compare" arrow runs from it to each card. Beside a magnifying glass, a label reads "you check the answer against the record". A banner reads: "A sure tone is a style, not a source."

Figure 0.1.3. A guess and a fact come out in the same form. The difference is outside the model.

Check yourself

Question 1 / 9

0 answered

When you send a message, what does the model do?

Sources

  1. What AI Actually Is: A Crash Course, Panaversity, first edition.↑↑2↑3↑4
Owner
Panaversity

On this page