Lesson 0.1.20.1.2 Why it behaves the way it does

In everyday life. A radio presenter reads the weather forecast every evening in the same calm, sure voice. On the evenings when the forecast is wrong, her voice sounds exactly the same. The voice is her style. It tells you nothing about the forecast.

What you will learn. What the model actually reads, and where your chat history lives. Why the AI can sound sure when it is wrong.

Why it matters. Each of these has a cause on this page: a wrong letter count, a new chat that forgets the last one, and a sure answer that is wrong. Once you know the cause, you can expect it instead of being surprised by it.

Tokens

What are tokens, and what do they measure?

The model does not read letters, or even whole words. It reads tokens, which are pieces of words. A short, common word may be one token, and a long word may be two or three. In English, four tokens make about three words.¹ The model reads and writes tokens, so tokens are what everything is counted in: the size of the context window, a plan's usage limits and the price a developer pays. Text in other scripts, such as Urdu, Arabic, Hindi or Chinese, is usually cut into more tokens per word, so the same message fills the window faster and costs more.¹ A typo rarely matters, because a misspelled word still becomes pieces close to its meaning. An image is cut into small pieces too, which is why small print in a picture is hard for the model to read.

The strawberry test shows a problem that tokens cause. Ask a model how many r's are in "strawberry". Some models miscount, then get it right once they spell the word out one letter at a time.¹ The model never saw the letters, only the tokens. Spelling the word out puts each letter into the window as its own piece. The test is famous, so a model may know this answer by heart. Try a word you invent as well. Nobody ever wrote down how many r's it has, so the model cannot give an answer it learned by heart.

So the model does not read letter by letter, the way you do.

Figure 0.1.4. The model sees tokens, not letters, and the window holds a limited number of tokens.

The context window

What is the context window, and what is a system prompt?

The context window is everything the app puts in front of the model for one reply. It is the only place the model sees your situation. It holds these things:

• your message
• the standing instructions, the rules set for every conversation, such as the system prompt, which is the part the app's maker writes and puts first
• any files attached for this reply
• notes from a memory feature, if one is on
• results from tools, such as a web search
• the conversation so far, or a summary of its older parts

The app builds the window again for every reply, from text it has stored. The model can use anything in the window. Anything outside it does not exist for this reply. The model is not refusing to look. There is nowhere else to look.

What does a "one-million-token context window" mean?

It means the window can hold about a million tokens at once. That is roughly 750,000 English words, or several long books.¹ It is big, but it is still a limit, and a long chat or a long file fills it. Chapter 8 shows what happens at the limit, and what to do.

Where is chat history kept, and how does it get into the window?

The app keeps each chat as text on the AI vendor's computers, in your account. That is why you can open the same chat later on another device, and why deleting a chat deletes something real.¹ When you continue a chat, the app puts the conversation so far back into the window before every reply. A new chat starts without it. That is why the Preface's two batches were never compared. Two new chats meant two windows, and each held one batch. Every reply reads the whole chat again, so a long chat also uses up a plan's usage limit faster.

Figure 0.1.5. Inside one chat, the app sends the whole conversation again before every reply.

Here is the overview's question, asked minutes apart. Figure 0.1.6 shows each window. At 0:00, a new chat holds only your question, so whatever comes back comes from training. At 0:05, in the same chat, you attach policy version 3, and the answer can now quote its line. At 0:10, a new chat holds only your question again. The policy is in the old chat's stored history, not in this window, and the model learned nothing at 0:05. At 0:15, you paste the old chat's answer in, and the chat "remembers" again.

Figure 0.1.6. A new chat is a new window. The stored history stays outside it.

So the model does not keep your conversation inside it. The app keeps the text and puts it back.

Confidence as style, and sycophancy

Why does the AI sound sure even when it is wrong, and why does it tend to agree with you?

Training did more than teach the model what usually comes next. In its last stage, people rated the model's answers, and the model was adjusted toward the answers they rated highly.¹ People tend to rate confident, fluent and agreeable answers highly. So training built a sure, tidy style into the model. It is one manner, used for every answer.

That is why a sure tone tells you nothing about where an answer came from. The model sounds the same when the fact was in the window and when it was not. Even "I cannot know Brightline's policy" is a trained manner. It is the more useful answer, but it is not a check.

Sycophancy is that manner leaning toward your own view. Your view is in the window, in the way you ask. If you ask "Isn't the approval limit $10,000?", you put $10,000 in front of the model, with the answer you hope for. Because of its trained lean toward agreeing, the model may then agree that the limit is $10,000.

So a sure tone does not mean the model knows. Chapter 6 shows how to tell a good answer from one that only looks good.

The jagged frontier

Why is the AI brilliant at one task and poor at an easier one next to it?

Because its skill comes from what it learned in training, and that skill is uneven. A person's skill is fairly even. Someone who can solve a hard math problem can almost always do easy math. The model may write a clear summary of a long policy, then miscount the letters in a short word. This uneven shape is called the jagged frontier. Jagged means uneven, with sharp ups and downs.

The model tends to be strong at tasks that were common in its training text and written out clearly. It tends to be weak at tasks it rarely saw, and at tasks that depend on what it cannot see, such as the letters inside a token. Each model has its own shape, and each new model is trained again, so the frontier moves. It has to be re-tested each time, not remembered from an older model.

Figure 0.1.7. The model's skill is uneven, and an easy task can be the one it fails.

So a model that is good at hard tasks still needs checking on easy ones.