Lesson 20.720.7 What a stateful mode hides

Where this fits. This is 20.5's array, kept by someone else. Some APIs offer to keep the conversation for you, so your program sends only what is new. This concept shows what that saves, and what you can no longer see.

In everyday life. Two clinics treat the same patient. The first hands the visiting doctor the whole paper folder. The second keeps the folder in a back room, and the doctor gets a number to quote: "continue from visit 42". The second is lighter to carry. But when the doctor wants to know exactly what the last doctor wrote, the answer is in a room she cannot enter, and the clinic charges for every page read aloud. In this concept, the paper folder is your conversation array, and the number is a previous-response id.

What you will learn. What changes when the provider keeps the history for you, what that saves, and what you can no longer check.

Why it matters. A convenience that hides what the model saw also hides the cause of a bad answer. Knowing what it keeps lets you choose it on purpose.

One request, start to finish. Ana's question runs again on the second API, this time stateful:

1. Your code sends the first request with storage on. The provider stores the response and gives it an id.
2. The model asks for a tool, as before. Your code runs it.
3. The second request sends only two new things: the tool's output, in its input list, and the id of the previous response, in a field of its own. No question, no reasoning, no function call. (The instructions and tools still go with it, as on every request.) The provider joins them to the history it kept. It still bills that stored history as input on every call, like the clinic's charge for each page read aloud.
4. The model answers in words.
5. Your code prints what it sent. Call 1: the question. Call 2: one function call output and an id. The model's function call and its reasoning are nowhere in your program's record. They live under that id, on the provider's side.

The run worked. But your own log can no longer show what the model saw.

What does a stateful API keep for you?

A stateful API keeps the conversation on the provider's side between calls, and each new request names the earlier response instead of resending it. The second API offers this with a previous-response id, and also as a longer-lived conversation object (a saved conversation the provider keeps for you, with its own id). The first API has a stateful option too, in beta (still being tested) at the time of writing: message threads, which keep a thread for a few minutes. Both save you from resending the history. Here is what that trade costs:

Why does this chapter teach the stateful mode last?

A stateful mode is not wrong. It is convenient, and for a chat app it can be the right choice. But it hides the very thing this chapter teaches: what the model sees on each call. Learn the loop with the whole array in your hands. Then, when you choose a stateful mode, or an SDK that uses one (Chapter 21), you know exactly what it is keeping for you, and what you can no longer inspect.

Figure 20.12. Your array against the provider's hidden one.

At work. In Lab 20, Task 7, your coding agent runs npm run stateful with Ana's question. At the end it prints everything your program sent, call by call, beside the full input list from Task 6. You write down, in ports/ch20.md, three things the stateless run showed that the stateful one does not.

The operating rule: keep the conversation yourself unless you have a reason not to. When you choose a stateful mode, write down what it keeps, where, for how long, and what you can no longer see.

Part B in one minute. The second API runs the same loop with different shapes: items instead of blocks, a call id, arguments as a string you parse, reasoning items you send back, and "no calls" for done (20.6). With storage off, the array is still yours. A stateful mode keeps the history on the provider's side, which saves sending but not paying, and hides what the model saw (20.7). Now do Sitting 2.