Lesson 20.920.9 When the loop goes wrong

Where this fits. This is Ana's question when step 1, the call, fails. Any call to the model can fail. This concept sorts failures by whose fault they are, so you know which to fix and which to try again.

In everyday life. When a parcel does not arrive, the first question is whose fault it was. A wrong address is the sender's to fix, and sending it again changes nothing. A storm that closed the depot is no one's fault, and trying again tomorrow works. A parcel the recipient declined is not a failure at all. And every parcel has a tracking number, which is the first thing the courier asks for. In this concept, the wrong address is a request error, the storm is an overload, the parcel declined is the model declining, and the tracking number is the request id.

What you will learn. How to sort a failed call by whose fault it is, and what to do for each kind: fix it, try again later, or show it.

Why it matters. Sending your own mistake again only pays for the same failure twice, and the service's trouble often needs nothing but a short wait.

One request, start to finish. A builder copies an old sample into the loop. It worked last year. Now:

1. The request goes out with temperature set to 1. Temperature is a sampling setting (it shapes how the model picks each next word): it used to make answers more or less varied, and older samples often set it.
2. The response is an error, status 400, type "invalid request", with a message naming temperature, and a request id.
3. The builder's code does not retry it, because a 400 means the request itself is wrong. Sending it again would fail the same way, and cost a call each time.
4. The builder reads the message, removes the setting, and the call works.
5. The builder writes down why: the newest API version of the first API rejects temperature, top_p and top_k, even at their default values. The sample was written for an older version. An API version is a dated set of rules for the API, and the SDK sends it with every request (20.13 says more). Two similar names matter here:

With the exact SDK versions the lab uses, you usually meet this one step earlier: the SDKs no longer have a temperature option, so the type check (the step that checks names and settings before code runs, Chapters 13 and 14) stops it before anything is sent. The 400 comes from the API only for code that slips past the type check, such as an old untyped helper (an older piece of code that skips that check), or a request written out as JSON by hand. Task 9 sends it that way on purpose.

Whose fault is a failure?

Every call can fail, and the first question is always the same: is it my fault, the service's, or no fault at all? Each failure comes back with a status code, an error type, a message, and a request id, a unique label for that one request, which the provider asks for when you report a problem.

Which three status codes should you remember?

A 400 means fix your request. A 401 means check your key (your API key, the secret password your program sends with each request, 20.13). A 429 or a 500 and up means wait and try again, which the SDK already does for you. The table below is for looking up, not for learning by heart.

What are retries and backoff?

A retry is sending the same request again after a failure. Backoff is waiting longer before each retry: perhaps half a second, then one second, then two, with a little randomness so many clients do not all retry at once. Both SDKs do this for you, twice by default, for connection errors and for status codes 408, 409, 429 and 500 and above. So the loop itself does not retry. When an error reaches your loop, the SDK has already tried. The loop lets it through, unchanged, and the caller reads it. A test in the lab proves the loop passes a 400 straight through. (One more case, an error in the middle of a stream, waits for 20.11.)

Integration fault or model fault?

A model fault is the model doing the wrong thing with a request that was right: calling the wrong tool, misreading a result, writing a wrong answer. An integration fault is your code sending the wrong thing: a missing tool result, an edited thinking block, a setting the API no longer accepts. Integration faults show up as errors with a 400. Model faults show up as normal responses with wrong content, and you find them in the conversation log and the trace (20.10), not in an error.

Figure 20.15. From an error to its fix. Write down the request id every time.

At work. In Lab 20, Task 9, your coding agent runs npm run break -- temperature, then forced-tool, then bad-key. For each, you predict the status and whose fault it is, then read what came back: the status, the type, the message and the request id. You write each request id in ports/ch20.md.

The operating rule: sort every failure by whose fault it is before you act. Retry only the service's faults, with backoff, fix your own, and record every request id.