Lesson 20.1020.10 Reading a trace

Where this fits. This is proof of which steps ran for Ana's question. After a run, how do you know which steps really happened? A trace records each model call and each tool call, so you can check.

In everyday life. An office building keeps a visitor sign-in book at the front desk. Each visitor writes one line: name, floor, time in, time out. When a manager asks "did the plumber ever come up to the fourth floor?", the book answers in a minute. But the book cannot say whether the plumber fixed the leak. Someone has to go and look at the pipe. In this concept, each line in the book is a span, the day's book is a trace, and looking at the pipe is checking the answer.

What you will learn. How to tell from a run's record which steps really happened, and what that record can never tell you.

Why it matters. An answer can look right after a lookup that never happened. A record of every step shows that in a minute.

One request, start to finish. Ana asks: "What does our policy say about an invoice with no purchase order? Cite the line." On a test branch, search_policy has been removed from the worker's tools by mistake. The worker's answer is "An invoice with no purchase order is held until a PO is raised (po-matching.md:5)." The rule is real, but it is line 9, not line 5. Where did line 5 come from?

1. The builder opens the run's trace, a file of spans, one line each.
2. It shows only chat spans. Each span is one line saying which step ran: chat for a model call, execute_tool for a tool call. Here there is one, chat and the model's id, then nothing else, and the run ended. No execute_tool search_policy span at all.
3. So the worker never searched the policy. It answered from what it had absorbed in training and from the instructions' example citation (a source note such as a file and line), which happens to be (po-matching.md:5). That example is where the line number came from.
4. The builder checks the request in the log: only three tools were sent. The fix is to restore the tool, and the rule is to check the tool list a run was given.

The trace did not say the answer was wrong. It showed that a step the answer depended on never ran.

What do you need from this concept?

Two span names and one label: chat for each model call, execute_tool for each tool call, and the call id that links a tool span to the model's request. In the visitor book, chat and execute_tool are the two kinds of visitor, and the call id is the badge number. The rest of this concept is labels you can look up.

What is a trace?

A trace is a record of one run, made of spans. Each span is one step, with a name, a start and end time, whether it succeeded, and labels called attributes. Spans in one trace share a trace id, so a tool for reading traces can draw them as a timeline. The open standard for this is OpenTelemetry, and its GenAI semantic conventions say how to name and label spans for model calls and tool calls. Those conventions are still at Development status (not yet final), so their names may change, and the lab says so where it uses them. They describe:

• A model call as a client span (a span for a call out to another service). It is named for the operation and the model, such as chat followed by the model's id. Its attributes include gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.response.finish_reasons (the trace's name for the stop reason).
• A tool call as an internal span (a span for work done inside your own program) named execute_tool and the tool's name, such as execute_tool list_open_invoices, with gen_ai.tool.name and gen_ai.tool.call.id, the same id the model gave the call. When the tool failed, the lab marks its span as an error, with error.type set to tool_error, so the trace shows which tool failed.
• No message content by default. Questions, answers and tool results are not recorded unless you choose to. Spans say what ran, not what was said.

Figure 20.16. One trace: a span per model call and per tool call. It shows what ran, not whether it was right.

What does a trace prove, and what does it not?

A trace proves which steps ran, in what order, how long each took, and which failed. It is the fastest way to answer "did the worker look that up, or not?" It does not prove that the answer was right: a run with every span green can still give a wrong answer. Chapter 19 checked answers against their citations. A trace tells you where to look.

At work. In Lab 20, Task 10, your coding agent runs Ana's policy question with --trace, once normally and once on a throwaway branch with search_policy removed from tools/ap-tools.json. You compare the two trace files, trace.jsonl,, and diagnose the second run from its trace alone before reading its answer.

The operating rule: write a span for every model call and every tool call, with the call id. Read the trace to see what ran, and check the answer separately.

Part C in one minute. A loop needs stops of its own: a turn limit, a cost cap and a time limit, checked before every call (20.8). Every error is sorted by whose fault it is, retried only when it is the service's, and recorded by its request id (20.9). And a trace shows what ran and what did not, though never whether the answer was right (20.10).

Now do Sitting 3.