20.12 Tokens, caching and what a call costs (advanced path)
- Approved by
- Panaversity ·
Where this fits. This is what steps 1 and 2 cost. Every call to the model costs money, counted in tokens. This concept shows how to read what a call cost, and how caching makes the repeated part of each request cheaper.
In everyday life. A copy shop charges per page. A regular customer prints the same 40-page manual, plus one new page, every day. The shop offers to keep the manual on its machine for a while and charge a small fee to reprint it, instead of the full price. The new page still costs full price. In this concept, the pages are tokens, and keeping the manual on the machine is prompt caching.
What you will learn. How to read what each call to the model costs, and how keeping the repeated start of each request makes later turns cheaper.
Why it matters. Every turn of the loop sends the same instructions and tools again. Paying full price for them every time adds up fast.
One request, start to finish. A five-turn run of one of Ana's longer questions sends the same instructions and the same four tool definitions on every turn, about 1,300 tokens, before the conversation even starts. (The numbers in this story are illustrative, not from a live run.)
- Without caching, each turn pays the full input price for those 1,300 tokens again, plus the growing conversation.
- With caching on, the first turn writes that repeated start to the provider's cache, which costs a little more than normal input.
- Turns 2 to 5 read it from the cache, and the conversation so far can be cached too. On turn 5, almost all of the input is read from the cache. Only about 200 tokens are new: the newest entries in the conversation.
- The cache read is priced at a fraction of normal input, so the five turns together cost noticeably less.
How do you read what a call costs?
A model reads and writes tokens, small pieces of text (often part of a word). Prices are per million tokens, with different prices for input and output. Every response reports its usage: how many tokens it read and wrote. Your loop adds them up for the cost cap (20.8).
What is prompt caching?
Prompt caching keeps the start of a request on the provider's side for a few minutes or more, so the next request that starts the same way reads it from the cache instead of paying full price. It fits the loop well, because every turn resends the same instructions, the same tools and the same earlier conversation.
| The first API | The second API | |
|---|---|---|
| How you turn it on | You ask for it: one setting on the request, or markers on up to four places in it | It caches long repeated starts automatically |
| How long it lasts | Five minutes by default, renewed when used, or an hour if you ask | At least 30 minutes by default, or 12 hours if you ask (see its caching page) |
| What usage shows | Tokens written to the cache, and tokens read from it, as separate counts | Cached tokens read, and tokens written to the cache, each as its own count inside the input total (F5) |
| What it costs | A write costs a little more than input, a read much less | A read costs less than input. A write is priced as F5 says |
The exact prices and minimum lengths for caching are on each provider's pricing and caching pages, and change, so the lab copies them on the day you run it (F5).
What keeps caching working?
Two rules keep caching working. Put what does not change first: the tools and the instructions, then the conversation. (The first API builds its cached start in the order tools, instructions, conversation.) And do not change the start between turns: a different instruction or tool list, even by one character, misses the cache. The stateless loop in this chapter already does both.
Figure 20.19. Cost per turn, with and without caching. The repeated start becomes cheap after the first turn. (To keep the picture simple, it does not show that the earlier conversation can be read from the cache too.)
At work. In Lab 20, Task 11, your coding agent runs the same question twice with --max-turns 5, once plain and once with --cache, and shows you each turn's usage side by side. You find the turn where cache_read_input_tokens first appears.
The operating rule: put what never changes at the start of every request, and turn caching on. Read the usage on every response, and count cached tokens at their own price.
Check yourself
Question 1 / 8
0 answered
Why does caching suit the agent loop so well?
- Owner
- Panaversity