Lesson 14.714.7 Async and await

In everyday life. You have two errands. You drop a suit at the dry cleaner, which takes them ten minutes to label. You pick up a prescription, which takes the pharmacy ten minutes to prepare. You can stand at the cleaner for ten minutes, then at the pharmacy for ten. Or you can hand in both, take a ticket at each counter, and come back when both are ready. The waits overlap, so the total is the longest wait, not the two waits added up.

What you will learn. When slow calls can run at the same time, which failures are worth another try, and why every call needs a time limit.

Why it matters. Most of a program's time goes on waiting. Ask at the same time for answers that do not depend on each other, try again only when another try can work, and set a limit on every wait. A call that hangs, which means it waits forever, stops the whole run when nobody is watching.

Why is most slow work waiting?

Because the slow part is usually somewhere else. The program sits idle while a server answers, a file arrives or a database replies.

• Async code lets a program start a slow task, do other work while it waits, and pick the result up when it is ready.
• A promise in TypeScript is like the ticket from the counter: it stands for a result that is not ready yet. This is a new meaning of the word, not the promise a signature makes in 13.5. Python's version is a coroutine, which differs in one way, as What if I forget await? explains.
• await means "pause this function here until the result is ready". Work already started keeps running meanwhile, but the next line waits.

So two awaits in a row wait one after the other. To overlap two calls, start both, then wait for both together.

Reading an answer in pieces. Some APIs send a long answer in pieces as it is produced. This is called a stream. A model's answer often arrives this way, word by word. Async code reads a stream with a loop that waits for each piece: for await in TypeScript, async for in Python. Concept 14.8 shows one.

When can two calls run at the same time?

Only when neither needs the other's answer. Fetching vendors and fetching open invoices are independent. Page 1 has to come first, because it says how many pages there are. After that, pages 2 and 3 could be fetched at once too. Brightline's client fetches its pages one after the other, which is simpler, and runs the vendors call at the same time. With more pages, fetching a few at a time is a good next step.

At work. Ana's API client fetches the vendors and every page of open invoices. The sample API waits 300 milliseconds per answer, and answers the first vendors call with 503. One after the other, the client would take about 1.5 seconds: two vendor calls and three invoice pages. With vendors and invoices fetched at the same time, it takes about one second, set by the three invoice pages. A test she decided on checks that the vendors and invoices requests reach the server at about the same time, so a later change that stops them running together fails the test.

In TypeScript. wait stands in for a slow call. wait(300, "vendors") waits 300 milliseconds, then gives back "vendors". performance.now() reads a clock in milliseconds. Predict both times.

const wait = (ms: number, label: string) => new Promise<string>((resolve) => setTimeout(() => resolve(label), ms));

let start = performance.now();
const first = await wait(300, "vendors");
const second = await wait(300, "invoices");
console.log("one after the other", Math.round(performance.now() - start), first, second);

start = performance.now();
const both = await Promise.all([wait(300, "vendors"), wait(300, "invoices")]);
console.log("at the same time", Math.round(performance.now() - start), both);

one after the other 602 vendors invoices
at the same time 300 [ 'vendors', 'invoices' ]

In Python. asyncio.sleep stands in for the slow call. time.perf_counter() reads a clock in seconds, so the code multiplies by 1,000 to print milliseconds.

import asyncio
import time

async def wait(seconds: float, label: str) -> str:
    await asyncio.sleep(seconds)
    return label

async def main() -> None:
    start = time.perf_counter()
    first = await wait(0.3, "vendors")
    second = await wait(0.3, "invoices")
    print("one after the other", round((time.perf_counter() - start) * 1000), first, second)
    start = time.perf_counter()
    both = await asyncio.gather(wait(0.3, "vendors"), wait(0.3, "invoices"))
    print("at the same time", round((time.perf_counter() - start) * 1000), both)

asyncio.run(main())

one after the other 601 vendors invoices
at the same time 301 ['vendors', 'invoices']

Your times will differ by a few milliseconds. The ratio will not.

Which errors are worth a retry, and how often?

Only those that can succeed next time, and only a few times. A network call can fail for a moment or hang. The edge, where answers come in, needs rules for time too, for how long to wait and when to ask again. A 503 or a 429 may succeed on the next try. A 401 or 400 will fail again. If the pharmacy says "come back in ten minutes", coming back works. If it says "no prescription in your name", it will not. Brightline's client retries once on 503 and never on anything else. A client that retries forever sends request after request while a server is down.

A wait timeout or a deadline?

Every call needs a deadline, a timeout on the whole call. A call with no timeout can wait forever, and the program waits with it. There are two kinds of time limit. A wait timeout limits each wait inside a call: connecting, or waiting for the next piece of the answer. The Python library httpx sets these to five seconds by default. A deadline limits the whole call, however the time is spent. A wait timeout is leaving a queue that has not moved for five minutes. A deadline is leaving the shop at 5:30, whatever the queue does. A server that sends a small piece every four seconds can keep a call going for an hour under a five-second wait timeout. Only a deadline stops it. AbortSignal.timeout on a fetch and asyncio.timeout in Python are deadlines. Brightline's client gives every call a two-second deadline, in both languages.

What if I forget await?

It depends on the language.

Where the languages differ. In TypeScript, the call starts as soon as you make it, and await only waits for it. A Python coroutine does nothing until it is awaited or handed to asyncio.gather. So in Python, forgetting await means the call never happens. Pyright reports it when nothing uses the result, and Python warns that the coroutine was never awaited. In TypeScript, forgetting await means the call happens but nobody waits for its result or its error. In the errand story, TypeScript has handed in the suit, and forgetting await is never going back for it. Python has only filled in the order, and nothing happens until await hands it over. Both are common in agent code. Look for every async call and check that it is awaited.

The operating rule: run slow calls at the same time only when they do not depend on each other. Retry only the errors that can succeed next time, and only a few times, and give every network call a deadline.

Figure 14.8. One after the other, or at the same time. In Brightline's client the pages run in turn, which is simpler. The vendors do not wait for them.

Practice it. Practice: async and await, in your own tool, fifteen minutes.