9.3 Scheduled and event-triggered work

In everyday life. An alarm clock wakes you at a set time. A doorbell wakes you when someone comes, and it can ring twice for one visitor.

What you will learn. The two ways work can start without you, at a set time or when something happens, and five facts that every job nobody watches must be designed for. You practice it in steps 3 and 4 of the lab.

Why it matters. A run that nobody watches cannot ask you what you meant. Everything it needs has to be written down before it starts, and every way it can fail has to reach a person.

Work that runs without you starts in one of two ways. A scheduled run starts at a set time on the clock: every Thursday at 7 a.m. An event-triggered run starts when something happens: an email arrives, a file changes, a message is posted. Choose by what the work waits for. A weekly note before a payment run waits for a day of the week. A check for an invoice received twice waits for an invoice.

At Brightline, the book's example company, the AP Worker is the AI Worker that handles the bills Brightline owes. When Maria, the office manager, moved the Thursday review from her own chat to a schedule, four things changed.¹ The brief she used to type each week, the written instructions for the task, became a standing spec: written instructions the run reads every time. Her habit of checking each note by reading it became examples with known answers. These are past weeks whose right answers she knows, so she can test the spec. Before, she was there to answer questions. That became designed exits: points where the run stops and calls a named person. And the chat became a run in a runtime: the replaceable harness that reasons and acts for the worker. A harness is the software that runs a model in a loop with its tools. Here, the runtime also starts the job on its schedule.

Here are five facts about runs. Design for them on either AI vendor.

Assume each run starts fresh. Many runs begin with no memory of the last one. Some products can return a scheduled task to an earlier chat, but a chat is not a governed record. So anything a run must know about earlier runs has to be written in a place it reads. Concept 9.4 is about where.

A run sees only what it is given. An event-triggered run starts from the event: one email, one file. If the check needs other records, such as earlier invoices, the spec must tell the run to read them.

A trigger can fire twice. A trigger is what starts a run, such as an email arriving. One piece of work can start two runs. A sender sends the same thing again. A caller, the system that sent a request, tries again. Two events arrive close together. Design every run so that running it twice does no harm. This book's first edition has rules that always hold, called invariants. Its seventh invariant names the workforce's nervous system, and triggers are part of it.² Chapter 24 covers two more things: recovering a half-finished run after a crash, and limiting how many runs happen at once.

A run acts under someone's identity. The name on the work decides what it can reach and who is responsible for it. A run under one person's account can do anything that person can, and it may stop when that person leaves. Put team work under an identity the team controls, with only the access the job needs.³

Silence must mean success.³ In other words, a failure must never be silent. Every run should end with a result in a named place, and a failure should reach a named person. But a run that never starts says nothing. So something outside the run must check that the result arrived: a separate check, or a named person who looks at a set time.

A duplicate run does not have to mean a duplicate payment. DSoR, the Data System of Record, as specified, treats two identical requests as one when they have the same caller, the same action and the same idempotency key.⁴ The key names the intended action the same way every time, for example "pay invoice 5207 in the November 20 payment run." If a second run sends the same request, it gets the first result back. In that case, nothing is paid twice. This is what idempotency means: the same request, sent twice, has the effect of one. The key works only when the same request is sent again. A run on a later date makes a new key. So the invoice's paid status in the accounting system must still stop a second payment.

DSoR protects the action, not the run. It does not stop the second run, the second note to Dave, the controller, or the second log line.

To stop those duplicates, the run checks whether its output already exists before it creates it. That check works only if two runs never happen at the same time. Otherwise both can read "not done yet" before either one writes. So run such jobs one at a time, or use a shared record that accepts each intended action only once.

Figure 9.3. Clock or event: the two ways a run starts.