The AI Agent Factory

11.2 Ran is not the same as worked

Status
stable
8 min read
Owner
Panaversity
Approved
Panaversity ·

In everyday life. The dishwasher beeps "done." The plates are still dirty, because the soap door never opened. The beep proved that the cycle ran, not that the dishes are clean.

What you will learn. How to tell a run that worked from one that only finished, and how to make every recurring job say what it checked, every time. You practice it in steps 5 and 6 of the lab.

Why it matters. A job that speaks only when it finds a problem looks the same as a job that never ran. A well-organized result can also come from the wrong file.

In this chapter's story, the AP Worker of Brightline Wholesale Supply, the book's example company, ran two recurring jobs on its own. It is an AI Worker that handles the bills the company owes. In one week in January 2027, neither job failed loudly, with an error that reached anyone. On Tuesday, January 12, the duplicate check, which runs on each invoice email, checked none of the nine that arrived. It posts a message only when it finds a duplicate, so it produced nothing, but nothing was also what success looked like. On Thursday, January 14, the pre-run note, the summary the controller reads before he approves each Friday's payment run, said "No invoices due this week." It was a well-organized note, but the job had read the closed 2026 register. In fact, 31 invoices were due.

Run evidence shows that the mechanism ran: the job started, finished and produced an output. Outcome evidence shows that the work succeeded. That is, the right input was read, the result meets its Review Contract (Chapter 6), and it reached the right place. A Review Contract lists the checks you agree before the work. A scheduler shows run evidence.1 Only you, as the job's owner, can define outcome evidence.

A success signal is a visible sign that a run did its job.1 Give every recurring job a success signal. Three rules make a good signal:

  • Say something every time. "Checked 9 invoices, 1 possible duplicate" is a result. Silence means success only when a check outside the job shows it ran (Chapter 9). A job that never starts cannot report on itself. So for an event-triggered job, a line like that cannot come from the job itself. It comes from a separate scheduled check, like Chapter 9's deadline check. The separate check compares the invoice emails received with the emails checked, by message ID, the unique code each email carries. Counts are not enough: if one email was checked twice and another was missed, the counts still match.
  • State the input first. The Thursday note should have begun "Register: AP Register 2027, last entry January 13, 31 open invoices due." Then a wrong file would show on the first line.
  • Fail loudly when the input looks wrong. If the input is missing, out of date or far smaller than usual, stop and say so.

When an expected output is missing, do not change the brief, the job's written instructions, yet. The brief may not be the cause. First check for a pending approval or question, a pause, a usage limit, a broken connector or a different destination.1

The title reads "Ran is not the same as worked." The line below it reads "A completed run does not prove a correct result." A panel on the left, "Run evidence," asks "Did the mechanism run?" Its three rows each have a check mark: Started, Finished and Output produced. Its last line reads "The run status shows execution." A panel on the right, "Outcome evidence," asks "Did the work succeed?" Its three rows each have a check mark: Right input read, Meets the review contract and Reached the right destination. Its last line reads "The owner defines what must be verified." Between the panels, a not-equal sign carries the words "Verify separately." Below, under "Brightline: the Thursday note," a panel marked with a cross reads "Tidy output, wrong source" and quotes "No invoices due this week." Its note reads "The job read the closed 2026 register." A second panel, "A checkable success signal," lists the source, AP Register 2027, and the last entry, Jan 13, 2027. It also lists the due-by date, Jan 22, 2027, and 31 open invoices, USD 86,240.15. Its note reads "Check these details against the source." A banner reads "State the input. Verify the result. Confirm delivery." The last line reads "Silence can mean no run. Use an independent check for missing runs."

Figure 11.2. A job that only reports problems cannot tell you it never ran.

Check yourself

Question 1 / 8

0 answered

A job started, finished and saved a note. What has that proved?

  1. General Agents on the Web: A Crash Course, Panaversity, first edition.↑↑2↑3

On this page