AI Agent Testing: The Business Owner Defines Good

Public AI benchmarks cannot tell you which agent to trust on your workflows. The business owner should define good, test the outcome first, then every step, and when a build looks finished, force a visual workflow and a full process map.

Key takeaways. The business owner writes the tests; engineering should not invent success alone. Test the outcome first, then each step. When it appears to work, force the guide, visual workflow, and process map. Check new models, including Gemini 4 Carbon, on your own cases. Rerun tests after every update. Fund checking before more output.

Monday checklist (as spoken). You do not need to write this down while you listen.

• Outcome tests owned by the business. The owner writes what good and bad finished results look like. Why: they know what good looks like, not only what is easy to automate.
• Step tests across variations. Change the inputs. Require intermediate values and sources. Score logic breaks between steps. Why: a good last message can hide a broken middle step.
• Decline and escalate cases. Include the cases where the right answer is no, not enough information, or send it to a person. Score an unjustified action separately from unfinished work. Finished is not the same as right. That is the light version of this week's Visa lesson, without importing a consumer shopping score into your shop floor. Why: completion is not correctness.
• The same case, repeated. Run an identical case several times. Consistency is a separate score from a single lucky pass. Why: one pass is not consistency.
• Compare against past human-handled cases. Take closed work a competent person already decided. Blind the agent. Compare outcomes and the follow-ups required. Why: your closed work is local ground truth.
• Finished-product map, then rerun after every vendor or model update. Require the guide, the visual workflow, and the full process map after it appears to work. Then treat every platform change as a release candidate and rerun the suite. We learned that the hard way when a tool update broke multi-step workflows my team relied on. Memory kept outside the tool let us move. The test suite is how you notice the change in minutes, not after customers do. Why: leftover paths and vendor updates show up here.

Examples named. Invoice, claim, and customer-change cases (types, not industry claims). Leftover data writes or folder references. Gemini 4 Carbon's benchmark claims. Elvis Saravia and Lauren (9 Nov). Meta AutoRefine. DoorDash's Ask DoorDash harness (hidden tool path only).

Terms. Public benchmark: a published model test, not your workflow. Intermediate values: mid-process numbers, materials, and sources. Blind the agent: hide the prior human decision. Grader: your scoring method; AutoRefine found older graders go stale.

Sources: Elvis Saravia, 9 Nov; Lauren, 9 Nov; Google, Gemini 4 Carbon; Meta, AutoRefine; DoorDash evaluation; Visa benchmark release; Visa whitepaper.

The AI Brief from the YPO Technology Network is a daily executive briefing on the AI developments that matter to business leaders. Hosted by Stephen Forte.