Testing Is Phase Zero for AI Agents

Episode Summary

Dave and Dan dig into service readiness — the industry calls it "harness engineering" — and why testing just moved to the #1 spot of their 13 repo-readiness pillars. Before you let agents ship code in the background, the question isn't 90% coverage; it's whether anyone actually trusts the tests. They walk through the onboarding plugin that gets a repo test-ready and how test decisions get baked into the agent's planning phase instead of left to human best-effort.

Key Topics

• Why "no one has confidence in the tests" is universal — slow, flaky, and thin coverage everywhere
• Testing as pillar #1 of 13 for agentic-engineering readiness ("phase zero")
• The test-onboarding plugin: research versions → design the greenfield ideal → reconcile what to keep vs. throw out → a reviewed task list
• All four layers in scope: unit, integration, end-to-end, smoke — plus migrations, seeds, and integrations
• Baking "which tests does this need?" into the agent's planning phase rather than relying on a human to remember
• Coverage dogma vs. reality: test what matters, but keep 90% in CI as the practical proxy
• The daily cron that catches coverage gaps and staleness; local-first test execution for agents

Notable Quotes

• "Every organization we've pretty much ever worked for — no one has confidence in the test. The tests are slow, the tests are flaky, and they don't cover enough stuff."
• "How do you know that you can trust your agent to ship code that's gonna work?"
• "We said 90% coverage — great philosophy. Three months later we realized maybe the 10% is the one that matters."

About The Velocity Lab

Dave O'Dell and Dan McAulay work inside engineering organizations every day helping them ship faster with AI. No hype, no BS — just what's working in the field.

Subscribe: RSS