The System Beats the Model

Episode Summary

Teams are finally staring at their AI bills — and the reflex is to use the LLM less. Dave and Dan make the opposite case: the answer isn't a cheaper model or less AI, it's a system. The harness around the model — how you plan, review, patch, and deploy — matters more than which model you run.

Key Topics

• The harness beats the model — a real system on an older/cheaper model out-executes ad-hoc prompting on the latest one.
• Why bills exploded — adoption without measurable speed gains, and the instinct to "just stop using LLMs" is the wrong lever.
• Deterministic pipelines — code → review → patch → deploy, ~14 steps each, so every change gets the same treatment every time.
• Model selection per task — only ~3% of tasks hit the expensive default (Opus); 70%+ run on a model 84% cheaper.
• Systems are optimizable, ad-hoc work isn't — you can't standardize or tune what everyone does differently.
• Enforcing org standards — one shared system means changing how everyone builds APIs or runs migrations is trivial.

Notable Quotes

• "The harness — basically how you use the LLM — is much more important than the actual model itself."
• "If you don't have a system, then you cannot enforce standards across your organization."

About The Velocity Lab

Dave O'Dell and Dan McAulay work inside engineering organizations every day helping them ship faster with AI. No hype, no BS — just what's working in the field.

Subscribe: RSS