The State of AI Engineering: What a Thousand Companies' Telemetry Reveals

Five Moves for Leaders

1. Adopt a model gateway — centralize routing, failover, governance
2. Build deprecation discipline — retire models deliberately
3. Instrument agents deeply — especially with frameworks
4. Audit prompt caching — fix layout (stable first, dynamic later)
5. Implement budgets & backpressure — cap loops, build queues

Seven Key Takeaways

1. Multi-model is the norm (70%+ use 3+ models); use a gateway
2. LLM tech debt compounds; retire old models deliberately
3. Framework adoption doubled; observability burden doubled too
4. 69% of tokens are system prompts; only 28% use caching
5. Context windows exploded but quality beats volume
6. Rate limits are the #1 failure mode
7. Agents are still mostly monoliths; distributed shift is coming

Key Quotes

"The gap between a good demo and a dependable system is closed by effective evaluation and operational discipline." — Datadog

"The next wave of agent failures won't be about what agents can't do. It'll be about what teams can't observe." — Guillermo Rauch, CEO, Vercel

"Context quality, not volume, is the new limiting factor for LLM agents."

Hosted on Acast. See acast.com/privacy for more information.