Phase 3: Systems & Production (Days 8–10) — Scale evaluation to real projects and production constraints.
Understanding Core Concepts
1. The Demo-to-Prod Gap
A demo agent works great: you run it once in a notebook, it produces beautiful output, everyone's impressed.
A production agent runs hundreds of times on varying input, costs money per run, fails silently in ways you don't see until a user complains.
The jump from demo to production is where most agent projects fail.
2. Observability: You Can't Fix What You Can't See
In production, you need:
- Logs of what the agent attempted
- Input it received
- Output it produced
- Errors it hit (not just the final result)
- Latency and cost per run
Without this, you're blind. You'll optimize for the wrong things.
3. Cost and Latency Scaling
Evaluation might run 100 test cases. Production might run 1000 or 10,000.
A harness that costs $1 per run (many LLM calls) becomes $1000/day fast. You need:
- Cheaper evaluation methods (caching, smaller models)
- Fallback patterns (if expensive approach fails, try cheaper one)
- Budget awareness (know the cost per improvement)
4. Regression Testing
Once an agent works well, you don't want it to regress. This means:
- Keeping your test cases
- Running them regularly
- Catching performance drops before they hit production
- Having a rollback plan
This is the same discipline as production code. Same rigor needed.
Tiny Action for Today (20–25 minutes)
Design a minimal observability plan for an agent task:
- What would you log?
- How would you detect failure?
- How would you measure cost?
- What would trigger a rollback?
Write it as comments or a short checklist. This is your production checklist—reference it before shipping.