Phase 3: Systems & Production (Days 8–10) — Scale evaluation to real projects and production constraints.
Understanding Core Concepts
1. Output vs Process
Traditional code review looks at output: Is the code correct? Is it readable?
Agent evaluation needs both:
- Output quality: Does the code work? Is it maintainable?
- Process quality: How did the agent produce it? Did it reason well? Did it test assumptions?
An agent can produce good code by accident. A good process produces good code reliably.
2. What Good Process Looks Like
A strong agent:
- Reads existing code before writing new code
- Tests incrementally, not just at the end
- Asks clarifying questions when context is missing
- Reverts failed approaches instead of compounding on them
- Documents non-obvious decisions
These are process markers. You can score them separately from output.
3. Why Process Matters
If you only measure output, you miss the brittleness. An agent that produces working code by guessing will fail on variation. One that understood the problem will adapt.
Over time, process quality predicts output quality. A good process on Day 1 means fewer fires on Day 30.
4. Scoring Process
Process is harder to score automatically than output, but you can:
- Did it reference the existing codebase? (Yes/No)
- Did it test the output? (Yes/No)
- Did it handle errors explicitly? (Score 1-5)
- Is the reasoning documented? (Yes/No)
These are weaker signals than "does it compile?" but stronger than gut feel.
Tiny Action for Today (25–30 minutes)
Run the agent on a recent task and review not the output, but the reasoning:
- What files did it look at?
- What assumptions did it state?
- Where did it test?
- What did it skip?
Write down 3 process observations (good or bad). Next time you see those patterns, you'll know whether to encourage or correct them.