Phase 2: Core Techniques (Days 4–7) — Build the tools to measure and improve agent output systematically.
Understanding Core Concepts
1. Deterministic Checks
These are binary, objective tests. They cost almost nothing to run:
- Does the code compile?
- Do the existing unit tests pass?
- Does the output match the expected schema?
- Are all required fields present?
If your success criteria can be checked automatically, use deterministic checks first.
2. LLM-as-a-Judge
For qualities that can't be checked mechanically—code clarity, security best practices, maintainability—you can ask a strong LLM to evaluate the output.
Trade-off: More flexible than deterministic checks, but slower and less reliable. Use it only for what can't be measured objectively.
3. The Scoring Budget
Every evaluation has a cost: latency and model expense.
Start with the cheapest checks (deterministic), then layer in expensive ones only if needed. This is your evaluation budget—don't waste it scoring what you can verify for free.
4. Combining Signals
A good evaluation uses both:
- Deterministic: Did the agent follow the contract? (Pass/fail)
- Judge: Is it maintainable? Is it secure? (Partial credit)
Deterministic catches the binary mistakes. Judge catches the subtle ones.
Tiny Action for Today (15–20 minutes)
Take one feature the agent produced recently. List 5 things you'd want to verify:
- 2–3 that are objective (can be automated)
- 2–3 that are subjective (require judgment)
Separate them. The deterministic ones are your evaluation foundation. Start there, add the rest later.