Phase 2: Core Techniques (Days 4–7) — Build the tools to measure and improve agent output systematically.
Understanding Core Concepts
1. What Is an Eval Harness?
An eval harness is a script that:
- Runs the agent on a set of test cases
- Scores each output against your criteria
- Reports the results
- You can run it again after changes to compare
It's the difference between "does this feel better?" and "it improved from 6/10 to 9/10 cases."
2. Test Cases Over Manual Review
Instead of reviewing each output by hand, you create a small set of representative test cases once.
Then every time you change instructions, context, or agent, you run the same test cases and get immediate feedback. The harness does the repetitive work.
This scales. A harness with 10 test cases saves you hours over time.
3. The Minimal Harness
You don't need fancy infrastructure. A simple harness has:
- Input cases (problem descriptions, code snippets, requirements)
- Expected outputs (or scoring rules)
- A loop: run agent, score output, record result
- A summary: how many passed, how many failed
That's it. You can build one in a few hours.
4. Iterating Against the Harness
Once you have it, the cycle becomes:
- Run the harness (baseline)
- Change instructions or context
- Run the harness again (compare)
- Keep changes that improve score, discard others
This is the real power: you iterate with data, not feelings.
Tiny Action for Today (30–45 minutes)
Pick a feature with 3–5 test cases you care about. Write them down:
- Input: what you're asking the agent to do
- Expected output: what success looks like
- Scoring rule: how to check it (deterministic)
Save this as a simple list or spreadsheet. Run the agent on each case manually and score. This is your harness skeleton.
Next, automate the scoring step. You've built the core.