Agent Evaluation: Methods and Frameworks

Preface

• Claude Cookbooks: https://github.com/anthropics/claude-cookbooks
• LangFuse documentation: https://langfuse.com/docs/evaluation/overview
• promptfoo framework: https://github.com/promptfoo/promptfoo

1. Why Agent Evaluation Matters

LLM output has uncontrollable factors. For production-grade LLM applications, stability is paramount. Mature evaluation frameworks not only make applications more stable but also reveal model potential and boundaries for better iteration.

Key quotes:

1. Teams’ inability to effectively evaluate model performance is the biggest barrier to LLM production use cases, turning prompt design into art rather than science.
2. Although evaluation takes significant time, doing it upfront saves developers time long-term and enables better products to ship faster.

Evaluation benefits for LLM application development — quantifying model boundary capabilities:

1. Iterative prompt optimization: Is our V2 prompt better than V1?
2. Pre/post-deployment quality assurance: Did our latest prompt update cause performance degradation?
3. Objective model comparison: When switching to a more advanced model, can we maintain or improve evaluation performance?
4. Potential cost savings: When switching to a faster, cheaper model, can we maintain evaluation performance?

2. Evaluation Components

A well-designed evaluation framework has four main components:

1. Example inputs: Instructions or questions for the model — key is designing inputs that accurately represent real-world usage scenarios
2. Gold standard answers: Correct or ideal responses as baselines — creating high-quality standards often requires domain expert involvement
3. Model output: What the LLM actually generates based on example inputs
4. Score: A quantitative or qualitative value representing model performance on that specific input

At least 100+ sets of example inputs and gold standard answers are needed for meaningful evaluation.

3. Evaluation Workflow

Step-by-step:

1. Prepare test cases (example inputs + gold standard answers)
2. Split into two batches: 80% development set, 20% holdout set
3. Design first-version prompt based on intuition
4. Test on development set
5. If results are poor, optimize prompt based on test results — iterate until satisfactory
6. Test against the holdout set to verify generalization
7. If the gap between holdout and development results is within ~10%, it passes
8. If the gap exceeds 10%, the prompt is overfitting to the development set — return to prompt optimization

Two additional evaluation factors beyond accuracy:

• Edge case coverage: Model performance on extreme inputs
• Performance testing: Model response time

The two most important aspects:

1. Writing evaluation questions and gold standards: Time-consuming but one-time cost, reusable
2. Ongoing scoring costs: Frequent evaluation runs incur model costs if using LLM scoring — build fast and economical evaluation systems

4. Evaluation Methods

Three common methods:

• Code-based scoring: Standard code to match and judge model output
• Human scoring: Manual review and scoring
• Model-based scoring: Another LLM evaluates the output

Prioritize model and code-based scoring over human scoring, which is more expensive and slower.

4.1 Code-Based Scoring

Characteristics: Programmatic approach for tasks with clear, objective criteria.

Advantages: Speed and scalability — can consistently process thousands of evaluations. Limited ability to handle nuance or subjectivity.

Forms:

1. Exact string matching: Output must exactly match gold standard
2. Keyword presence checking: Whether output contains certain key words/phrases
3. Regular expressions: Check complex text patterns

Example

Using the relatively weak Qwen2-7B-Instruct model to demonstrate the full evaluation flow with a sentiment analysis task:

Step 1 — Prepare evaluation dataset:

let testCases = [
  { id: 1, text: 'Amazing! Very satisfied, five stars!', expected: 'positive', reason: 'Clear positive words' },
  { id: 2, text: 'Fast shipping, quality exceeds expectations', expected: 'positive', reason: 'Multiple positive descriptions' },
  { id: 3, text: 'Garbage product, completely unusable, got refund', expected: 'negative', reason: 'Clear negative words' },
  // ... more test cases
];

Step 2 — First-version prompt:

let promptV1 = (text: string) => `
Determine the sentiment of the following text, answer "positive", "negative", or "neutral".
Text: ${text}
Answer with one word only.
`;

Step 3 — Run and score: Accuracy 83.33% (2 failures)

Step 4 — Optimize prompt with rules for sarcasm detection and transition sentence handling

Step 5 — Run optimized prompt: Accuracy 100%

Step 6 — Proceed to holdout set testing or pass evaluation

4.2 Human-Based Scoring

Characteristics: Gold standard for tasks requiring nuanced understanding or subjective judgment.

Advantages: Excels at evaluating tone, creativity, complex reasoning, and factuality. Disadvantages: Time-consuming, potentially expensive at scale, and susceptible to inter-rater inconsistency.

Forms:

1. Expert review: Domain experts evaluate accuracy and depth
2. User experience panels: Groups evaluate clarity, helpfulness, and engagement

4.3 Model-Based Scoring

Characteristics: Falls between code-based and human-based methods, using another LLM to evaluate output.

Advantages: Handles more complex and subjective evaluations than code-based scoring while being faster and more scalable than human scoring. Requires strong prompt engineering for reliable results, with risk of the scoring LLM introducing its own biases.

Writing Evaluation Model Prompts

Core elements:

• Original prompt or question
• Model output to evaluate
• Evaluation criteria or guidelines
• Instructions for how to evaluate and score

Common evaluation criteria:

1. How apologetic is this response?
2. Is the response factually accurate given the context?
3. Does the response excessively reference its own context?
4. Does it truly answer the question appropriately?
5. How well does it align with our tone/brand/style guidelines?

Evaluation Model Positioning

A qualified evaluation model must maintain objectivity. Most models default to being friendly with apologetic tendencies.

Add perspective constraints to the evaluation prompt: “Do not apologize or use apologetic language. Be objective and neutral.”

Evaluation Example

Step 1 — User input: “Write a slogan for a children’s toy store”

Step 2 — Tested model output: “Our store offers high-quality toys, welcome to purchase.”

Step 3 — Evaluation LLM scores against criteria: child-friendly, lively tone, total score 10

Step 4 — Result: 4/10. Reason: Tone is too formal and serious, doesn’t match the lively positioning of a children’s toy store.