WakeUp-Jin

Design Philosophy of the ActSpace Evaluation Module

3 min

When building an Agent, we often start from intuition, or rather from experience.

We feel that adding this tool will be effective, that handling context this way will work, that structuring the prompt like this should be fine.

This approach, for an experienced LLM application engineer, will produce at least a passable Agent.

But what about the next step? If I want to make this Agent better, we fall into a state of confusion. We might look at other building approaches and experiences, but other people’s experience built in different contexts may not apply to your current situation.

The main problem is: we don’t know where the current Agent falls short — we can’t quantify the details of Agent execution.

Relying on intuition and experience can get you started, but to go further, we need a rationale for optimization.

We can turn our attention to Agent evaluation: build our own evaluation datasets from existing data, test with public datasets, observe issues in execution chains, and evaluate context quality based on actual execution environments.

Agent evaluation helps us determine the direction of Agent development, and provides powerful data so that each build decision can be more decisive.

Behind every great Agent, there must be a qualified Agent evaluation module.

I’m currently building ActSpace, a desktop Agent application. I’ve organized the design philosophy of its evaluation module here, hoping to provide some reference.

Research references:

1. ActSpace’s Agent Evaluation Module Design

Let’s start understanding this module from its inputs, which might be easier. The evaluation module I designed has three core input sources: behavioral evaluation datasets, internal datasets, and external public datasets.

ActSpace evaluation module design
ActSpace evaluation module design

For behavioral evaluation datasets, the main purpose is to evaluate the Agent’s execution chain and context quality:

  • Execution chain: whether necessary tools are called, whether the calling order is correct, how failures are handled, etc.
  • Context quality: whether tool results correctly enter the next round, whether task objectives are lost after compression, whether tool errors pollute the context, etc.

For internal evaluation datasets, the main purpose is to distinguish from public datasets and prevent overfitting during optimization. This primarily evaluates the Agent’s execution results. Worth discussing in detail is the method of building datasets: summarizing from failure cases and internalizing excellent datasets from peer Agents.

The evaluation flow is: the Agent develops features or fixes bugs in a codebase based on user input, and after code completion, test files are executed. If all test cases pass, the Agent’s task execution is considered successful.

So the core of the evaluator is: the codebase has complete test files and test execution commands.

For public evaluation datasets, we only run the Agent CLI and collect some information into prediction files, then the evaluator uses the official library’s built-in Harness framework.

The core evaluation approach: in the same codebase, at the corresponding commit branch, git apply is used to apply the generated diff code to the codebase, then the corresponding test commands are executed. If all test cases pass, the Agent’s modification is successful.

NOTE

This is similar to our internal dataset evaluation method, except the public evaluation dataset method is more complete, ensuring environmental consistency. The public evaluation dataset uses SWE-bench Lite.

// prediction.json file
{
  "instance_id": "django__django-11099",
  "model_name_or_path": "your-model-or-agent-name",
  "model_patch": "diff --git a/... b/...\n..."
}

The Agent CLI in the diagram is my packaging of ActSpace’s core Agent module into a CLI command for easy invocation. Meanwhile, the test set execution environment is uniformly inside Docker containers, ensuring local environment safety.

After CLI execution completes, there’s a post-run processor for data organization, which formats the Agent CLI output into various formats needed by the evaluation module.

Agent Evaluation ActSpace