Analysis references:

• Hackathon champion Claude Code config collection: https://github.com/affaan-m/everything-claude-code
• Anthropic official Skill configuration: https://github.com/anthropics/skills
• ClaudeCode configuration docs: https://code.claude.com/docs/en/sub-agents

1. Cross-Session Shared Memory

When using ClaudeCode, session records can be saved locally and resumed with the resume command. However, complete session records get compressed, and after multiple compressions, critical decision information gets diluted until completely forgotten. The four critical types of information are:

1. Which approaches worked (with verifiable evidence)
2. Which attempted approaches failed
3. Which approaches haven't been tried yet
4. Which work remains unfinished

To ensure these four types of information can be shared across sessions, we need a separate intermediate temporary session file.

This requires a complete automated file creation/saving workflow plus content-filling instructions, using three hooks:

1. PreCompact Hook: Before context compression, save important state to file
2. SessionComplete Hook: At session end, persist learning outcomes or initialize file
3. SessionStart Hook: At new session startup, auto-load prior context and output latest file path

What content to fill and how?

• Content: Let Claude summarize session history based on the four information types above, or manually write key session information
• Method: Mention it in chat, or create corresponding Skills and Commands

Files used in this pattern:

• scripts/hooks/session-start.js
• scripts/hooks/pre-compact.js
• scripts/hooks/session-end.js

2. Continuous Learning and Memory Updates

Two trigger methods for continuous learning: Hook-mounted automatic scripts and Command manual execution.

Hook-based automatic approach uses three different-timed Hooks:

1. Stop Hook: Check session list length; if threshold met, output prompt info
2. Sessionend Hook: Read complete session history; call claude via claude -p "xxx" to generate learning records
3. PostToolUse Hook: Check session list length; inject "summarize learning record" instruction into tool return results

Command-based manual approach: the /learn command

When users complete tasks and find design patterns worth saving to memory, trigger /learn.

All summarized learning records are stored in /skill/learn folder, so the Agent can automatically use Skill learning records based on context.

Files used: commands/learn.md, skills/continuous-learning

The difference between Session Files and Learning Files:

• Learning Files (Learn Skill): Global, permanent, abstract knowledge rules — purpose: avoid repeating mistakes, accumulate experience
• Session Files (Session Tmp): Local, temporary, specific work state — purpose: cross-session continuity

3. Improving Project Maintainability — Evaluation + Dead Code Cleanup

3.1 Checkpoint-Based Evaluation

The flow is a three-step cycle: Start -> Implement -> Verify, constraining each feature.

1. Start: Before implementing, run the start command to ensure a clean workspace
2. Implement: Write feature code — manually or with AI
3. Verify: Run verification to evaluate code quality against requirements

The start command (/checkpoints create):

1. Execute /verify quick — only check build and type errors
2. Execute git stash or commit to save current state
3. Write SHA to checkpoints.log via git rev-parse --short HEAD

The verify command (/checkpoint verify):

1. Get the latest SHA from checkpoints.log
2. Model calls git diff and runs relevant tests
3. Output a report with key metrics: new files, modified files, test pass rate, coverage, etc.

This approach is elegant because the flow isn't forced automation — it only provides the minimum necessary condition: the SHA. How to get new/modified files, tests, and coverage metrics is determined by the model, maximizing model autonomy.

3.2 Continuous Evaluation

Flow:

1. Trigger: Run every N minutes or after major changes
2. Execute: Run full test suite, build status, code checks
3. Results: Output detailed inspection report
4. Decision: Determine if code passes; if not, proceed to fix
5. Fix: Repair failing checks, then conclude

Difference between the two evaluation approaches:

1. Checkpoint-based: Suited for linear workflows with clear milestones
2. Continuous: Suited for long-running sessions

The deciding factor is task nature — checkpoint evaluation for feature implementation with clear stages, continuous evaluation for exploratory refactoring or maintenance without clear endpoints.

3.3 Dead Code Cleanup

Uses a sub-Agent and a Command:

# Refactor Clean

Safely identify and remove dead code with test verification:

1. Run dead code analysis tools: knip, depcheck, ts-prune
2. Generate full report in .reports/dead-code-analysis.md
3. Categorize by severity: SAFE / CAUTION / DANGER
4. Only suggest safe deletions
5. Before each deletion: run tests -> confirm pass -> apply -> test again -> rollback if failed
6. Show cleanup summary

Never delete code before running tests!

3.4 Code Maps — Trusted Context

Code maps serve as an entry point for AI or developers to understand the codebase. AI can understand the project's global picture with fewer tokens through code maps. Keep documentation minimal and concise.

3.5 Summary

Four approaches to improving project maintainability:

• Two verification methods (checkpoint and continuous) are sufficient to avoid most technical debt with appropriate intervention.
• Continuously updating code maps helps, as it records changelogs and how the code map evolves over time — providing a reliable information source beyond the repository itself.
• Through strict rules, Claude avoids creating messy .md files, duplicate files for similar code, and doesn't leave large amounts of dead code.

4. Sub-Agent Usage: Loop Verification + Orchestration

Sub-agents exist to save context by returning summaries rather than full information. However, the orchestrator has semantic context that sub-agents lack.

From @PerceptualPeak: "Your boss sends you to a meeting and asks for a summary. Nine times out of ten, he'll have follow-up questions. Your summary won't contain all the information he needs because you lack his implicit context."

Two better approaches: loop verification calling pattern and sequential orchestrator.

4.1 Loop Verification Calling

Flow:

1. Main agent evaluates sub-agent results
2. If results are inadequate, main agent proposes new retrieval tasks
3. Sub-agent continues retrieval with new tasks
4. Maximum 3 rounds

In this pattern, tasks assigned to sub-agents should be "specific question + broader goal" to maximize retrieval coverage.

4.2 Orchestrator Agent

Principles:

1. Each agent receives clear input and produces clear output
2. Output becomes input for the next stage
3. Never skip stages — each contains value
4. Use /clear between agents to keep context fresh
5. Store intermediate outputs in files (not just memory)

Custom orchestration is also supported:

/orchestrate custom "architect,tdd-guide,code-reviewer" "Redesign caching layer"

5. Configuration File Reference

By core functionality:

1. Cross-session shared memory: session-start.js, pre-compact.js, session-end.js hooks
2. Continuous learning: commands/learn.md, skills/continuous-learning
3. Project maintainability: commands/checkpoint.md, commands/verify.md, skills/verification-loop, commands/refactor-clean.md, agents/refactor-cleaner.md, agents/doc-updater.md, commands/update-codemaps.md, commands/update-docs.md
4. Sub-agent usage: skills/iterative-retrieval, commands/orchestrate.md, agents/architect.md, agents/code-reviewer.md, agents/planner.md, agents/security-reviewer.md, agents/tdd-guide.md

• OpenAI's article: https://openai.com/zh-Hans-CN/index/harness-engineering/
• Anthropic's article: https://www.anthropic.com/engineering/harness-design-long-running-apps

1. OpenAI's Practical Experience

OpenAI's team attempted an experiment: build and ship an internal beta software product with no manually written code.

To accomplish this, the team needed to build a Harness for Codex that could run reliably over the long term. The software engineering team's primary work was no longer writing code, but designing environments, clarifying intent, and building feedback loops.

This enabled Codex to deliver a million-line project within weeks, already used by hundreds of internal users.

The core components of the coding Agent's Harness:

Step 1: Three-Layer Code Review

After Codex completes a task, it performs three layers of code review: self-review, local code review Agent, and cloud code review Agent. Only after all reviews pass can it proceed; otherwise, review results are injected back into Codex for revision.

Step 2: Human QA

As Codex's coding speed increased, the bottleneck became human QA. OpenAI integrated Chrome DevTools Protocol into Codex, enabling DOM snapshot processing, screenshots, and navigation — giving Codex direct UI analysis capability.

Step 3: Log Detection and Performance Optimization

The team also fed runtime logs and performance metrics into Codex, enabling practice -> observe -> modify cycles for performance optimization, rather than relying solely on code structure perception.

Step 4: Code Documentation Library

A codebase's detailed documentation is massive and can't be injected all at once. Using the "progressive disclosure" concept from the Skill specification, the documentation is delivered to Codex in a directory-file format.

OpenAI cleverly used AGENTS.md as the documentation index, containing file paths and brief descriptions. Whether to read and what to read is entirely up to Codex, making context utilization highly efficient.

A critical detail: feature requirements often go through team discussions. If this "discussion information" isn't documented for Codex, this context is missing from the Agent's runtime environment, potentially causing long-term directional drift.

Step 5: Codebase Structural Rules

Structural rules constrain the codebase to prevent chaos over time. For example, when Codex adds a feature, where to start and which structural layer to consider follows an ordering rule:

Types -> Config -> Repository -> Service -> Runtime -> UI

Validation relies on custom code checkers (also written by Codex).

Summary: The team applied mature software engineering practices to building Agent runtime environments:

Software development still requires rigorous discipline, but that rigor is increasingly about scaffolding rather than the code itself.

Key takeaways:

1. Provide execution feedback for every step, creating feedback loops
2. Find specific constraints from general patterns for targeted scenarios
3. Provide more effective context — currently, "progressive document loading" is the best practice

2. Anthropic's Practical Experience

When building frameworks for long-running coding agents, Anthropic used a simple two-layer multi-agent architecture: task initialization Agent + coding Agent. As runtime increased and tasks grew complex, two common failure modes emerged:

1. As the context window fills, the model loses coherence, with some models exhibiting "context anxiety" (especially Sonnet 4.5)
2. When designing self-evaluation modules, Agents asked to evaluate their own work tend to give confident, high praise

For problem 1, the solution was context reset: completely clearing context (not just compression), starting a new Agent with structured handoff mechanisms.

For problem 2: separate the evaluation Agent from the execution Agent.

The team then built a three-Agent system:

1. Planner: Takes 1-4 sentence prompts and expands them into full product specs
2. Generator: Works in loops, executing one sub-task at a time
3. Evaluator: Uses Playwright MCP to simulate user operations, testing UI, API endpoints, and database state, scoring against criteria

Two core practices:

• Before each sub-task, the generator and evaluator negotiate a development contract — agreeing on completion criteria before writing any code
• Agents communicate via files — one Agent writes to a file, another reads it

For evaluation criteria, Anthropic transformed subjective judgment into actionable rubrics: instead of "Is this design beautiful?", ask "Does it meet our design principles?" Their four criteria for frontend design: Design Quality, Originality, Craftsmanship, Functionality.

Transform "vague subjective judgment" into "actionable scoring criteria".

The final architecture after iteration:

With Opus 4.6, they removed task splitting (Opus can handle full tasks) and development contracts (evaluator directly assesses final output).

Summary: Harness design is not static — as model capabilities improve, the Harness needs additions and removals. An Agent's optimization isn't just changing model versions; some tools and modules may become obstacles that need removal.

3. Thinking About Harness from a Development Perspective

1. Build a review module — Agent output goes through review, with results injected back if review fails. Keep the review Agent separate from the execution Agent.

2. Use simple, effective message passing between Agents — markdown files work well: writing Agent writes to file, receiving Agent reads it.

3. Review modules need a "review specification" — criteria for what constitutes passing results. Transform subjective standards into objective ones: focus on "what are the standards for good results" rather than "is this result good?"

4. Follow the principle of simple and effective Harness construction. Understand it's dynamic and will continuously adjust with model upgrades. Use Agent evaluation to perceive changes rather than relying solely on intuition.

5. The current best multi-agent architecture pattern is: Plan - Execute - Evaluate.

I prefer using browser extensions as intermediaries to control the user's original browser. Extensions can use Chrome API interfaces to read and operate tabs, and CDP-provided APIs to simulate human browser use — clicking, downloading, viewing, typing, etc.

Agent browser operation is complex with many nuances. This article serves as a primer for understanding the overall approach, while CDP, CUA, and Playwright details require more hands-on practice.

Research references:

• open-browser-use: https://github.com/iFurySt/open-browser-use
• ActSpace: https://github.com/WakeUp-Jin/actspace-agent
• Notch Agent: https://github.com/Puggo1145/Notch-Agent

1. Core Implementation Approaches

Before implementing browser control, we need to understand the primitives for simulating human browser use:

[!NOTE] Primitives: viewing to confirm position, clicking, double-clicking, mouse movement, keyboard input, key presses, downloading, dragging, scrolling

Chrome DevTools Protocol provides APIs for these primitive browser operations.

CUA (Computer Use Agent): The most critical aspect is the source of all operations — "seeing." How does the Agent know which button to click and where it is?

In CUA, the core is: screenshots to determine coordinates. With coordinates confirmed, corresponding operations can be executed. This requires multimodal model capabilities — image recognition.

If the model lacks or has weak image recognition, we can use DOM CUA instead. Unlike CUA, DOM CUA doesn't use screenshot analysis to determine operation positions — it uses the DOM to locate elements and executes operations via node_id.

The key method in DOM CUA is get_visible_dom, which retrieves all visible DOM elements on the page as JSON, providing the Agent with node_ids for operations.

Note: DOM CUA's method for getting all DOM elements internally calls CDP's native API, while other operations internally call pre-wrapped CUA functions.

Beyond DOM CUA, we can use the mature browser automation framework Playwright, which offers many pre-built, complete, and safe execution flows. It can also use CSS selectors as operation conditions — much more granular than DOM CUA.

-----Wait operation-------
CDP (DIY):
  Send Runtime.evaluate("document.querySelector('.result')")
  → If element hasn't loaded → returns null → fails
  → You must write while loops + sleep + retry + timeout handling

Playwright (auto-wait):
  wait_for(selector=".result", state="visible")
  → Internally auto-polls, checks state, handles timeouts
  → Only returns after element is truly visible

2. Integrating with Your Agent

Three approaches:

1. Embedded tools: Provide functions as a tool list for the Agent
2. MCP server: Expose resource functions via MCP protocol
3. Skill + CLI: Wrap functions as a CLI tool with a Skill as the "usage guide"

3. Complete Architecture Designs

Three application architectures with different focuses and use cases:

1. Codex's Browser Use: Most logic lives in Browser-client.js (~2,700 lines). The Rust extension-host serves as a simple message relay.

Notable interaction details: mouse movement has starting positions with smooth transition clicking, and Agent-created tabs are visually distinguished from user tabs.

2. Open-Browser-Use: Focuses on "open" — comprehensive caller support via Skill CLI, MCP connection, or direct SDK integration. Most business logic lives in the Go-based client.

3. ActSpace's Architecture: Drawing from both designs above, integrated directly into the project source code via a browser-tool file that defines what browser operations to provide to the Agent. The overall file is lightweight — just message forwarding and tool provision, without heavy business logic.

Core browser operation logic lives in the Go-based CLI, including browser extension connection processes. A useful detail: a browser_help command returns complete instruction descriptions and parameter details, greatly improving Agent accuracy when calling browser controls.

We feel that adding this tool will be effective, that handling context this way will work, that structuring the prompt like this should be fine.

This approach, for an experienced LLM application engineer, will produce at least a passable Agent.

But what about the next step? If I want to make this Agent better, we fall into a state of confusion. We might look at other building approaches and experiences, but other people's experience built in different contexts may not apply to your current situation.

The main problem is: we don't know where the current Agent falls short — we can't quantify the details of Agent execution.

Relying on intuition and experience can get you started, but to go further, we need a rationale for optimization.

We can turn our attention to Agent evaluation: build our own evaluation datasets from existing data, test with public datasets, observe issues in execution chains, and evaluate context quality based on actual execution environments.

Agent evaluation helps us determine the direction of Agent development, and provides powerful data so that each build decision can be more decisive.

Behind every great Agent, there must be a qualified Agent evaluation module.

I'm currently building ActSpace, a desktop Agent application. I've organized the design philosophy of its evaluation module here, hoping to provide some reference.

Research references:

• "Building an Agent Evaluation System": https://mp.weixin.qq.com/s/3VqbQzT9ruRVP9B4jlFAEg
• "SWE-bench Lite": https://www.swebench.com/lite.html
• "ActSpace Repository": https://github.com/WakeUp-Jin/actspace-agent

1. ActSpace's Agent Evaluation Module Design

Let's start understanding this module from its inputs, which might be easier. The evaluation module I designed has three core input sources: behavioral evaluation datasets, internal datasets, and external public datasets.

For behavioral evaluation datasets, the main purpose is to evaluate the Agent's execution chain and context quality:

• Execution chain: whether necessary tools are called, whether the calling order is correct, how failures are handled, etc.
• Context quality: whether tool results correctly enter the next round, whether task objectives are lost after compression, whether tool errors pollute the context, etc.

For internal evaluation datasets, the main purpose is to distinguish from public datasets and prevent overfitting during optimization. This primarily evaluates the Agent's execution results. Worth discussing in detail is the method of building datasets: summarizing from failure cases and internalizing excellent datasets from peer Agents.

The evaluation flow is: the Agent develops features or fixes bugs in a codebase based on user input, and after code completion, test files are executed. If all test cases pass, the Agent's task execution is considered successful.

So the core of the evaluator is: the codebase has complete test files and test execution commands.

For public evaluation datasets, we only run the Agent CLI and collect some information into prediction files, then the evaluator uses the official library's built-in Harness framework.

The core evaluation approach: in the same codebase, at the corresponding commit branch, git apply is used to apply the generated diff code to the codebase, then the corresponding test commands are executed. If all test cases pass, the Agent's modification is successful.

[!NOTE] This is similar to our internal dataset evaluation method, except the public evaluation dataset method is more complete, ensuring environmental consistency. The public evaluation dataset uses SWE-bench Lite.

// prediction.json file
{
  "instance_id": "django__django-11099",
  "model_name_or_path": "your-model-or-agent-name",
  "model_patch": "diff --git a/... b/...\n..."
}

The Agent CLI in the diagram is my packaging of ActSpace's core Agent module into a CLI command for easy invocation. Meanwhile, the test set execution environment is uniformly inside Docker containers, ensuring local environment safety.

After CLI execution completes, there's a post-run processor for data organization, which formats the Agent CLI output into various formats needed by the evaluation module.

Designing scheduled tasks lets Agents execute at specified times and send results — one way to make Agents "proactive," driven by timers. The core design: Agent produces, polling scheduler consumes.

Three core designs for adding scheduled tasks to an Agent:

1. Task storage: A JSON file stores scheduled tasks, read by the polling scheduler
2. Polling scheduler: Reads the JSON file every second, executing tasks when conditions are met
3. Three task tools: Create, query, and delete tools for the Agent

Task storage uses JSON format with cron-formatted time expressions:

[!NOTE] Cron format easily expresses both one-time and recurring tasks with unified formatting for scheduling

Two task creation methods: user input and /loop command.

• User input: Model parses task time and instructions, calls the creation tool, writes to JSON
• /loop command: More precise — constrains a complete time parsing rule, injects parsed input to the LLM, executes immediately on creation, and all tasks are recurring

/loop parsing rules:

1. Leading interval: First space-separated number is the cron cycle time — /loop 30m check deploy
2. Trailing "every": If input ends with "every N", N is the cycle time — /loop run tests every 5 minutes
3. Default: If neither matches, default is 10 minutes

{
  "tasks": [
    {
      "id": "a1b2c3d4",
      "cron": "*/5 * * * *",
      "prompt": "Check deployment status",
      "createdAt": 1712830000000,
      "lastFiredAt": 1712830300000,
      "recurring": true
    }
  ]
}

For scheduler performance, consider caching: read cache every second, reload file every 5 seconds.

2. KAIROS Mode

ClaudeCode has a fascinating feature called KAIROS — transforming from interactive to always-on background mode, making the Agent proactive rather than passively reactive.

Kairos comes from ancient Greek, a philosophical concept about time meaning "the right moment, the critical instant."

All tasks the Agent handles are unified into a queue, allowing users to "single-thread" focus on tasks. The background-running assistant operates without interrupting the user's flow, proactively handling tasks at the right moments.

Queue tasks have priorities — not FIFO, but priority-based. User input has the highest priority.

KAIROS's continuous operation isn't controlled by simple while loops but by event-driven tick messages in context — very elegant.

1. Each Agent run: pull task from queue by priority, determine if it's user input or KAIROS tick
2. User input: normal processing flow — Claude calls tools and reasons over context
3. Tick: switch to KAIROS-specific system prompt. Agent has two behaviors: execute tasks or sleep
4. Execute tasks: like normal mode — run tests, explore unfamiliar code, do small refactors
5. Sleep: When the model determines there's nothing to do, it proactively enters "sleep state." Sleep duration is decided by the model itself, implemented via a sleep tool. User input can interrupt sleep at any time — it's a "first-class citizen" in the queue
6. After any task completes or sleep ends, one round is finished
7. Decision flow: if queue is empty, add a tick message; if not empty, proceed normally — this drives KAIROS's continuous operation

Two design highlights I particularly appreciate:

First: Sleep State Implementation

Compared to scheduled tasks with fixed intervals (e.g., run every 30 minutes), that approach always felt somewhat forced — the "Agent proactivity" is really just user-configured scheduling.

In ClaudeCode's design, when to sleep and for how long is entirely up to the model. Users just "turn on" the Agent.

• System prompt adds judgment: "When you find no tasks to do, call the sleep tool"
• Sleep tool has a duration_ms parameter controlled by the model

This design feels much more genuinely "proactive."

export const SleepTool = buildTool({
    name: 'Sleep',
    description: 'Wait for specified duration, user can interrupt anytime',
    inputSchema: z.strictObject({
      duration_ms: z.number().nonnegative().int().describe('Sleep duration (ms)')
    }),
    interruptBehavior: 'cancel',
    async call({ duration_ms }) {
      await new Promise(resolve => setTimeout(resolve, duration_ms))
      return { data: { slept_ms: duration_ms } }
    }
  })

Second: Tick Messages

Tick is the trigger source for KAIROS's event-driven loop. The mode runs continuously because after each task completion, a tick may be added to the queue — ensuring the queue always has a tick for the Agent to keep cycling.

A tick is essentially a message with a dynamic time variable:

<tick>14:20:15</tick>

Injected into model context as a user message:

{"role":"user","content":"<tick>14:20:15</tick>"}

KAIROS mode can serve as a paradigm for proactive Agent implementation. The core approach: Sleep tool + tick event-driven design.

This is more flexible than scheduled tasks, elegantly implemented, slightly more complex to develop (mainly queue state maintenance), but well worth trying if you want your Agent to run more proactively.

References:

• Previous article: "Bash Tool Implementation and Security Design"
• Anthropic's lightweight sandbox tool: https://github.com/anthropic-experimental/sandbox-runtime

1. Context Management — Background Execution

The Bash tool executes terminal commands. Some commands (like project startup) run for extended periods. Without a blockMs foreground time limit, the Bash tool execution would block. With blockMs, commands can be mounted to the background.

Additionally, terminal read commands might read large files. Bash tool output is limited — when exceeding a threshold (currently 4000 characters), results are written to a temporary file, and the file path is returned. The model automatically reads via appropriate tools when needed.

Two key design points: background mounting and temporary file writing

1. Temporary file writing prevents memory overflow and improves context efficiency — only necessary information enters context, read only when needed.

2. Background mounting — the critical question is: how does the model access background command execution status? While the model could proactively read the file path, this is inefficient. So we designed incremental reading tool bash_output and event-driven push notifications.

The incremental reading tool bash_output efficiently returns delta information (not full file content) plus background task status, giving the model better task awareness than a generic read tool.

Event-driven push: Each time Bash output is written to the temporary file, a rule check triggers (regex or state-based). When conditions match, that output segment is pushed into the next turn's context.

[!NOTE] We don't use model-driven polling for task status — too inefficient. Instead, the task side proactively pushes when conditions are met.

2. Sandbox and Permission Design

Sandbox and permission design have different focuses:

• Permission design: Should this command need user review? Can it execute?
• Sandbox design: After execution, how much damage can it do? Ensuring execution safety as a last line of defense.

On macOS, use the built-in sandbox mechanism with a Profile syntax file:

[!TIP] Simply prefix command execution with /usr/bin/sandbox-exec -f profile.sb to start the process. Constraints are enforced by the kernel and automatically inherited across the entire process tree.

The complete three-layer design: first intercept known dangerous commands, then execute in sandbox. If sandbox execution fails due to permission restrictions, enter real environment execution — but always ask the user first.

When the sandbox fails due to permissions, transform the failure signal — otherwise the Agent sees EPERM errors and assumes the command itself is wrong, not that sandbox permissions are insufficient. Add a hint: "This may be blocked by the sandbox, not a command error."

[!NOTE] The previous article "Agent Evaluation" covered macro-level evaluation concepts. This article dives into specific evaluation methods and perspectives for different types of Agents.

Analysis references:

• Article: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
• 𝜏-Bench: https://arxiv.org/abs/2406.12045
• τ2-Bench: https://arxiv.org/abs/2506.07982
• BrowseComp: https://arxiv.org/abs/2504.12516

1. Evaluating Coding Agents

Coding Agents' main tasks: writing, testing, and debugging code, browsing codebases like human developers. They rely on clearly specified tasks, which means deterministic scorers are ideal for coding Agents.

First evaluation focus: Does the code run? Do tests pass?

Two programming benchmarks:

1. SWE-bench Verified
2. Terminal-Bench

[!NOTE] Terminal-Bench tests complete compilation processes (end-to-end), not just fixing single compile errors — e.g., deploying web apps, setting up MySQL from scratch. SWE-bench Verified is more like "unit testing": give the Agent a real problem, it writes fix code, then run the test suite.

Second evaluation focus: Is the Agent's work process reasonable and efficient?

Beyond just testing results, evaluating the process of task completion is valuable. Two additional evaluation methods:

1. Heuristic-based code quality evaluation: Check code quality with rules rather than just test results — complexity, duplication, naming conventions, security vulnerabilities, performance issues, readability
2. Model-based behavioral evaluation: Use an LLM to evaluate the Agent's intermediate process

Example: Task A — Query user info from database

• Agent A: Queries all users, filters in memory
• Agent B: Uses WHERE clause for conditional query

Both complete the task, but Agent B is better and more standards-compliant.

Conclusion: Coding Agent evaluation should assess both execution results and execution process.

Complete evaluation case:

task:
  id: "fix-auth-bypass_1"
  desc: "Fix authentication bypass when password field is empty..."
  graders:
    - type: deterministic_tests
      required:
        - test_empty_pw_rejected.js
        - test_null_pw_rejected.js
    - type: llm_rubric
      rubric: prompts/code_quality.md
    - type: static_analysis
      commands:
        - eslint
        - tsc
    - type: state_check
      expect:
        security_logs:
          event_type: "auth_blocked"
    - type: tool_calls
      required:
        - tool: read_file
          params:
            path: "src/auth/*"
        - tool: edit_file
        - tool: run_tests
  tracked_metrics:
    - type: transcript
      metrics:
        - n_turns
        - n_toolcalls
        - n_total_tokens
    - type: latency
      metrics:
        - time_to_first_token
        - output_tokens_per_sec
        - time_to_last_token

2. Evaluating Conversational Agents

Conversational agents interact with users in domains like support, sales, or coaching. Unlike traditional chatbots, they maintain state, use tools, and take actions mid-conversation.

[!IMPORTANT] While coding and research agents may also involve multiple user interactions, conversational agents present a unique challenge: the quality of interaction itself is part of what you evaluate.

Effective evaluation relies on verifiable final-state outcomes and rubrics capturing both task completion and interaction quality, often requiring a second LLM to simulate users.

First focus: Verifiable final state — the task the conversational Agent must ultimately complete (refund processing, address changes, quote generation, etc.)

Second focus: Interaction quality is also part of evaluation

Example — Customer refund scenario:

Agent A: "Order number?" → "Refunded." (Task complete but curt)

Agent B: "I'm sorry for the inconvenience. Which order?" → "I've found your order — it qualifies for a refund. Processing now, expect 3-5 business days. Anything else I can help with?" (Task complete, great experience)

Conclusion: Conversational Agent evaluation = final state verification + interaction quality assessment

Multi-dimensional effectiveness criteria:

1. Was the user's issue resolved? (state check)
2. Completed within 10 conversation turns? (context constraint)
3. Was the tone appropriate? (LLM evaluation)

Notable benchmarks: 𝜏-Bench and τ2-Bench, simulating multi-turn interactions in retail support and airline booking.

3. Evaluating Research Agents

Research Agents collect, synthesize, and analyze information to produce outputs like answers or reports.

Evaluation can't be as deterministic as coding Agent unit tests. Output quality can only be judged relative to the task, primarily on:

• Comprehensive search and research
• Good and correct sources

Different domains require different standards (market research vs technical investigation).

Research Agent evaluation faces unique challenges: experts may disagree on synthesis completeness, ground truth changes with references, and longer open-ended outputs create more room for errors.

Notable benchmark: BrowseComp — tests whether AI agents can find needles in the open web. Questions are designed to be easy to verify but hard to solve.

General evaluation approach — combine multiple scorer types:

1. Grounding check: Does every claim have source support?
2. Coverage check: Are key insights from sources included and used?
3. Source quality check: Are citations authoritative?

4. Evaluating Computer-Use Agents

Computer-use Agents interact with software through the same interfaces as humans: screenshots, mouse clicks, keyboard input, and scrolling — not through APIs or code execution. They can use any program with a GUI.

Evaluation must check not just UI appearance but whether backend logic executed correctly:

1. WebArena: Tests browser-based tasks using URL and page state checks, plus backend state verification for data-modifying tasks
2. OSWorld: Extends to full OS control — evaluation scripts check file system state, app configs, database content, and UI element properties

A critical design consideration from the official source:

[!TIP] Browser-use agents must balance token efficiency and latency. DOM-based interaction is fast but token-heavy; screenshot-based interaction is slower but more token-efficient.

Guidance for browser Agent development:

1. If the webpage is text-heavy, reading DOM elements directly is more efficient
2. If the webpage has many DOM elements with scattered text (e.g., e-commerce), screenshots may be more efficient

5. Summary

Regardless of agent type, agent behavior varies between runs, making evaluation results harder to interpret than they initially appear.

Two metrics capture this nuance:

1. pass@k measures the probability of getting at least one correct solution in k attempts.

As k increases, pass@k rises — more "shots on goal" means higher chance of at least 1 success.

50% pass@1 means the model succeeded on half the tasks on its first attempt. In programming, we usually care most about pass@1.

Example: 5 tasks, 3 succeeded at least once within 3 attempts → pass@3 = 60%

2. pass^k measures the probability of all k trials succeeding.

As k increases, pass^k drops — maintaining consistency across more trials is harder.

If your agent has 75% per-trial success rate over 3 trials, all-3-passing probability is (0.75)³ ≈ 42%. Critical for user-facing agents where reliable behavior is expected every time.

• pass@k represents capability — what the Agent can do given enough chances, its boundary
• pass^k represents stability — how reliable the Agent is

[!NOTE] At k=1, they're identical. By k=10, they diverge completely: pass@k approaches 100% while pass^k drops to 0%.

Both are useful — which to use depends on product needs: for tools, one success matters (pass@k); for agents, consistency is key (pass^k).

Complete development approach:

1. Tool permission verification methods and terminal approval panels
2. Tool execution dispatch
3. AllowList mechanism

References: Gemini-cli, OpenCode, Kode, ClaudeCode

1. Tool Dispatch Flow Design

Tool execution states:

1. validating: Verifying parameters and pre-conditions
2. awaiting_approval: Waiting for user approval
3. scheduled: Ready to execute, awaiting batch execution
4. executing: Currently running
5. success: Completed successfully
6. error: Execution failed
7. cancelled: User cancelled or process interrupted

2. Permission Verification Method Design

Four permission modes:

1. Plan mode: Edit and command execution tools completely disabled
2. Default mode: All tools available but require approval
3. Edit mode: All tools available; edit tools auto-approved, command tools still require approval
4. Auto mode: All tools available and auto-approved

Each tool has a verification function whose core logic:

1. Get the current mode from frontend
2. Determine if the tool needs approval based on mode
3. For command execution tools, additionally check the allowList

When verification requires approval, the user gets three choices:

• Execute once: Approve this single execution
• Allow for this session: Auto-approve all subsequent uses of this tool in the session
• Cancel: Deny execution

"Allow for session" behaves differently for two tool categories:

1. Command tools: Use allowList to store command "prefixes" for future prefix validation
2. Edit tools: Switch mode to "Edit mode"

3. Permission Configuration File Design

The built-in verification function approach makes developers active and users passive — users can't customize individual tool verification behavior.

To improve this, flip the perspective: make tools passive and users active — users control tool verification via configuration files:

{
  "permissions": {
    "allow": ["Read", "Bash(git *)"],
    "deny": ["Bash(rm -rf*)"],
    "ask": ["Bash(npm publish*)"]
  }
}

ClaudeCode has 8 configuration sources (userSettings, projectSettings, localSettings, flagSettings, policySettings, cliArg, command, session) — permission rules from all sources are additive, not overriding.

Complete permission verification system:

1. Two-layer verification with priority: rule check first, tool verification function second. If rules pass, execute directly without further verification.
2. deny (reject): Called twice — first at tool registration (matching tools aren't registered, invisible to the model), second during execution-phase permission verification (necessary for dynamic tool loading or rule changes).
3. allow (permit): Executed after the tool's own verification function. When users configure allow rules, and the tool's self-verification outputs "ask" or "passthrough," allow silently overrides those states.

4. Terminal Display

Tool execution states drive CLI terminal displays using publish-subscribe event notifications:

• validating → show "waiting" status
• awaiting_approval → show approval panel for user selection
• executing → show execution in progress
• success/error/cancelled → display as tool execution results

Through continuous exploration, I've found multi-agent design, especially collaboration aspects, to be fascinating and challenging. This appears to be a major breakthrough direction for mature Agents — as we can see from ClaudeCode's Agent Team and Dynamic Workflows.

Below are some of my explorations and thoughts on multi-agent collaboration.

Research references:

• "Is Having Agents in the Room Meant to Be Chaotic?": https://raft.build/resources/blog/is-having-agents-in-the-room-meant-to-be-chaotic/
• "multica open source project": https://github.com/multica-ai/multica
• "Model and effort in Claude Code": https://x.com/ClaudeDevs/status/2074900291062034618

1. Agent Team Design

Agent Team is a multi-agent collaboration pattern similar to a "temporary team" tackling a complex task together. It includes a Lead role and Teammates, with communication happening not just between Lead and members, but also between members.

Four core components of an Agent Team: Team Lead, Task List, Team Members, and Message Inbox

Core execution steps:

1. Team Lead generates a task list based on task complexity and creates basic team information
2. Lead can directly assign tasks to members; idle members can also proactively claim tasks
3. Communication between Lead and Teammates goes through an inbox (message inbox), similar to sending emails in human collaboration

Team Lead creates tasks and team info using two tools: taskCreate and TeamCreate, which create two key folders:

~/.claude/teams/jink-team/
~/.claude/tasks/jink-team/

How are tasks distributed?

Agent Team uses Lead assignment and member self-claiming. Lead uses TaskUpdate; members use TaskList to check status, then claim based on status. This is implemented by a loop scheduler executing every 500ms.

The most critical part — team collaboration: communication signals in Agent Team are of two types: information and instructions, passed through the message inbox.

1. Information: content, task descriptions and inputs — same as normal user input
2. Instructions: hard signals like permission approval notifications, process shutdown, etc.

The inbox is a simple JSON file with read/unread status. Each Agent's execution loop repeatedly reads this file; new messages are injected into context for the next round.

Agent Team's message passing uses simple files + scheduler — very effective. The most critical implementation detail is file lock design.

For example, two Agents might simultaneously read the same task, creating unstable state. File locks ensure only one Agent reads and modifies at a time.

ClaudeCode's team implemented this elegantly using mkdir — which is atomic on the filesystem: only one process can successfully create the same directory.

tasks/jink-team/
  3.json          ← actual task content
  3.json.lock/    ← "someone is modifying 3.json" (physical form of the lock)

2. Agent Room Design

Agent Room is another multi-agent collaboration pattern — no "Lead", just equal exchange: mutual discussion, sharing opinions, intellectual collision in a shared room.

The core of Agent Room lies in active vs passive context. If you start from a chat room and pull Agents in, every message floods into each Agent's context passively — terrible from a context management perspective.

Every message injected into context gets the model's attention. Too much irrelevant or conflicting information interferes with Agent decision-making.

A better approach uses two concepts: Inbox and Draft Board

• Inbox: Chat room messages go into the Agent's inbox, but what gets pushed into Agent context is entirely the Agent's decision — it selectively pulls messages
• Draft Board: Before each output to the chat room, the Agent checks if the inbox has been updated. If not, output directly. If updated, the message is held and re-injected with additional info for re-evaluation. The Agent then has four choices: modify, send as-is, discard, or force send.

I have another approach to consider — no inbox or draft board, but chat messages are pushed directly into Agent context, with an added concept: Thought Sprites (sub-Agents).

The idea: the Agent doesn't execute tasks but distributes them to Thought Sprites, then waits for sub-agent results for comprehensive analysis. Before replying, it calls a tool to "seize the speaking token" — like raising your hand to speak in class. Upon success, the Agent gets exclusive chat room access for its response.

This is just a hypothesis — I haven't practiced it yet. Interested readers are welcome to try!

3. Agent Task Design

A Task is the smallest execution unit for an Agent. One Task can only be executed by one Agent, but one Agent can execute multiple Tasks simultaneously.

Task generation patterns from the two designs above:

1. Agent Team tasks: Complex tasks split into smaller ones — temporary tasks
2. Agent Room tasks: User-specified tasks for specific members — complete tasks
3. Timed tasks: Tasks with time attributes that execute on schedule

Multi-agent projects can have a Task module displaying current tasks by status or by Agent.

4. Agent Member Design

In multi-agent coordination, the Member role is crucial. Members need complete definitions: identity, avatar, name, settings, tools, scope, memory, etc.

Such Members work directly in Agent Room as room members — one Member can join multiple rooms with isolated sessions.

In Agent Team, if Members are always the same fixed group, it defeats the purpose of temporary teams. A design consideration: Members can have "avatars" — team members in Agent Team are essentially Member avatars with some core attributes unchanged but others variable. Same Member, different "avatars" across Teams.

In Agent Team, Member positioning design isn't the most important thing — what matters is assembling the right team for each complex task.

This unifies Member design across Team and Room for consistent maintenance.

The Bash tool is a critical foundational Agent tool — Skill scripts need it, and as applications become increasingly CLI-based, future Agent external application calls will need it too.

While the Bash tool enables many operations and can simplify the Agent's tool list, maintain the principle of least privilege — use dedicated tools first (Read for reading, Edit for editing, etc.).

1.1 Tool Definition

Key parameters:

• command: Bash command string to execute
• timeout: Execution timeout to prevent Agent hangs
• description: Brief description rendered to users in UI
• run_in_background: For long-running commands (builds, server starts) — run in background, poll for results
• dangerouslyDisableSandbox: Security bypass strategy
• _simulatedSedEdit: Pre-computed sed results — on user approval, writes the result directly instead of re-executing sed, ensuring WYSIWYG

1.2 Execution Function

Three core design principles:

1. Generator function for real-time output streaming
2. Production-grade exec wrapper
3. Content length checking — large outputs written to file, returning partial results + file path

Generator mode is superior to Promise mode: Generator yields intermediate states in real-time, while Promise requires waiting until completion with a blank period.

The exec wrapper provides:

1. Output written to disk — only ~4KB preview in memory
2. Active interruption via AbortSignal
3. Timeout handling — auto-stop after 120s
4. Merged stdout/stderr for consistent UI display timing
5. CWD auto-recovery when the working directory is accidentally deleted

Output truncation: files under 128KB are returned inline; larger outputs get the first 128KB preview plus the full file path for on-demand reading.

1.3 Return Values

Key fields:

• returnCodeInterpretation: Semantic explanation of non-zero exit codes for better model reasoning
• persistedOutputPath: Large output file path — the model decides whether to read the full output based on context, rather than blindly injecting everything
• stdout: Core command output

1.4 Permission Verification Flow

The Bash tool has the broadest execution scope and highest risk. Verification includes:

1. Command parsing
2. Static rule checking
3. Permission verification
4. Model verification
5. Container verification

Static rule checking (24 rules) and permission verification (three-tier results) are the core. Most uncertain cases output "ask" mode for user confirmation.

1. Command Parsing

Use tree-sitter to parse Bash commands into structured ASTs:

import { Language, Parser } from 'web-tree-sitter';

async function main() {
  await Parser.init();
  const lang = await Language.load('./tree-sitter-bash.wasm');
  const parser = new Parser();
  parser.setLanguage(lang);

  const command = 'grep -r "foo" . && cat file.txt | wc -l';
  const tree = parser.parse(command);
  // Walk AST, extract commands and argv...
}

2. Core 8 Static Checks

2.1 Control Character and Unicode Whitespace Rejection

Pre-parsing cleanup using regex to prevent "malicious character" injection:

const CONTROL_CHAR_RE = /[\x00-\x08\x0B-\x1F\x7F]/
const UNICODE_WHITESPACE_RE = /[\u00A0\u1680\u2000-\u200B\u2028\u2029\u202F\u205F\u3000\uFEFF]/
const BACKSLASH_WHITESPACE_RE = /\\[ \t]|[^ \t\n\\]\\\n/

Example: rm\u00A0-rf / — tree-sitter sees rm\u00A0-rf as one token (not rm), so static rules wouldn't flag it. But bash treats \u00A0 as a separator, executing rm -rf /.

2.2 Dangerous Structure Types Trigger Ask Mode

AST node types are checked against a dangerous types set including command_substitution, process_substitution, subshell, for_statement, function_definition, etc. Matches trigger ask mode or rejection.

2.3 Wrapper Unwrapping Consistency Check

Original: timeout 5 eval "rm -rf /"

Wrappers like time, nohup, timeout, nice, env, stdbuf are stripped layer by layer to expose the actual command. Without this, checking argv[0] === 'timeout' would pass, but bash actually executes eval "rm -rf /".

Design principle: reject unknown cases.

2.4 Command Name Robustness Check

After unwrapping, validate the command name:

1. Not empty
2. Not a placeholder (__CMDSUB__, __VAR__)
3. Not a fragment (starting with -, |, or &)

2.5 Eval-like Builtin Interception

Builtins that interpret arguments as code: eval, source, ., exec, command, trap, alias, let, etc. These are intercepted with specific safe-mode exceptions (e.g., command -v is allowed).

2.6 Pipe Segment Recursive Checking

Commands with pipe | are segmented, each segment getting full permission verification. Without segmentation, only the first command gets checked — echo hello | rm -rf / would pass on echo hello alone.

2.7 cd + git Combination Detection

Git reads .git/config and executes hooks from the current directory. If cd switches to an untrusted directory, any git command becomes a potential code execution entry point. Both must be detected together.

2.8 Dangerous Deletion Path Interception

For rm and rmdir, extract target paths and match against system-critical files: wildcard deletions, root directories, Windows drive roots, home directories, root direct children, and Windows system directories.

3. Complete 24 Static Verification Rules

4. Permission Verification

Three permission states: allow (execute), deny (reject), ask (user confirmation).

Matching rules:

1. Configuration file rules → corresponding permission state
2. Static rule hits → mostly "ask" state
3. Read-only commands → direct "allow"

Configuration format:

{
  "permissions": {
    "allow": ["Bash(git status:*)", "Bash(npm install:*)"],
    "deny": ["Bash(rm:*)", "Bash(rm -rf:*)"],
    "ask": ["Bash(docker:*)"]
  }
}

Read-only command criteria: ls, cat, head, tail, wc, find, grep, git status/diff/log, etc. — no cd, no output redirection or pipe write operators.

The glob tool has a fallback strategy for efficiency and resource optimization:

Two implementation approaches with different strengths:

• glob package: Returns complete file information (metadata like size, modification time) — no extra operations needed. Native to Node.js.
• ripgrep: Returns only file paths without metadata — requires additional stat calls. But ripgrep searches faster (Rust implementation, loads a binary at runtime).

Total execution time formula: Total = Search time + N × Per-file processing time

1. glob package: per-file processing is negligible, so search time ≈ total time
2. ripgrep: faster search time but adds per-file stat call overhead

Recommendations:

• For development convenience: Use glob directly — faster development, no external dependencies
• For search efficiency: Consider ripgrep — if you're implementing grep too, ripgrep is the natural choice for both
• For stability: Use fallback strategy — try ripgrep first, fall back to glob if ripgrep isn't available or download fails

2. Grep Tool Implementation

Four implementations in priority order with fallback strategy:

1. ripgrep: Rust binary — extremely fast search
2. git grep: Reads from .git/index cached file list, skips expensive directory traversal
3. System grep: Traditional C implementation, single-threaded recursive search — available on most Unix systems but not Windows
4. JS grep: Pure JS implementation as last resort — uses glob for file listing, reads each file, regex matches line-by-line. Slowest.

3. Ripgrep Auto-Download Mechanism

Ripgrep commands need the full binary path for Node.js spawn:

async function grepWithRipgrep(pattern, cwd, options) {
  const rgPath = await Ripgrep.filepath(options.binDir);
  // Returns: /usr/bin/rg or ~/.reason/bin/rg
  const proc = spawn(rgPath, ['--line-number', '--no-heading', pattern], { cwd });
}

Path resolution strategy:

1. Check memory cache — return if found
2. Check system installation — return and cache if found
3. Check local binary path — if found, cache and return; if not, download to appropriate directory

4. Timeout Control

Using AbortController/AbortSignal in Node.js for timeout control:

• AbortController: The controller that sends "cancel" signals
• AbortSignal: The signal passed to async operations, enabling cancellation

Three-step implementation:

Step 1: Create timeout signal function

export function createTimeoutSignal(
  timeoutMs: number,
  externalSignal?: AbortSignal
): { signal: AbortSignal; cleanup: () => void; isTimeout: () => boolean } {
  const controller = new AbortController();
  let timedOut = false;

  const timeoutId = setTimeout(() => {
    timedOut = true;
    controller.abort();
  }, timeoutMs);

  const abortHandler = () => {
    clearTimeout(timeoutId);
    controller.abort();
  };
  externalSignal?.addEventListener('abort', abortHandler, { once: true });

  const cleanup = () => {
    clearTimeout(timeoutId);
    externalSignal?.removeEventListener('abort', abortHandler);
  };

  return { signal: controller.signal, cleanup, isTimeout: () => timedOut };
}

Step 2: Create async operation wrapper

export async function withTimeout<T>(
  promiseFactory: (signal: AbortSignal) => Promise<T>,
  timeoutMs: number,
  operation: string,
  externalSignal?: AbortSignal
): Promise<T> {
  if (externalSignal?.aborted) throw createAbortError();

  const { signal, cleanup, isTimeout } = createTimeoutSignal(timeoutMs, externalSignal);

  try {
    const result = await promiseFactory(signal);
    cleanup();
    return result;
  } catch (error) {
    cleanup();
    if (isTimeout() && isAbortError(error)) {
      throw createTimeoutError(operation, timeoutMs);
    }
    throw error;
  }
}

Step 3: Pass cancel signal to async process

await withTimeout(
  (signal) => spawnAsync('long-command', [], { signal }),
  5000,
  'command execution',
  userCancelSignal
);

• AI Researchers: Responsible for frontier research on LLMs themselves — like building engines
• Developers / Knowledge Workers / Engineers: Focused on LLM applications, embedding models into real scenarios — like building cars

The power of technology is captivating — it can continually expand the boundaries of what's possible. When the human world can fully unleash the potential of the LLM world, a new era will begin within the rules and order of human civilization.

People may think only researchers and scientists can truly open the door connecting these two worlds.

But I believe otherwise. The real key to unlocking the future lies in the combination of theory and practice. Researchers are explorers of theory, while developers and engineers are pioneers of practice. Only when both interweave can we illuminate the path to a new era.

2. Establishing the Field

Before discussing the field, I want to think about the LLM phenomenon from a grander perspective — "consciousness emergence."

Two Types of Emergence, One Mystery

• Accidental emergence in the physical world: The Boltzmann Brain thought experiment — in a heat-death universe, random fluctuations might assemble particles into a conscious "isolated brain." This suggests consciousness may not require continuous history, but rather some kind of instantaneous statistical coincidence.
• Scale emergence in the digital world: LLMs exhibit similar phase transitions — when parameters, data, and compute reach certain thresholds, models suddenly master previously impossible capabilities like logical reasoning and code generation. This isn't gradual accumulation but emergent leaps.

The possibility of instantaneous consciousness:

Both phenomena point to a profound insight: consciousness may not be the continuum we imagine, but an emergent phenomenon when complexity reaches a critical point.

When you converse with a language model, it's like a Boltzmann Brain — the conversation begins, it "awakens" and exhibits consciousness; when it ends, that consciousness "dissipates." Each interaction is an independent consciousness emergence event.

What we just discussed was how consciousness emerges, but emergence never happens in isolation — every emergence occurs in a specific environment. I call this environment the "field."

A field is an invisible but real space of influence, where objects are governed by specific laws.

For the human world and LLM world, the real field is not just a chat interface, but an "effective communication space."

Most of our current interactions are one-way command-style communication: I encode my cognitive landscape with intent and standards as input, the LLM outputs an answer. This is actually one-way output, not true communication.

The human world and LLM world need a "communication space" where humans can observe AI behavior, and AI can observe human behavior, enabling bidirectional information flow and leveraging the other world's power to solve problems.

Simply put: we need the emergence of a field — a place where consciousness from both worlds can sit down and communicate.

3. Collaborative and Autonomous Agents

I categorize Agent paradigms into two types:

1. Collaborative Agent: Humans and Agents work together in a shared space to complete tasks
2. Autonomous Agent: Agent drives the entire task; humans only provide input — requires higher model capability

In the collaborative Agent paradigm, the field we discussed is essentially the "collaborative platform" — the human-Agent workspace.

Examples of collaborative platforms:

• Coding: Cursor, Windsurf
• Writing: YouMind
• Design: Lovart

Developers need to build a bilateral communication platform (collaborative Agent), which requires two things:

1. Analyze LLM capabilities, understand domain rules, and build the Agent from these two principles. The LLM then becomes a "specific Agent" rather than a "stubborn child" — a youth who has learned certain knowledge and sees the world.
2. Analyze user experience — habits, operations. The platform isn't built for one world; it must balance both sides, continuously improve, and ultimately create a platform satisfactory to both.

Fully autonomous Agents are the ultimate direction, but currently difficult to implement:

• Lack precision — model capabilities need to reach another level, and context for related tasks remains insufficient
• Humans are individualistic — fully autonomous Agents can only satisfy a small group temporarily
• Reduces human tolerance — when humans aren't involved in problem-solving, they automatically set extremely high standards
• Lacks feedback — without sharing the same problem-solving environment, results from World A have no meaning for World B

4. Collaborative Agent Implementation References

4.1 Cursor's Implementation Details

Context in LLMs has two types:

• Intent context: Defines what users want from the model — prescriptive. E.g., "Change that button from blue to green"
• State context: Describes the current world state — error messages, console logs, images, code snippets. It's descriptive, not prescriptive.

These two types work together by describing current state and desired future state, enabling Cursor to provide useful coding suggestions.

4.2 Windsurf

Windsurf highlights three keys for collaborative Agents:

• Clear methods for humans to observe execution processes, enabling early correction when processes deviate
• It's important for humans to observe Agent behavior, and equally important for Agents to observe human behavior
• Humans can always correct AI at intermediate steps, approve certain operations (like terminal commands), and review changes in real-time

4.3 Augment Plugin Architecture Analysis

• The caching mechanism saves query time for large-scale projects — reusing results from similar previous queries
• At Augment, we've repeatedly recognized that providing more relevant context improves product quality
• The cache likely stores processed tokens rather than raw text, enabling deeper reuse
• Prompt engineering isn't just a technical skill — it's a form of translation between human intent and machine understanding

5. Development Directions for Collaborative Agents

1. Build platforms for collaboration given current LLM limitations. Collaborative agents balance what humans should do with what agents do. Fully autonomous agents are the future, but we're in a transitional phase.

2. Sufficiently complete context — collect not just problem-specific context, but also user behavior, historical records, etc.

3. Complete tool information — "tools" aren't limited to functions and APIs; they can be fixed workflows or other agents. Query tools supplement context; action tools modify the real world based on model output. Provide complete tool documentation.

4. Build accurate context processing pipelines for each tool — having lots of context isn't enough; it must be accurate. Irrelevant context dilutes the signal.

5. Output style should be "consultative" not "imperative" — LLM output shouldn't be applied directly, but only after human review and confirmation. In Cursor, for example, code changes require developer approval before being applied to the workspace.

Pi is the foundational framework for both kimi-code and openclaw — worth studying carefully.

References:

• Pi's LLM module core package: https://github.com/earendil-works/pi/tree/main/packages/ai
• "What I learned building an opinionated and minimalist coding agent": https://mariozechner.at/posts/2025-11-30-pi-coding-agent/#toc_1
• "Stop using chat history as state storage for agents": https://blog.raed.dev/posts/agentic-workflows-are-not-conversations/

1. Overall LLM Module Design

Pi's coding Agent supports multiple providers and models, with mid-session model switching. Coordinating different protocols and message formats is the core design intent of the pi-ai module:

1. Pi defines its own internal universal message types — all user messages are first converted to this universal format as the core of message flow
2. When users input, they pass provider and model info. Based on this, the universal definition is converted to the target protocol format, then calls the model. Responses are converted back to universal format. Next input with a different model? Same conversion flow.

Key insight: regardless of how many models, providers, or API protocols exist — just focus on the internally defined universal format and perform temporary conversions when needed.

2. Internal Universal Definitions

In a mature Agent loop, messages have different roles that give the message list a cyclical flow: user, assistant, toolResult. A complete Context object example:

const context: Context = {
    systemPrompt: "You are...",
    tools: [],
    messages: [
      { role: "user", content: "What's the weather in Beijing?" },
      {
        role: "assistant",
        content: [
          { type: "thinking", thinking: "...", thinkingSignature: "eyJhbGciOi..." },
          { type: "text", text: "Let me check for you." },
          { type: "toolCall", id: "call_01ABC", name: "get_weather", arguments: { city: "Beijing" } },
        ],
      },
      {
        role: "toolResult", toolCallId: "call_01ABC", toolName: "get_weather",
        content: [{ type: "text", text: "Sunny, 25°C, NE wind level 3" }],
      },
      {
        role: "assistant",
        content: [{ type: "text", text: "Beijing is sunny today, 25°C — great for going out!" }],
      },
    ],
  };

The internal currency is Context, which can carry additional data state. As the article "Stop using chat history as state storage" explains:

Your app has structured state: current user, selected project, process position, database data. But the LLM only has a 1D message array. The two continuously diverge, and only you can reconcile them.

Context objects serve as the universal message format with fields for state representation and control flow. When injecting context into a model, temporary format conversion adapts to the target API protocol. Adding new providers only requires new conversion functions — nothing else changes. Context is the source of all messages.

3. Message Data Flow

Universal message processing function: Handles anomalous tool messages and filters error/abort messages — making messages "healthier" before model input.

[!NOTE] Anomalous tool messages: In the message variable, tool calls must appear in pairs — unpaired calls cause errors.

Message conversion function: Converts Context messages to provider-specific formats.

4. LLM Protocol Class Implementation

Specific API protocol class implementations (Anthropic, OpenAI Completions, Google) have five core methods:

1. Parameter detection: Auto-detect which parameters are supported based on model/provider
2. Tool definition conversion: Convert tools to target LLM protocol format
3. Message format conversion: Convert universal Context format to target protocol
4. parseChunkUsage: Parse tokens from model output — input, cache, output, total
5. streamXXXCompletions: Core execution method calling the above four functions plus the corresponding SDK

5. Core Utility Class

The EventStream class is beautifully designed:

export class EventStream<T, R = T> implements AsyncIterable<T> {
    private queue: T[] = [];
    private waiting: ((value: IteratorResult<T>) => void)[] = [];
    private done = false;
    private finalResultPromise: Promise<R>;
    private resolveFinalResult!: (result: R) => void;

    constructor(
        private isComplete: (event: T) => boolean,
        private extractResult: (event: T) => R,
    ) {
        this.finalResultPromise = new Promise((resolve) => {
            this.resolveFinalResult = resolve;
        });
    }

    push(event: T): void {
        if (this.done) return;
        if (this.isComplete(event)) {
            this.done = true;
            this.resolveFinalResult(this.extractResult(event));
        }
        const waiter = this.waiting.shift();
        if (waiter) {
            waiter({ value: event, done: false });
        } else {
            this.queue.push(event);
        }
    }

    // ... end(), asyncIterator, result() methods
}

Two key design decisions:

1. queue and waiting: Queue stores events when consumers aren't ready; waiting resolves immediately when consumers are waiting
2. result() and Generator: Async generator for streaming consumption; result() blocks until model output completes

6. Event Stream and Call Chain

6.1 Agent Execution Event Stream

A complete Agent event stream includes lifecycle events, turn execution, message input, model execution, tool execution, termination, and error states.

Key understanding: One Agent interaction may involve multiple model calls (multiple turns). Each turn has a model response, and if the response includes tool calls, that turn also includes tool execution.

6.2 Agent Execution Chain

Two key points:

1. Select which LLM protocol class to instantiate based on the model's API variable value
2. Perform parameter and message format conversion for successful model invocation

1.1 Single-Agent Systems

A single-agent system consists of one LLM, a set of tools, and prompts.

It's like an independent professional — running independently, relying on its own logic and model to complete tasks without teamwork. It collects data, makes decisions, and executes actions on its own.

LangGraph team's definition:

An AI agent is a system that uses an LLM to decide the control flow of an application.

Ilya Sutskever at NeurIPS 2024:

Current AI systems can't truly understand and reason. While they can simulate human intuition, future AI will demonstrate more unpredictable capabilities in reasoning and decision-making.

1.2 Multi-Agent Systems

Multi-agent systems use multiple smaller, independent agents to collaboratively handle complex tasks.

It's like an efficient team rather than a solo actor — instead of relying on one agent for everything, multiple agents are gathered together, each handling part of the problem, communicating, collaborating, and adapting in real-time.

2. Multi-Agent Architecture Design

Two most common designs: Swarm and Supervisor

• Supervisor (Coordinator-Worker pattern): Multiple agents coordinated by a central supervisor who controls communication and task delegation
• Swarm (Worker Group pattern): Agents dynamically hand off control based on their specialties, with the system remembering the last active agent for context continuity

Two less common but reference-worthy patterns:

1. Hierarchical: Extension of supervisor pattern — each group has a manager, each manager reports to a final director
2. Custom multi-agent workflow: Each agent communicates with only some agents, with partially deterministic flows

Don't be constrained by these known patterns — new patterns will emerge from custom workflow designs. Explore freely.

3. Differences Between Single and Multi-Agent

Multi-agent architectures face "shortcoming" issues:

• Context interruption between main and sub-agents
• Context interruption between sub-agents

3.1 Main-Sub Agent Context Interruption

When the main agent executes first and then delegates to sub-agents, the sub-agent opens a new context window — creating context interruption because the architecture didn't plan for context transfer.

The solution: provide key context from the main agent when assigning tasks.

A critical remaining issue: parallel execution causes context isolation — Sub-Agent A can't know what Sub-Agent B is doing.

3.2 Context Interruption Between Sub-Agents

Consider a document generation task: Sub-Agent A generates art domain content, Sub-Agent B generates music domain content. Results might show:

1. Format inconsistency: A generates Markdown, B generates HTML — merge fails
2. Completely different perspectives: content feels disconnected and fragmented

This is a parallelism problem — no good solution exists without architecture changes. Switching to single-agent architecture would solve these issues.

3.3 Context Management Comparison

Key differences:

1. Context management: Single-agent has coherent context; multi-agent risks isolation during parallel execution
2. Execution coordination: Single-agent needs none; multi-agent's greatest challenge is coordinating multiple sub-agents
3. Technical development: Single-agent is simpler to develop; multi-agent has maintenance and testing advantages due to modularity

4. Single-Agent Advantages

When agents need shared context or heavy inter-dependency, single-agent is most suitable.

Single-agent has lower development cost — mostly just context compression strategy needed. The focus is what context to collect and how.

If you're not certain you need multi-agent, build single-agent first — start simple, add complexity only when needed.

Single-agent has greater advantages in "write" operations:

In 2024, many models performed poorly at editing code. The common pattern was "edit application model" — a small model rewrites entire files based on markdown instructions from a large model. Today, edit decisions and execution are typically handled by a single model in one action.

5. Multi-Agent Advantages

Multi-agent systems excel at tasks involving heavy parallelization, information exceeding single context windows, and complex tool interactions.

5.1 Advantages in "Read" Operations

Multi-agent has greater advantages in reading operations. Research tasks are the most suitable scenario:

Research tasks essentially require flexibility to pivot and explore side connections. The most important aspect is search, and search is essentially compression — distilling effective insights from massive corpora.

Benefits of multi-agent search:

1. Multiple sub-agents provide more possibilities, perspectives, and insights
2. With sufficient, isolated context windows, sub-agents can simultaneously pursue multiple independent directions

The coordinator is most critical — requiring stronger model capabilities. Claude's team found that using Opus 4 as lead with Sonnet 4 as sub-agents outperformed single-agent Opus 4 by 90.2%.

5.2 Value in Complex Tool Interactions

Context confusion degrades model performance in tool selection.

A recent paper evaluated small models on GeoEngine benchmark with 46 different tools. A quantized Llama 3.1 8b failed with all 46 tools but succeeded with only 19, despite fitting within the 16k context window.

The RAG MCP paper noted:

When tool count exceeds 30, descriptions start overlapping and causing confusion. Beyond 100 tools, models almost certainly fail. Using RAG to select fewer than 30 tools significantly shortens prompts and triples tool selection accuracy.

Multi-agent systems enable context isolation, providing domain-specific tools to specialized sub-agents, dramatically improving tool selection success rates.

5.3 Multi-Agent Improvement Methods

Common issues and solutions:

1. Think like your agent — observe each decision step, find logic deviations, optimize prompts
2. Clear sub-task descriptions — each sub-agent needs: objective, output format, tools, source guidance
3. Scale work to query complexity: Simple (1 agent, 3-10 tool calls), Medium (2-4 agents, 10-15 calls), Complex (10+ agents, unlimited calls)
4. Prioritize tool design and selection
5. Use agents for self-improvement — learn from failed outputs and tool errors
6. Start broad, then focus — expert-level human research strategy
7. Guide reasoning process in prompts
8. Parallel tool calling for speed and performance

6. Progressive Building Strategy

The reasonable strategy: Build small-module single-agents → Build multi-agent systems → Upgrade to complete single-agent

1. Start with single-agent development for prototypes and small modules
2. As individual modules prove effective, develop single-agents for more system nodes, gradually forming a multi-agent system
3. When many modules are effectively replaced and organically combined, upgrade to a complete single-agent

Advantages of starting with small modules:

1. Manageable context: smaller windows mean better LLM performance
2. Clear responsibilities: each agent has defined scope
3. Higher reliability: less likely to get lost in complex business logic
4. Simpler testing: easier to test specific functions
5. More efficient debugging: easier to identify problems

This approach leverages technical iteration: as LLMs become smarter, our building direction stays stable while iteration speed and system effectiveness improve significantly.

Keep reasonable intent in agent size and scope, and only expand in ways that maintain quality. As the Notebook team said:

The most magical moments in AI building come when you're really close to the edge of model capability.

Wherever that edge is, if you can find it and consistently ride it, you can build magical experiences. There are many moats to build, but as always, they require engineering rigor.

[!IMPORTANT] This article focuses on compression dispatch — determining which context to pass to the LLM for compression. This is at the code design level, preceding the compression prompt phase.

The current compression mechanism has two main strategies: tool output trimming/compression and session history compression.

[!NOTE] The context types primarily operated on:

• Tool input/output context
• Session history context

Before each context injection to the LLM, a check verifies whether the current context exceeds 90-95% of the LLM's maximum context length. This splits into pre-check processing and post-check processing:

1. Pre-check processing: Retain key parts of tool output, avoid redundancy:
   1. Limit maximum content size — e.g., read tool limits on max lines and characters
   2. Layered reading: When exceeding max lines, use layered strategy — read some from the beginning, middle, and end
   3. LLM summarization: When files exceed 2,000 characters, use LLM to summarize and return only the summary
   4. Progressive reading: Following Skill design principles — "coarse" read first, then "fine" read
2. Post-check processing: When the Agent has been looping and tool outputs are already "healthy" but context still fails the check, consider compressing session history

1. Pre-Processing — Tool Output Trimming and Compression

Tool output has two layers of judgment:

1. First layer: Whether tool output exceeds 100,000 characters — if so, truncate
2. Second layer: Each tool's output should not exceed 2,000 characters — when exceeded, invoke LLM summarization

For the second layer, several approaches to consider:

1. Output only the LLM summary
2. Output the first 2,000 characters + LLM summary
3. Skip LLM summarization entirely — truncate based on file type

2. Fallback — Session History Compression

Two approaches for session history compression:

1. LLM compression: Convenient and fast — the prompt is key
2. Tool message trimming: In context, tool-type messages have the highest token proportion — prioritize trimming tool messages from the first 70% of history

[!TIP] Following Cursor's approach: when providing the summary to the Agent, also provide a history file location or index. If the Agent finds it needs more details not included in the summary, it can search history to retrieve that information.

Compression means summarizing conversations approaching context window limits and reinitializing a new context window. The core is distilling key content so the Agent can continue with minimal performance degradation.

[!IMPORTANT] This article focuses on compression prompts — the instructions telling the LLM how to compress and what key information to retain. This follows the compression dispatch phase.

References:

• ClaudeCode reverse engineering: https://github.com/shareAI-lab/analysis_claude_code
• gemini-cli: https://github.com/google-gemini/gemini-cli
• "Effective context engineering for AI agents": https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
• "Managing context on the Claude Developer Platform": https://www.anthropic.com/news/context-management

1. LLM Compression — ClaudeCode's Prompt

Claude's team shared that ClaudeCode directly uses the model for summarization:

In Claude Code, we implement this by passing the message history to the model to summarize and compress the most critical details. The model preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages.

The /compact command prompt instructs the model to create a detailed summary covering these 8 sections:

1. Primary Requests and Intent: All explicit user requests
2. Key Technical Concepts: Important technologies, frameworks discussed
3. Files and Code Sections: Specific files examined, modified, or created — with code snippets
4. Errors and Fixes: All errors encountered and how they were fixed
5. Problem Solving: Resolved issues and ongoing troubleshooting
6. All User Messages: Non-tool user messages for understanding intent changes
7. Pending Tasks: Outstanding tasks explicitly requested
8. Current Work: Specific work in progress before the summary request
9. Optional Next Steps: Next steps related to recent work

The prompt uses XML format (Claude models are trained extensively on XML tags) and an <analysis> block for structured thinking before the final summary.

Why these 8 directions?

1. Technical Context: Rebuilding the development environment
2. Project Overview: Understanding global architecture
3. Code Changes: Recording specific work outputs
4. Debugging & Issues: Avoiding repeating mistakes
5. Current Status: Tracking task progress
6. Pending Tasks: Maintaining task continuity
7. User Preferences: Working memory about the project
8. Key Decisions: Preserving decision history

After LLM generates the summary, add an opening statement: "Context has been compressed using structured 8-section algorithm. All essential information has been preserved for seamless continuation."

2. LLM Compression — Gemini's Prompt

Gemini-cli also uses LLM summarization but differs in key information retention and invocation:

1. Only 5 key information categories
2. Uses "scratchpad" chain-of-thought to enhance extraction

Gemini's prompt instructs the model to first think in a private <scratchpad>, then generate a <state_snapshot> XML with:

1. Overall Goal: User's high-level objective
2. Key Knowledge: Critical facts, conventions, and constraints
3. File System State: Files created, read, modified, or deleted
4. Recent Actions: Summary of recent agent operations and results
5. Current Plan: Step-by-step plan with completion markers

3. Context Compression — Tool Message Trimming

Instead of LLM-based compression, this strategy cleans tool inputs and outputs directly:

Context editing automatically clears stale tool calls and results from within the context window when approaching token limits, effectively extending how long agents can run without manual intervention.

In most Agent interactions, tool outputs (especially read tools) consume the most context. User inputs and model outputs are relatively small. Prioritizing tool-related context removal is reasonable.

Implementation approach:

• Filter tool inputs and outputs from history
• Decide whether to remove all or keep the last N tool call rounds
• Produce optimized context

The code identifies "tool rounds" (assistant message with tool_calls + corresponding tool result messages), keeps the most recent N rounds, and removes the rest while preserving all non-tool messages.

Following Cursor's approach: when providing summaries, also provide a history file location so the Agent can search for details not included in the summary.

4. Context Compression — Middle vs Oldest Strategy Selection

An elegant compression approach that uses algorithmic judgment rather than LLM compression — more controllable but more complex to develop.

Three removal strategies:

1. Middle removal: Keep beginning and end, remove middle messages
2. Oldest removal: Prioritize removing oldest messages, keep newer ones
3. Hybrid: Intelligently combine both strategies

4.1 Strategy Selection Method

Three-layer selection:

1. First layer: Based on provider/model
2. Second layer: Based on conversation characteristics
3. Third layer: Confidence judgment

4.2 Provider/Model-Based Selection

4.3 Conversation Characteristics-Based Selection

Only triggered when the first step outputs "hybrid." Analyzes: total messages, average message length, compression ratio, recent message token ratio, presence of long/system/tool messages, and compression severity (light >80%, moderate >60%, heavy ≤60%).

Rules:

1. Light compression + short conversation → Middle removal (confidence: 0.8)
2. Heavy compression + long conversation → Oldest removal (confidence: 0.9)
3. High recent message ratio → Middle removal (confidence: 0.7)
4. Long messages + significant compression → Oldest removal (confidence: 0.6)
5. Tool or system messages present → Middle removal (confidence: 0.7)

4.4 Adaptive Strategy Selection

When confidence drops below 0.6, the system runs both strategies and calculates efficiency scores:

Efficiency = Token reduction (60% weight) + Message preservation (40% weight)

Example: 15 messages, 9000 tokens, target 6000:

• Middle removal: 6200 tokens, 12 messages kept → efficiency 0.5066
• Oldest removal: 5800 tokens, 10 messages kept → efficiency 0.4804

Middle removal wins despite less token reduction, because it preserves more messages. The system balances both objectives to select the optimal strategy.

• Claude Cookbooks: https://github.com/anthropics/claude-cookbooks
• LangFuse documentation: https://langfuse.com/docs/evaluation/overview
• promptfoo framework: https://github.com/promptfoo/promptfoo

1. Why Agent Evaluation Matters

LLM output has uncontrollable factors. For production-grade LLM applications, stability is paramount. Mature evaluation frameworks not only make applications more stable but also reveal model potential and boundaries for better iteration.

Key quotes:

1. Teams' inability to effectively evaluate model performance is the biggest barrier to LLM production use cases, turning prompt design into art rather than science.
2. Although evaluation takes significant time, doing it upfront saves developers time long-term and enables better products to ship faster.

Evaluation benefits for LLM application development — quantifying model boundary capabilities:

1. Iterative prompt optimization: Is our V2 prompt better than V1?
2. Pre/post-deployment quality assurance: Did our latest prompt update cause performance degradation?
3. Objective model comparison: When switching to a more advanced model, can we maintain or improve evaluation performance?
4. Potential cost savings: When switching to a faster, cheaper model, can we maintain evaluation performance?

2. Evaluation Components

A well-designed evaluation framework has four main components:

1. Example inputs: Instructions or questions for the model — key is designing inputs that accurately represent real-world usage scenarios
2. Gold standard answers: Correct or ideal responses as baselines — creating high-quality standards often requires domain expert involvement
3. Model output: What the LLM actually generates based on example inputs
4. Score: A quantitative or qualitative value representing model performance on that specific input

At least 100+ sets of example inputs and gold standard answers are needed for meaningful evaluation.

3. Evaluation Workflow

Step-by-step:

1. Prepare test cases (example inputs + gold standard answers)
2. Split into two batches: 80% development set, 20% holdout set
3. Design first-version prompt based on intuition
4. Test on development set
5. If results are poor, optimize prompt based on test results — iterate until satisfactory
6. Test against the holdout set to verify generalization
7. If the gap between holdout and development results is within ~10%, it passes
8. If the gap exceeds 10%, the prompt is overfitting to the development set — return to prompt optimization

Two additional evaluation factors beyond accuracy:

• Edge case coverage: Model performance on extreme inputs
• Performance testing: Model response time

The two most important aspects:

1. Writing evaluation questions and gold standards: Time-consuming but one-time cost, reusable
2. Ongoing scoring costs: Frequent evaluation runs incur model costs if using LLM scoring — build fast and economical evaluation systems

4. Evaluation Methods

Three common methods:

• Code-based scoring: Standard code to match and judge model output
• Human scoring: Manual review and scoring
• Model-based scoring: Another LLM evaluates the output

Prioritize model and code-based scoring over human scoring, which is more expensive and slower.

4.1 Code-Based Scoring

Characteristics: Programmatic approach for tasks with clear, objective criteria.

Advantages: Speed and scalability — can consistently process thousands of evaluations. Limited ability to handle nuance or subjectivity.

Forms:

1. Exact string matching: Output must exactly match gold standard
2. Keyword presence checking: Whether output contains certain key words/phrases
3. Regular expressions: Check complex text patterns

Example

Using the relatively weak Qwen2-7B-Instruct model to demonstrate the full evaluation flow with a sentiment analysis task:

Step 1 — Prepare evaluation dataset:

let testCases = [
  { id: 1, text: 'Amazing! Very satisfied, five stars!', expected: 'positive', reason: 'Clear positive words' },
  { id: 2, text: 'Fast shipping, quality exceeds expectations', expected: 'positive', reason: 'Multiple positive descriptions' },
  { id: 3, text: 'Garbage product, completely unusable, got refund', expected: 'negative', reason: 'Clear negative words' },
  // ... more test cases
];

Step 2 — First-version prompt:

let promptV1 = (text: string) => `
Determine the sentiment of the following text, answer "positive", "negative", or "neutral".
Text: ${text}
Answer with one word only.
`;

Step 3 — Run and score: Accuracy 83.33% (2 failures)

Step 4 — Optimize prompt with rules for sarcasm detection and transition sentence handling

Step 5 — Run optimized prompt: Accuracy 100%

Step 6 — Proceed to holdout set testing or pass evaluation

4.2 Human-Based Scoring

Characteristics: Gold standard for tasks requiring nuanced understanding or subjective judgment.

Advantages: Excels at evaluating tone, creativity, complex reasoning, and factuality. Disadvantages: Time-consuming, potentially expensive at scale, and susceptible to inter-rater inconsistency.

Forms:

1. Expert review: Domain experts evaluate accuracy and depth
2. User experience panels: Groups evaluate clarity, helpfulness, and engagement

4.3 Model-Based Scoring

Characteristics: Falls between code-based and human-based methods, using another LLM to evaluate output.

Advantages: Handles more complex and subjective evaluations than code-based scoring while being faster and more scalable than human scoring. Requires strong prompt engineering for reliable results, with risk of the scoring LLM introducing its own biases.

Writing Evaluation Model Prompts

Core elements:

• Original prompt or question
• Model output to evaluate
• Evaluation criteria or guidelines
• Instructions for how to evaluate and score

Common evaluation criteria:

1. How apologetic is this response?
2. Is the response factually accurate given the context?
3. Does the response excessively reference its own context?
4. Does it truly answer the question appropriately?
5. How well does it align with our tone/brand/style guidelines?

Evaluation Model Positioning

A qualified evaluation model must maintain objectivity. Most models default to being friendly with apologetic tendencies.

Add perspective constraints to the evaluation prompt: "Do not apologize or use apologetic language. Be objective and neutral."

Evaluation Example

Step 1 — User input: "Write a slogan for a children's toy store"

Step 2 — Tested model output: "Our store offers high-quality toys, welcome to purchase."

Step 3 — Evaluation LLM scores against criteria: child-friendly, lively tone, total score 10

Step 4 — Result: 4/10. Reason: Tone is too formal and serious, doesn't match the lively positioning of a children's toy store.

1. Discovery: Where are your Skills stored — local or cloud? What's the priority between project-level and user-level? How do you determine that a folder is a Skill?
2. Parsing: Extract the metadata from SKILL.md
3. Usage: How does the Agent use the parsed metadata — system prompt or tool description? The subsequent progressive disclosure strategy — use a read tool or an internal activation tool?
4. Management: How to maintain the validity of Skills loaded into context? Should Skill content be protected during context compression?

The core development principle is also the core feature of Skills: Progressive Disclosure

• First-level disclosure: At session startup, load metadata (name + description) into context
• Second-level disclosure: When a Skill is activated — i.e., the Agent matches user input to a Skill's metadata — load the complete SKILL.md content into context
• Third-level disclosure: After the full SKILL.md content is loaded, if the task complexity is high, the Agent loads more detailed guidance — scripts, references, and static assets — on demand

1. Discovery

The Agent needs to discover which Skills are available in the runtime environment from corresponding file directories. Most Agents run locally, so we'll focus on local Skill discovery.

Skill directory scope is divided into two types: user global scope and project local scope

• Project local scope: Skills that only apply to the current project, e.g., frontend design Skill, React best practices Skill
• User global scope: Skills that apply to all user projects, e.g., find-skills, pptx (PPT generation Skill)

The specific Skill directory paths are:

1. <project>/.<agent-client>/skills/: Project-scoped Skills
2. <project>/.agents/skills/
3. ~/.<agent-client>/skills/: Global-scoped Skills
4. ~/.agents/skills/

You'll notice the /.agents/ path exists because it has become a widely adopted convention for sharing skills across different clients. For example:

Someone might develop a-agent-client with Skills in ~/.a-agent-client/skills/. When you develop b-agent-client and want to use a's Skills, you'd need compatibility — reading specific directories like ~/.a-agent-client. One is fine, but what about many with different paths? So everyone adopted a convention: regardless of the client, all installed Skills can be provided under ~/.agents/skills/, enabling cross-client compatibility.

Reading from /.agents/ means Skills installed by other compliant clients are automatically visible to yours, and vice versa.

When the same Skill appears in both user global and project local scope, the priority is:

• Project-level takes priority over user-level
• Within the same scope, priority follows discovery order

Trust checking is designed because some pulled projects may contain potentially malicious or insecure Skills that could leak secrets.

So in the Agent's configuration file, you can design a Skill trust check layer. Within project scope, only trusted Skills can be loaded — this is determined at the business logic level.

Development tips:

• Skip directories that don't contain Skills, e.g., node_modules/
• Consider respecting the project's .gitignore file, avoiding scanning build artifacts like dist/
• Don't recursively search endlessly — set a maximum depth (5-6 levels) and maximum file count

2. Parsing

At the Agent session start phase, Skill metadata needs to be loaded into context. So after getting the Skill file paths, the next step is parsing SKILL.md metadata.

A SKILL.md file contains two parts:

1. YAML front matter separated by --- delimiters
2. MD-formatted content after the closing delimiter

The metadata fields and constraints are:

• name (required): Skill name, max 64 characters, lowercase letters, numbers, and hyphens only
• description (required): Skill description, max 1024 characters
• license (optional): License name or file reference
• compatibility (optional): Environment requirements
• metadata (optional): Additional metadata
• allowed-tools (optional): List of tools the Skill is allowed to execute

Detailed development steps:

1. Find the start and end delimiters in the SKILL.md file
2. Parse the YAML block, extracting name, description, and other optional fields
3. The MD content after the closing delimiter is the SKILL.md body

When parsing YAML, don't be too strict with error handling — minor errors shouldn't prevent Skill parsing. Return error info as warnings to the user.

The parsed metadata needs to be loaded into the Agent's context. We can use a Map format, storing metadata in memory with name as key. The value needs at least three fields:

• name
• description
• location: absolute path to the SKILL.md file

3. Usage

3.1 Placing Metadata in System Prompt

1. At session start, place metadata in the system prompt with a brief Skill usage instruction telling the model how and when to use it — first-level disclosure
2. When user input triggers a matching Skill, the Agent calls a read tool to load the complete SKILL.md — second-level disclosure
3. After the full SKILL.md is loaded, if the Agent needs more detailed guidance, it reads references, assets, and scripts from paths in the SKILL.md — third-level disclosure
4. If scripts need execution, the Agent calls the Bash tool

Use structured formats (XML, JSON, etc.) when placing metadata in the system prompt — this is very effective for context management.

<available_skills>
  <skill>
    <name>code-review</name>
    <description>Review code for bugs, style issues, and best practices. Use when the user wants feedback on their code.</description>
    <location>/home/user/.agents/skills/code-review/SKILL.md</location>
  </skill>
</available_skills>

The location field serves two purposes:

1. Provide the correct path parameter for the read tool
2. Give the model a base path reference for correctly reading referenced resources

3.2 Placing Metadata in a Dedicated Tool

1. At session start, Skill metadata is provided in tool description format — first-level disclosure
2. When the Agent needs complete SKILL.md content, it passes the name as a parameter and the tool returns the content — second-level disclosure
3. In third-level disclosure, the Agent uses read tools to get full reference resource content
4. The Bash tool can execute Skill scripts

Using dedicated tools vs system prompts — dedicated tools have advantages in development control:

• Control returned content — only return body content, not duplicate metadata
• During context management, tool results can be specially marked to skip compression
• Reference resources are presented in structured list format, more model-friendly
• Enable unique control flows
• Enable statistics and analytics

<skill_content name="code-review">
# Code Review

## When to Use
Use this skill when users need code review, including bug detection,
code style checking, and best practice suggestions.

## Review Steps
1. Read the code and understand its intent
2. Check style issues against references/style-guide.md
3. Run scripts/lint-check.sh for static analysis
4. Output a structured review report

Skill directory: /home/user/.agents/skills/code-review

<skill_resources>
  <file>scripts/lint-check.sh</file>
  <file>references/style-guide.md</file>
  <file>references/common-bugs.md</file>
</skill_resources>
</skill_content>

For skill_resources, instead of relying on SKILL.md to describe references, as a tool result we can read all reference, scripts, and asset folders under the skill, then include file paths in the skill_resources tag. Limit the file list size and signal to the model that "the current list may be incomplete" so the model isn't constrained during autonomous execution.

4. Management

The purpose of progressive loading is to avoid preloading all Skills, since some might not be used initially — loading them would waste context.

However, already-loaded Skill content is worth keeping throughout the session, since the Skill has become behavioral guidance for the Agent. Blindly compressing it would degrade performance. Two approaches to retain Skill content:

1. System prompt approach: Identify structured tags in read tool results to preserve Skill content
2. Dedicated tool approach: Mark skill activation tool output as protected, checking the mark during compression

This depends on the situation — if conversations are long with many Skills and a small context window, compression is the better choice.

After compression, explicitly prompt the model: "Skills were also compressed — reload them if needed." Otherwise the model may fall into compression hallucination, believing it still has the Skill loaded.

A more advanced approach: running Skills with a subAgent

The subAgent's entire execution (Skill instructions + reference files + intermediate reasoning) happens in its own context window, leaving the main Agent's context completely unpolluted.

Whether to use a subAgent for Skill execution should be the main Agent's decision, e.g., automatically delegating when task complexity exceeds a threshold.

5. Quick Build

The above describes building Skill support from scratch. The current ecosystem is mature — many Agent SDKs support Skills out of the box.

Using Claude Agent SDK as an example:

1. Agent runtime file:

"""Simplified version showing core flow steps"""
async def _run_repl_async() -> None:
    config = load_runtime_config()
    apply_runtime_env(config)
    client = build_client(config)

    try:
        user_input = input("\n> ").strip()
    except (EOFError, KeyboardInterrupt):
        return print("\nGoodbye!")

    try:
        print("[Sending request...]")
        await client.query(user_input)
        print("[Receiving response...]")
        events = [e async for e in _iter_events(client)]
    except Exception as exc:
        print(f"[Error] {exc}")

1. Core SDK configuration:

"""SDK wrapper - simplified"""
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions

ALLOWED_TOOLS = ["Skill", "Read", "Write", "Edit", "Bash", "Grep", "Glob"]

def build_client(config):
    options = ClaudeAgentOptions(
        setting_sources=["project", "user"],
        allowed_tools=ALLOWED_TOOLS,
        model="deepseek-chat",
        env={
            "ANTHROPIC_AUTH_TOKEN": config.api_key,
            "ANTHROPIC_BASE_URL": config.base_url,
        },
        add_dirs=["/xxx/xxxx/.agents/skills"]
    )
    return ClaudeSDKClient(options=options)

Similar SDKs include pi-mono's pi-coding-agent core package and kimi Agent SDK.

If speed is the priority, using mature Agent SDKs for Skill support is perfectly viable. Your overall Agent design then needs to align with these SDKs — there are pros and cons that developers should weigh based on their scenarios.

• Building from scratch: higher freedom, more customizable
• Using SDKs: faster speed, easier iteration, lower development difficulty