Engineering Practices for Coding Agents: Lessons from OpenAI and Anthropic

2026-07-18 updated 2026-07-30 4 min

Table of Contents

1. OpenAI’s Practical Experience

OpenAI’s team attempted an experiment: build and ship an internal beta software product with no manually written code.

To accomplish this, the team needed to build a Harness for Codex that could run reliably over the long term. The software engineering team’s primary work was no longer writing code, but designing environments, clarifying intent, and building feedback loops.

This enabled Codex to deliver a million-line project within weeks, already used by hundreds of internal users.

The core components of the coding Agent’s Harness:

Step 1: Three-Layer Code Review

After Codex completes a task, it performs three layers of code review: self-review, local code review Agent, and cloud code review Agent. Only after all reviews pass can it proceed; otherwise, review results are injected back into Codex for revision.

Step 2: Human QA

As Codex’s coding speed increased, the bottleneck became human QA. OpenAI integrated Chrome DevTools Protocol into Codex, enabling DOM snapshot processing, screenshots, and navigation — giving Codex direct UI analysis capability.

Step 3: Log Detection and Performance Optimization

The team also fed runtime logs and performance metrics into Codex, enabling practice -> observe -> modify cycles for performance optimization, rather than relying solely on code structure perception.

Step 4: Code Documentation Library

A codebase’s detailed documentation is massive and can’t be injected all at once. Using the “progressive disclosure” concept from the Skill specification, the documentation is delivered to Codex in a directory-file format.

OpenAI cleverly used AGENTS.md as the documentation index, containing file paths and brief descriptions. Whether to read and what to read is entirely up to Codex, making context utilization highly efficient.

A critical detail: feature requirements often go through team discussions. If this “discussion information” isn’t documented for Codex, this context is missing from the Agent’s runtime environment, potentially causing long-term directional drift.

Step 5: Codebase Structural Rules

Structural rules constrain the codebase to prevent chaos over time. For example, when Codex adds a feature, where to start and which structural layer to consider follows an ordering rule:

Types -> Config -> Repository -> Service -> Runtime -> UI

Validation relies on custom code checkers (also written by Codex).

Summary: The team applied mature software engineering practices to building Agent runtime environments:

Software development still requires rigorous discipline, but that rigor is increasingly about scaffolding rather than the code itself.

Key takeaways:

Provide execution feedback for every step, creating feedback loops
Find specific constraints from general patterns for targeted scenarios
Provide more effective context — currently, “progressive document loading” is the best practice

2. Anthropic’s Practical Experience

When building frameworks for long-running coding agents, Anthropic used a simple two-layer multi-agent architecture: task initialization Agent + coding Agent. As runtime increased and tasks grew complex, two common failure modes emerged:

As the context window fills, the model loses coherence, with some models exhibiting “context anxiety” (especially Sonnet 4.5)
When designing self-evaluation modules, Agents asked to evaluate their own work tend to give confident, high praise

For problem 1, the solution was context reset: completely clearing context (not just compression), starting a new Agent with structured handoff mechanisms.

For problem 2: separate the evaluation Agent from the execution Agent.

The team then built a three-Agent system:

Planner: Takes 1-4 sentence prompts and expands them into full product specs
Generator: Works in loops, executing one sub-task at a time
Evaluator: Uses Playwright MCP to simulate user operations, testing UI, API endpoints, and database state, scoring against criteria

Two core practices:

Before each sub-task, the generator and evaluator negotiate a development contract — agreeing on completion criteria before writing any code
Agents communicate via files — one Agent writes to a file, another reads it

For evaluation criteria, Anthropic transformed subjective judgment into actionable rubrics: instead of “Is this design beautiful?”, ask “Does it meet our design principles?” Their four criteria for frontend design: Design Quality, Originality, Craftsmanship, Functionality.

Transform “vague subjective judgment” into “actionable scoring criteria”.

The final architecture after iteration:

With Opus 4.6, they removed task splitting (Opus can handle full tasks) and development contracts (evaluator directly assesses final output).

Summary: Harness design is not static — as model capabilities improve, the Harness needs additions and removals. An Agent’s optimization isn’t just changing model versions; some tools and modules may become obstacles that need removal.

3. Thinking About Harness from a Development Perspective

Build a review module — Agent output goes through review, with results injected back if review fails. Keep the review Agent separate from the execution Agent.
Use simple, effective message passing between Agents — markdown files work well: writing Agent writes to file, receiving Agent reads it.
Review modules need a “review specification” — criteria for what constitutes passing results. Transform subjective standards into objective ones: focus on “what are the standards for good results” rather than “is this result good?”
Follow the principle of simple and effective Harness construction. Understand it’s dynamic and will continuously adjust with model upgrades. Use Agent evaluation to perceive changes rather than relying solely on intuition.
The current best multi-agent architecture pattern is: Plan - Execute - Evaluate.

Agent Harness Engineering OpenAI Anthropic