← Back to all posts

11 min readUpdated 12 Aug 2026
AI AgentCodexAgent EngineeringHarnessRuntime+2 more

A Real AI Agent Is More Than an LLM: What OpenAI's Codex Harness Teaches Us About Agent Engineering

Yap Wei JunSoftware Engineer · Singapore

When people first hear Build Your Own Agent, a common assumption is:

Do I need to train my own model?

Usually, no.

Once an agent starts editing code, operating a browser, calling MCP tools, running shell commands, managing files, and working continuously for hours, the hard problem changes.

It is no longer only:

How intelligent is the LLM?

It becomes:

How do we place a probabilistic LLM inside an engineering system that is controllable, recoverable, secure, and verifiable?

That is the part of AI development I increasingly think of as Agent Engineering.

AI Agent is more than an LLM

The Model Is Only One Layer of an Agent

A modern agent is closer to this:

Model
+
Harness
+
Runtime
+
Tools
+
Memory
+
Skills
+
State
+
Security
+
Verification

The model provides intelligence.

But intelligence alone does not automatically become reliable action.

An LLM may decide that reading a file, executing a command, opening a browser, or calling an MCP tool would help. But something else still has to decide whether the action is allowed, how it is executed, how state is preserved, what happens if it fails, and how completion is verified.

This leads to an important architectural principle:

The user interface should not be the agent.

A web app, CLI, IDE extension, mobile interface, or desktop application can all be different clients of the same underlying agent system.

The agent should live behind a stable protocol and execution architecture rather than being tightly coupled to one interface.

What Is an Agent Harness?

I think of an Agent Harness as the layer that organizes how an agent uses model intelligence.

A mature harness may be responsible for:

  • goal framing
  • context construction
  • tool discovery
  • tool selection
  • planning and reasoning loops
  • verification strategy
  • continue / stop decisions
  • compaction

A useful simplification is:

Model   = Intelligence
Harness = Direction
Runtime = Control + Execution

Codex-style agent architecture

The harness may propose:

I want to read this file.

or:

I want to execute this command.

But it should not be the final security authority.

That responsibility belongs closer to the runtime.

A concise way to express the boundary is:

Harness proposes. Runtime governs and executes.

This separation becomes increasingly important as agents gain access to real systems.

A Larger Context Window Does Not Automatically Make an Agent Smarter

Modern models can accept enormous context windows, so it is tempting to think:

If it fits, put everything into the prompt.

But excessive context creates its own failure modes:

  • higher token cost
  • higher latency
  • more irrelevant information
  • more stale information
  • more conflicting instructions
  • harder tool selection
  • greater attention dilution

Good context engineering is therefore not about maximizing the amount of information sent to the model.

The better question is:

What is the smallest sufficient context required to complete the current task correctly?

I call this Smallest Sufficient Context.

The goal is not to starve the model of information. The goal is to avoid treating the context window like a storage dump.

Deferred Tools Will Matter More in the MCP Era

Imagine an agent connected to:

GitHub MCP
ERP MCP
KB-MCP
Gmail
Calendar
Browser
Finance
Other MCP servers

That environment can easily expose hundreds of tools.

The simplest but weakest design is to send every tool schema to the model on every turn:

System Prompt
+
400 Tool Schemas
+
Skills
+
Memory
+
Conversation

A better design is closer to:

Tool Registry
      ↓
Tool Search
      ↓
Relevant Tool Schemas
      ↓
Current Context

If the task is only:

Find a Sales Invoice.

then the model does not need to inspect every Gmail, Calendar, finance, and browser tool at the same time.

It can first discover that it needs an ERP invoice capability, then load only the relevant tools.

The benefit is not only lower token usage.

It also reduces the number of choices the model has to make at any one moment.

More tools do not always mean more capability. Without discovery and routing, more tools can also mean more opportunities to select the wrong one.

Skills and Memory Need a Context Budget Too

The same principle applies to skills, memory, project rules, and retrieved evidence.

A mature agent may eventually have:

Hundreds of skills
Hundreds of tools
Thousands of memories
Many project-specific rules

These should not all enter the prompt every time.

Context should be treated as a limited operational resource.

A task-specific context package might conceptually contain:

Instructions
User Goal
Relevant Memory
Retrieved Evidence
Relevant Skills
Relevant Tools
Working Context

The exact percentages are less important than the principle:

Every token entering the working context should have a reason to be there.

Sub-Agents Can Be Treated as Async Tasks

Another useful abstraction is to stop treating every long-running capability as a completely different system.

Consider:

  • a sub-agent
  • a background terminal command
  • a browser job
  • a research job
  • a long-running MCP call

They often share a similar lifecycle:

queued
running
waiting
completed
failed
cancelled

A runtime can therefore model many of these as a common Async Task Handle.

Instead of building a different manager and state machine for every capability, the system can share common primitives for:

  • task identity
  • status
  • cancellation
  • timeout
  • retries
  • progress
  • result retrieval
  • failure handling

This becomes especially valuable for agents that work for minutes or hours rather than a single chat turn.

Browser Agents Should Not Restart the World on Every Step

Suppose an agent needs to:

  1. open a website
  2. log in
  3. enter a dashboard
  4. locate a table
  5. extract information

A wasteful design recreates the browser environment at every step.

A stronger architecture keeps a Persistent Execution Session.

That means preserving things such as:

  • tabs
  • browser state
  • authenticated sessions
  • variables
  • execution context

across turns when appropriate.

This allows the agent to understand the environment first and then operate on it programmatically, instead of repeatedly taking screenshots and rediscovering the same page.

I think this is an important direction for computer-use agents:

Understand the environment, then programmatically control the environment.

Sandbox Is Not the Whole Agent, but It Is a Critical Boundary

Once an agent can:

  • delete files
  • execute shell commands
  • modify Git repositories
  • access the network
  • upload files
  • send email

security cannot rely on a prompt that simply says:

Be careful.

At the opposite extreme, asking the user for permission on every action creates approval fatigue.

A more mature architecture uses several layers:

Tool Call
   ↓
Deterministic Policy
   ↓
Allow / Review / Deny
          ↓
      Risk Evaluation
          ↓
Execute / Human Approval

An LLM may help with risk review, but:

The final security authority should not belong entirely to the LLM.

The runtime should own the enforceable boundary.

That is the difference between a model being instructed not to do something and a system being technically unable to do it without authorization.

“The Model Says It Is Done” Is Not the Same as Completion

This is one of the clearest differences between a normal chatbot and a real agent.

A chatbot often behaves like:

User
 ↓
Model
 ↓
Answer
 ↓
Stop

An agent should behave more like:

Goal
 ↓
Context
 ↓
Reason
 ↓
Act
 ↓
Observe
 ↓
Verify
 ↓
Recover or Complete

Suppose the goal is:

Fix the login bug.

Completion should not mean:

I changed login.js.

A reliable agent should collect evidence such as:

Implementation changed      ✓
Unit tests passed           ✓
Integration test passed     ✓
Browser verification        ✓
Expected behavior observed  ✓
No critical regression      ✓

Only then should the task move into something like:

VERIFIED_COMPLETE

This leads to one of my core principles for agent systems:

Completion must be evidence-backed.

The agent's statement is not the evidence. The system state is.

Compaction Determines Whether an Agent Can Work for a Long Time

If an agent runs for five turns, context management is relatively easy.

If it runs for hundreds of turns, keeping the complete transcript in every model request becomes inefficient and eventually impossible.

Long-running agents need Compaction.

But useful compaction should be more than a vague summary such as:

We changed some code and the work is still in progress.

A better compacted state looks more like:

{
  "goal": "...",
  "completed": [],
  "openTasks": [],
  "decisions": [],
  "constraints": [],
  "artifacts": [],
  "failures": [],
  "approvals": [],
  "currentState": {},
  "nextAction": "...",
  "evidenceRefs": []
}

In other words:

Do not only compact the conversation. Compact the agent state.

The original event log can remain the source of truth, while the working context receives a smaller operational representation.

The Important Part of WebSockets Is Incremental State

As model inference becomes faster, the bottleneck in agent systems may increasingly move elsewhere:

  • API processing
  • network round trips
  • tool execution
  • client-side work

Persistent connections such as WebSockets can help with long-lived sessions and incremental communication.

But the deeper architectural lesson is not:

Every agent must use WebSockets.

It is:

Do not repeatedly transmit state that both sides already know.

Different environments may use different transports:

HTTP
SSE
WebSocket
MessageChannel
postMessage

The architecture should not be tightly coupled to one transport mechanism.

What This Means for AGRUN

I am also exploring AGRUN, a Browser-first, BYOK, provider-neutral JavaScript Agent Runtime + Harness.

Studying Codex Harness reinforces a boundary that I think is important:

               Intelligence
                    ↓
                Harness
                    ↓
                 proposes
                    ↓
                Runtime
                    ↓
             governs + executes
                    ↓
          Browser / MCP / Tools
                    ↓
                 Reality

And conceptually:

Memory   → Harness
Skills   → Harness
Provider → Harness

Policy   → Runtime
Budget   → Runtime
State    → Runtime
Sandbox  → Runtime
Audit    → Runtime

Browser-first does not mean Browser-only.

The browser can remain the primary environment for:

  • UI
  • IndexedDB
  • Web Workers
  • HTTP MCP
  • Web APIs
  • local state

Then an optional Node companion can be introduced only when the agent truly needs capabilities such as:

  • shell access
  • full filesystem access
  • stdio MCP
  • local processes
  • Playwright

This keeps the core architecture lightweight without pretending the browser can safely or cleanly provide every capability.

How I Now Think About Harness Engineering

The simplest possible agent loop might look like this:

while (true) {
  const response = await llm();
  await executeTool(response.tool);
}

But a mature harness needs to manage something closer to:

Goal
 ↓
Frame
 ↓
Context
 ↓
Reason
 ↓
Plan
 ↓
Action
 ↓
Observe
 ↓
Verify
 ↓
Recover
 ↓
Continue / Complete

I increasingly think of this as Controlled Causal Loop Engineering.

For every action X, the system should know as much as possible about the expected result Y:

Action X
   ↓
Expected Result Y

Then it should ask:

  • Did Y actually happen?
  • What evidence proves it?
  • If it did not happen, why?
  • Should the agent retry, recover, roll back, ask for help, or stop?

That is much closer to the engineering required for an agent that can be trusted over long periods of time.

Final Thoughts

My biggest takeaway from studying Codex Harness is not simply that Codex has many advanced features.

It is this:

As models become more capable, competition in Agent Engineering is shifting from model capability alone toward the system capability surrounding the model.

A large part of the quality of future agents will come from:

  • Context Engineering
  • Tool Discovery
  • Skills
  • Memory
  • Runtime
  • Sandbox
  • State Management
  • Long-running Execution
  • Verification
  • Recovery
  • Observability

So if someone asks me:

Does developing your own agent mean training your own LLM?

My answer is:

No.

You can use a model trained by someone else.

What you still need to build is:

A system that turns model intelligence into reliable action.

The model provides Intelligence.

The harness provides Direction.

The runtime provides Control.

Tools connect the agent to Reality.

Verification tells the agent whether the work is actually complete.

That is the part of Agent Engineering I believe is worth studying deeply.

Sources & Further Reading

Note: Codex and Responses API capabilities evolve quickly. This article reflects publicly available information reviewed on 12 August 2026, together with my own architectural interpretation.

← Back to all posts