When to Build Your Own Agent Harness | Harrison Chase, LangChain
In a Nutshell
The decision to build a custom agent harness depends on how far your use case sits from models' training distributions: in-distribution tasks work fine with off-the-shelf harnesses like Claude Code or Codex, while out-of-distribution tasks require custom harnesses that keep in-distribution sub-tasks (e.g., file editing) close to the model layer through model profiles and middleware. Core harnesses share the same loop—LLM calling tools with observations fed back—but differ in their middleware hooks for context management, summarization, sub-agents, and external system integration. Success hinges on a data flywheel powered by evals and observability: track traces, collect feedback via environment or synthetic judges, then iterate on harness code, model fine-tuning, or context updates to compound intelligence over time.
These notes were generated by AI and may contain inaccuracies.
Harnesses are an important topic as many organizations are currently building their own. Harrison Chase, co-founder and CEO of LangChain, was one of the first people thinking about building harnesses around models in the 2022 GPT-3 era. The goal was to move models beyond simple auto-complete tasks to function as virtual collaborators or agents.
When discussing intelligence in the context of agents, LangChain identifies three main parts: the model, context, and harness. Owning your intelligence means controlling all three components.
Model ownership involves the ability to switch between different models to avoid lock-in and use the best available model. Context ownership includes all information the agent uses, such as memory, semantic knowledge, and previous conversations, which help personalize and guide the agent.
The harness is the primary focus, as it orchestrates the model and context together.
The main job of a harness is to bring context to the model at the right point in time. It orchestrates fixed context and dynamic context, bringing them into the model's context window, receiving responses, and processing the output.
Agents need to interact with external systems, which emit more context that feeds back into the agent loop. The harness orchestrates all these interactions together.
The simplest agent architecture is an LLM running in a loop calling tools. A request comes in, the LLM makes a generation that may include a tool call, the tool is invoked, and the observation is passed back to the LLM. This core loop forms the architecture behind virtually every agent today.
While the base loop is simple, harnesses differ in what they do at various stages. LangChain provides a minimal harness, while Deep Agents is a more model-agnostic, general-purpose version that connects to file systems, has skills, and supports sub-agents.
Customization occurs through middleware constructs that allow running code snippets before agent invocation, before each model call, and wrapping model and tool calls. This concept is similar to hooks and plugins used in coding agents.
Through middleware, agents can access sandboxes, file systems, sub-agents, memory, and summarization. Summarization occurs before model invocation when context is too long. Context offloading wraps tool calls to dump large outputs.
Before models could reliably run in loops, people built agents using explicit cognitive architectures with bespoke steps. Examples include deep research systems that generate sub-questions, fan them out, and execute them, and code review bots with specific sequential steps.
Many of these bespoke steps have now been incorporated into the harness as modifications to the core loop. For particular flows, cognitive architectures are still used to guide agents in specific ways.
The recommendation is to start with a general harness for quickest time to value, then add gates and checks as the use case becomes more defined.
Off-the-shelf harnesses like Claude Code, Claude Agent SDK, and Codex work with specific models. The general recommendation is that the more in-distribution your task is with what models were trained on, the better off-the-shelf harnesses will perform.
As tasks move out of distribution, custom harnesses become necessary. However, even within out-of-distribution tasks, certain sub-tasks may remain in distribution. For example, in legal AI, file editing is something models have been RL-trained on, even though the broader legal AI task is out of distribution.
OpenAI and Claude models edit files differently in their harnesses due to different RL training, making each model best at editing files in different ways. A custom harness for legal AI would use the edit file tool that matches the model's distribution.
Deep Agents implements model profiles that switch between different edit file implementations depending on which model is being used. This keeps in-distribution parts as close to the model layer as possible while customizing the overall harness for out-of-distribution tasks.
Evals and observability are essential for understanding what's happening inside agent systems. Three key principles from a recent article by Satya emphasize this importance:
Create your private evals because eval defines what good looks like inside the organization.
Retain ownership of your organization's memory, traces, feedback, decisions, and institutional context.
Create your own continuous learning loop hill climbing machine that will allow your AI investments to compound the value of your firm.
Every company building mission-critical agents will build benchmarks for their domain. Benchmarks can define and catch regressions or enable hill climbing by adjusting the harness or model.
Harbor is becoming the industry standard for defining benchmarks. Created by the makers of Terminal Bench 2, it's an open-source eval runner that works across domains. Frontier Bench is another coding benchmark using this system.
Each Harbor task includes an environment defined in a Dockerfile where the agent runs in a sandbox, a solution used as a golden reference for sanity checking, a test script that serves as the verifier (which can run code, unit tests, LLM judges, or agent judges), and instruction.md containing the prompt given to the agent.
LangSmith, LangChain's platform for evals and observability, integrates with Harbor to track feedback scores, latency, and token usage. When benchmarking agents, accuracy, latency, and cost should all be tracked.
Observability is underrated for agents. When agents fail, it's usually because an LLM call went wrong, either because the model isn't good enough or the context isn't good enough. The second reason causes issues more frequently.
Good observability requires visibility into what enters the model's context window, how context accumulates, what steps were run, what tools were called, and how context gets there.
Trajectories represent the list of messages in agent interactions, similar to what appears in Claude Code. Full traces allow clicking into specific model calls to see internal details.
Evals and observability enable a data flywheel that compounds intelligence. The process involves building an agent, running it, collecting traces, curating trace data, and running experiments on that data.
Feedback can come from the environment or synthetic sources. UX design significantly impacts feedback collection, as clever presentation can elicit feedback without explicit thumbs up/down interactions.
Synthetic feedback uses online evaluators over traces. While running large models like Opus over every trace is expensive, fine-tuned SLMs or off-the-shelf models with custom prompting can provide cheaper alternatives. Code-based evaluation works for simple test cases.
Trace data can update any of the three agent components: harness engineering for the harness, fine-tuning for the model, and memory updates for context.
LangSmith Engine is an agent that sits on top of traces to automate the data flywheel. It creates issue boards by exploring trace data, identifying common issues, and linking to supporting evidence. The system can suggest updates to context, instructions, or harness code.
Engine itself is a coding agent with access to LangSmith CLI for filtering traces by feedback and other criteria. It sends Slack reports about itself for dogfooding purposes.
Issue Bench is a Harbor-formatted benchmark for Engine that enables benchmarking different models and harnesses. Running different harnesses (Deep Agents, Codex, Claude Code) on this benchmark revealed that Codex writes small scripts against traces aggressively, leading to better performance. This learning was incorporated into Engine through what was called codexification.
General-purpose harnesses have become good enough for basic tasks, making it reasonable to start with off-the-shelf options like Deep Agents, Codex, or Claude Code. As tasks become more out of distribution, customization becomes more valuable.
At the extreme end of customization, complete cognitive architectures may be built for predictability and control, particularly relevant for financial services customers.
Organizations requiring predictability often find Deep Agents too uncontrolled and prefer custom cognitive architectures that allow fine-grained control over system behavior. There's a spectrum between fully custom solutions and off-the-shelf harnesses, with middleware and hooks serving as intermediate options. The more out-of-distribution the use case, the more custom harness development becomes necessary.
OpenAI and Anthropic have converged on strong coding capabilities but implemented fundamentally different approaches to file editing. These differences aren't just implementation details—some evaluators consider one approach strictly superior to the other. As model labs continue improving coding capabilities, their harnesses may converge on general coding tasks, yet small but meaningful differences persist that are difficult to explain.
Current differences between model labs manifest in relatively minor areas, but domain specialization could drive significant divergence. If one lab heavily invests in biological applications, their harnesses could become highly optimized for bio-agent tasks, creating specialized capabilities that differ substantially from other providers. This uncertainty highlights why evals and observability remain critical for measuring performance across these evolving systems.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
