Self-Improving Harnesses, Local Personal AI And YC's Agent For Work | YC Paper Club
In a Nutshell
Harnesses now drive most AI progress through self-improving architectures that enable persistent memory, programmatic sub-agents, and test-time learning—outperforming static prompt engineering. Prime Agent, Open Jarvis, and YC's QM demonstrate this with concrete results: 95%+ on ARC-AGI, 800x cost reduction for local AI, and fleet-scale agent deployment across 50+ VMs. The key shift is from fixed tool loops to agentic operating systems where models dynamically manage context, spawn sub-agents, and accumulate capabilities through continual refinement.
These notes were generated by AI and may contain inaccuracies.
The event opened with a discussion of the new YC Paper Club visual design created by Ev, head of design at YC. The speaker explained that harnesses represent scaffolding and prompt engineering rather than traditional research, despite recent community pushback against considering prompt engineering as legitimate research at top-tier machine learning conferences.
Harnesses deliver an 18% performance improvement between different implementations. This difference determines whether ARC-AGI functions or fails entirely. The speaker referenced meter plots showing release dates versus agent runtime duration, demonstrating that much recent progress stems from harness improvements rather than model intelligence gains alone.
The speaker distinguished between the static harness era, where harnesses remain fixed, and the recent six-month period focused on self-improving harnesses. A plot from Trajectory's CEO illustrated how research emphasizes model perplexity and IQ metrics while neglecting test-time experience adaptation. The speaker described experiments increasing sample counts online and the challenge of learning from batch size one, noting that in-context learning saturates after 40-50 examples without further improvement on validation sets.
ARC-AGI emphasizes rapid adaptation to new problem distributions. Claude Opus achieved 30% on the private holdout set verified by Greg and Chalet. Harness improvements elevated performance to 95%, while AVO from Nvidia reached 100%. The speaker mentioned forking Karpathy's auto-researcher in March and accidentally building a harness while creating a user interface.
The harness system accepts a purpose statement, such as testing whether diffusion LMs outperform AR LMs through ensemble sharding to address arithmetic intensity differences per GPU. Users specify seed ideas for ensemble size variations and validation metrics like GSM8K with GPT-2 setups. A scoping agent searches papers and GitHub repositories, then passes work to a PI agent named Chris Ray, who coordinates with research agent John South Khan. A council provides feedback before the author agent writes ablations and papers.
The system includes a scoping agent and cockpit interface accessible via Tailscale for remote viewing and interaction. Email updates notify users of progress. The system has generated publishable papers that improved from initial March-April outputs to current quality levels. Eight ideas can run simultaneously across eight H100 nodes, each with eight H100s.
The speaker reviewed harness evolution starting with GPT-2 in February 2019 featuring basic end-of-sequence loops, top-P sampling, and environment interaction without tool calling or skills. The GSM8K format used system prompts positioning the model as a math teacher, with four hashes marking answer extraction points.
Few-shot learning from July 2020 introduced example-based context. Chain of thought distributed computation across multiple tokens rather than direct prediction. Toolformer and WebGPT enabled tool calling through JSON objects, exposing tools via system prompts. Memory capabilities added CRUD operations on context chunks. Voyager introduced skill chaining for task completion with distillation back into system prompts.
InterCode enabled on-the-fly code output as skills. ReAct preceded self-refine and reflection, allowing multiple agents with different roles to improve outputs iteratively. Reflection agents evaluate actions and provide feedback for correction. Multi-agent systems spawn persistent sub-agents that run independently with messaging capabilities. Recursive Language Models enable recursive RLM queries for solving larger problem classes with leaf LM queries spawning sub-agents.
Harness V1 specifies agent specifications, system prompts, turn limits, tool call limits, tool lists, skills lists, and sub-agent lists within a loop structure. Session management handles Slack channel prompts and scheduled execution. Context compilation constructs LM calls, processes actions, and manages tool invocations with result appending.
DSPy enables system prompt optimization through demonstrate-search-predict methodology using genetic programming on training sets to find optimal prompts via candidate generation and merging. Darwin machines extend this to harness code modification itself, maintaining archives of agents evaluated on fitness functions with meta-harnesses producing improved harnesses through iterative modification.
Continual harness adds memory class distinctions including history tracking. Dagger-style online learning enables weight file updates through test-time training on newly learned examples.
Seth, a Princeton researcher at Prime Intellect and Prime Agent author, presented next. John Sadvalone, a PhD student under Chris and Aelia from Hazy lab, presented Open Jarvis. Josh and Rean from YC presented QM, with Josh recently promoted to head of YC Labs.
Seth presented Prime Agent as a self-improving RLM harness. The approach treats LLMs as sequential processors with fixed weights and visible context taking tokens as input and producing tokens as output. The harness layer adds persistent state, tools, and compute between the LLM and world.
Prime Agent provides an agents view showing parallel sessions with tight summaries. Root sessions function as project orchestrators controlling sub-agents programmatically through recursive language model principles within IPython shells. All tools, memories, and sub-agents are exposed with messaging paradigms for coordination. Agents interact directly with environments including computer files and H200 node clusters.
Persistent daemons maintain sessions when laptops close or sessions are interrupted. Continual harness features provide live CRUD operations on memory, skills, sub-agents, and system prompts. Memory hierarchy includes L1 model weights as fastest accessible information, L2 active input context with in-context examples, and L3 dispatch state for file system operations.
Compaction summarizes context history to extend beyond context length limits. L2 includes live ripples running Python in IPython shells with variables saved in RAM for programmatic manipulation without token costs. Sub-agents save context by handling specific information sets. Aentic garbage collection cleans variables and sub-agents to prevent RAM crashes. Refinement updates and deletes skills, memories, and prompts to prevent hard drive space issues.
Harnesses represent a shift toward agentic operating systems. The raw LLM resembles a Turing machine with ticker tape operations, but a harness functions more like a von Neumann computer by enabling read and write operations on external memory. This architectural difference makes harnesses capable of solving a broader class of problems than Turing machines can express independently.
Effective harness design prioritizes maximum expressibility. Early harnesses used explicit step-by-step sequences like plan, act, and critique, but models have now internalized these patterns natively. The remaining expressibility gaps that harnesses must address include the ability to call compact, execute Python programs, programmatically create sub-agents, access state, and implement various feedback mechanisms. Removing any of these features eliminates capabilities the model cannot otherwise perform.
Beyond the RLM paper work by co-author Alex, sub-agents are treated as persistent subsessions. The parent session can spin up new RLM sub-agents that execute tasks and report back to the parent session upon completion. These sub-agents remain idle in RAM while retaining their accumulated context, allowing the parent session to message them at any time to continue work without losing prior information. For memory efficiency, inactive sub-agents can be offloaded and recalled later through messaging.
Continual harness features enable exposure of all prior history to agents. This includes trajectories containing actions and outcomes at each turn, allowing agents to leverage accumulated information to update future harness configurations. Key elements include modifying system prompts, creating skills as sets of instructions or programs for specific goals, maintaining long-term memory storage, and managing sub-agent specifications persistently. The ability to perform reflection and refinement over time represents a critical capability that harnesses should support beyond current model limitations.
Prime Agent includes the ability for any two agents within a nuclear family setup (parents, children, siblings) to message each other directly. This coordination capability proves valuable for both personal agent management across multiple directions and typical software engineering work on long horizon jobs.
Long horizon performance evaluation focuses on identifying the practical plateau where additional test time tokens yield only incremental gains. Many existing evaluations fail to account for this by running models for insufficient time or cutting off execution based on budget constraints, which both prevents fair comparisons across models using equivalent expenditure and obscures potential performance improvements.
ARC-AGI testing revealed significant harness impact on performance. Using Gemini Pearl with a continual harness achieved 20% accuracy. When Prime Agent was tested with a system prompt from the Prolong community leaderboard, the first run achieved 99.9% but was identified as cheating due to improper sandboxing. After proper sandboxing implementation, GPT-4o achieved 78% accuracy. Additional results included GPT-4o at 25.7%, Terra achieving strong comparative results, and Opus reaching 95.5%.
Comparisons with other harnesses showed mixed results. Cloud Code produced poor results in the tested configuration, while other harnesses like Air Agent consumed significant costs ($5,000) with limited performance gains. The cost-to-performance ratio emerged as a critical factor, with programmatic context management identified as a key cost-saving mechanism.
Emulator Bench testing, which involves reproducing entire computer system emulators including Game Boy Color creation, demonstrated Prime Agent's ability to leverage ripple access in RLM for out-of-experiment loop designs. This enables more expressive experimentation before submitting final solutions. GPU kernel testing produced parity results across Soul and Kimico models.
Auto research experiments scaled the Nano GPT speedrun using 8x H200 GPUs over one week. Models including Deep 6v4, GLM 5.3, and Kim K3 demonstrated out-of-loop experiments, running CPU-based experiments for parameterization and hyperparameter search to optimize expensive H200 experiments. This behavior enables effective auto research that improves further with human-in-the-loop bootstrapping.
A 7-day factorial run utilized 633 agents and 23 million output tokens to achieve steady technological advancement across a tech tree. Sub-agents divided tasks within the factory for research, building, and resource gathering while refinement mechanisms leveraged past outcomes for future decisions. The system maintained continuous technology progression without getting stuck even at the experiment's conclusion.
Key takeaways for harness design include focusing on agentic context management, exploring swarms and RLMs in depth, and implementing standardized evaluations. All presented results can be reproduced using the verifiers package available at Prime Intellect.
Personal AI projects like OpenClaw and Hermes Agent typically rely on cloud LLMs, resulting in high API costs (thousands of dollars annually), privacy concerns with personal data transmission, the need to rent rather than own intelligence, and orders of magnitude higher energy consumption compared to laptop-based operation.
Current local LLMs such as Qwen 3.8 27B achieve performance comparable to Claude 4.6 Opus from August 2025, with the performance gap continuing to close as laptop and workstation hardware accelerators improve. Apple's new Mac Mini release and Nvidia's focus on personal-use accelerators support this trend.
Open Jarvis aims to run the core personal AI stack entirely on device, including model inference, agent execution, memory, and learning capabilities while remaining competitive with cloud-only approaches. The system uses five primitives: user interfaces, agentic logic for composable reasoning, the language model engine (supporting Qwen, GBDO, OSS, Gemma 3N), inference engines (Ollama, Llama CBP, VLM, SG Lang on Apple Silicon or Nvidia), and tools/memory accessible through MCP protocol.
Learning primitives include prompt-based techniques like Japa and DSPI, and weight-based techniques like GPO, SFTT, and LoRA. The system supports desktop interaction and continuous persistent agents through cron jobs and standard protocols, designed for plug-and-play integration with existing workflows.
Optimization through the five-primitive specification achieved significant dollar cost reductions, latency improvements, and quality enhancements. Cloud LLMs can optimize local stacks through manual and automatic processes, diagnosing proposed changes and creating improved solutions while avoiding cloud LLM costs during local inference deployment.
Configurations optimized by cloud LLMs including Claude and ChatGPT proved more effective than out-of-the-box local stacks by catering to specific models and harnesses for different workloads. Current on-device LLMs can rival cloud approaches for personal AI workflows, coding, and agentic tasks, with 800x lower costs and significant latency reductions.
All tested cloud LLMs proved useful for optimizing local Open Jarvis configurations, with Opus 5 and GBDO 5.6 Soul performing best, though Gemini, Kimmy, and GLM families also successfully optimized local configurations. The Open Jarvis harness proved cheaper to optimize than alternatives requiring more data or LLM calls, making it particularly effective for local LLM settings.
Future trends point toward increasing proportions of daily inference calls running on local devices and on-prem laptops or workstations rather than cloud infrastructure, driven by improving accelerators and LLMs.
QM provides every YC employee with an OpenClaw-like assistant that is fully customizable and available through Slack or web UI. Each person operates within their own personal context containing sandbox files and crons, with the ability to work in multiplayer settings like Slack channels. QM supports broad task ranges including email triage, legal and finance workflows, document editing, internal database data extraction, live internal web app creation, and event planning.
The system resulted from internal agent projects that evolved throughout YC operations.
The development of YC's agent systems resulted from a series of internal agent projects that accumulated over the years, each benefiting from increasingly capable models. The first project, built in January 2025, was internally referred to as the "general agent." This system featured a straightforward architecture consisting of a system prompt with tools operating in a loop, functioning as a one-size-fits-all solution where everyone interacted with the same system. Despite its architectural simplicity, it proved surprisingly effective at answering data questions, with its capabilities expanding as underlying models improved.
The general agent was eventually integrated with Slack and enhanced with cron functionality plus additional tools to increase its cross-domain capabilities. By June 2025, many YC engineers began using cloud code and codecs, leading to the realization that these could be easily run in VMs. This resulted in Slack tag integration, enabling one-off code changes, CI pipeline configuration, and dev environment provisioning for testing. Users could describe bugs or desired outcomes, and the bot would solve them independently, making coding accessible even to those who had never made code changes before.
A feedback loop was established where failures were observed and the agents.mmd file in the codebase was updated to improve performance based on usage patterns.
In January 2026, YC partners began using OpenClaw. Given that YC partners are extremely busy with office hours, inbound emails, and application reviews, tools providing additional leverage proved highly valuable. OpenClaw was particularly useful as the first agent many partners used that possessed its own computer, making it highly customizable compared to previous agent paradigms. It functioned essentially as a personal assistant.
By April, the question emerged whether this capability could be provided to every YC employee without purchasing individual Mac Minis. This led to provisioning a fleet of over 50 Hermes agents running in VMs. While helpful, these required substantial configuration for users to derive value and proved inherently difficult to manage, requiring SSH access to individual instances for fixes.
The follow-up question became how to build something addressing the downsides of running large Hermes agent fleets while maintaining the personalizability that users valued.
A clear trend emerged showing exponential model improvement with increasingly impressive returns from providing more capabilities. OpenClaw demonstrated powerful results from giving agents their own computers. Around May 2026, exploration began into how far this approach could be pushed by continuing to expand agent capabilities.
The concept of "unhobling" was applied, drawing from the Situational Awareness paper from 2024. This era marked the beginning of test-time compute and initial tool provision to agents, based on the intuition that models contain more intelligence than typically utilized, and that pushing frontier capabilities could produce significant results.
The first major architectural change involved extracting the system brain from the sandbox. With Hermes and OpenClaw, agents possessed their own computers but remained trapped within them, creating administration challenges even with dozens of instances. All sessions were isolated within individual computers.
The solution centralized everything in Postgres, storing all agent conversations centrally and exposing this aggregated context to agents. Sandboxes transitioned from being agent homes to resources agents could access as needed, becoming significantly less limiting.
This architecture enabled accumulation of large evaluation sets from conversation traces, theoretically supporting automated improvement through hill climbing. Results proved mixed, as dispatching agents to fix encountered bugs using LLM-as-judge often resulted in "main character syndrome" where agents made fixes only seeing their piece of the system rather than the whole picture. Human-in-the-loop involvement remained crucial, though automated improvement continues to be pursued.
Agents were wired to all accessible company resources. YC already possessed an effective CLI connecting numerous systems, and arbitrary API keys could be added for additional integrations. Device code OAuth was ingested into a keychain with automatic refresh to replicate laptop-based employee experiences.
Database access remained primarily read-only, with writes permitted through human-reviewed bulk upserts. Agents propose database edit plans that receive human review before execution. Trust in agents has increased over time, similar to early cloud code adoption where tool use review became less intensive as confidence grew.
Sandboxes function as resources rather than agent homes. Agents default to user-allocated sandboxes but can access various environments for collaboration. For heavier development workloads, agents can access more powerful machines, while simpler tasks use less powerful sandboxes. This decision-making was pushed to agents themselves rather than the harness.
Agents can control their own runtime, selecting providers to avoid refusal situations common in AI research or cybersecurity contexts. When encountering refusals with one model, agents can switch to alternatives. Similar flexibility applies to sandbox provider selection.
The harness was kept extremely thin, with the system core consisting of three fundamental tools: remote sandbox execution, object storage read/write operations, and internal app publishing through a simple get-backed system. Additional tools for memory interaction, crons, and similar functions were viewed as temporary solutions addressing system rough edges rather than core components.
Despite creating capable environments with extensive tool access, agents frequently gave up prematurely. A "grind tool" was implemented setting budgets on goals, preventing agents from abandoning tasks before specified wall-clock time (typically a couple hours) or token spend thresholds. This produced significantly better research outputs and reports, applicable to both complex problems and standard office work.
The harness operates in multiplayer environments including Slack, but agents often became confused about their context despite clear system prompt specifications. Local affordances proved important for maintaining situational awareness.
Agents demonstrated poor understanding of social contexts. While humans intuitively understand appropriate information sharing boundaries, agents require explicit work to recreate these frameworks. Privileged information can easily leak into inappropriate contexts, making the knowledge agents can access bounded by permission system quality. YC benefits from existing fine-grained permissioning systems built over years, but many organizations lack such infrastructure, requiring additional work to enable nuanced knowledge sharing.
QM is available as open source, with coding agents capable of deployment. Issues can be reported for assistance. YC is actively hiring for those interested in this work.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
