Continual Learning: How AI Agents Get Better With Every Use | Arjun Karanam, Trajectory
In a Nutshell
Trajectory's core thesis is that AI agents improve by learning from the interaction data currently being discarded, creating an experience axis orthogonal to intelligence that compounds over time. Their four-pillar approach—traceability of full action trees, specification-based evaluation, reinforcement learning on traces via SDPO, and intelligent harness updates—transforms discarded tokens into training signals that make agents faster, cheaper, and more reliable with continued use. The key practical insight is that feedback must be elicited and categorized: corrective actions like edits and retries provide actionable rewards, while globally relevant patterns train models and user-specific information stays in context.
These notes were generated by AI and may contain inaccuracies.
Arjun Karanam and Ronak are co-founders of Trajectory, a company building a platform for continual learning. Arjun's background includes work with Ronak at One Surf and Train Speed 1, while Michael worked on robotics at DeepMind. The company's mission centers on closing the experience gap in AI systems.
Current AI models excel on the IQ axis, becoming smarter over time, but lack experience. Using the analogy of having Terence Tao in your pocket but him being a poor accountant on day one, the speakers emphasize that experience compounds over time. This creates an orthogonal axis to intelligence that is equally important.
Experience comes from hundreds of millions of tokens generated by agents that are currently discarded. These represent real work being performed with user interactions, yet the data gets thrown away. Trajectory's core thesis is that this interaction data should serve as learning signals, mirroring how humans improve through experience.
Today's agents remain slow, expensive, and error-prone over time, with implementations that don't improve with usage. The vision involves learning agents that become better, faster, and cheaper with continued use, ultimately creating systems that compound in capability over time.
The approach follows four sequential steps:
- Traceability: capturing interactions and experience currently being discarded
- Model specs: defining what agents should do and learning against those specifications
- Model improvement: using algorithms like SDPO and reinforcement learning on full traces
- Harness optimization: determining whether feedback should update models or harnesses based on the nature of the information
Trajectory conducts research on extracting user interactions to generate rewards, converting traces into exact specifications for agent behavior. The SDPO algorithm and similar approaches use reinforcement learning on complete traces to improve models. A key research question involves determining when feedback should update harnesses versus models - for example, factual information like company delisting should remain in harness context rather than being trained into model weights.
The speakers present four wishes to advance the agent ecosystem, organized across the same four categories: traceability, evals, harnesses, and models.
Two specific improvements needed:
- Tracing the entire tree of actions including sub-agents and tool calls, rather than just main actions
- Building products to both capture interaction data and elicit the right feedback
The traditional thumbs up/down feedback proves insufficient and noisy. Corrective behaviors like edits, undos, and retries provide more valuable signals but require explicit elicitation and capture.
Three requirements for better evaluations:
- Evals should closely match actual product usage, drawing from real traffic patterns and frontier requests
- Every task should be replayable so user actions can be reproduced for testing
- Grading should use the real production harness rather than modified versions
Three harness-related improvements:
- Shift from defensive harnesses that prevent bad actions to harnesses that expose primitives (search tools, private information) and let agents orchestrate them
- Make agent interfaces as close to user interfaces as possible, enabling agents to perform any UI action via tool calls
- Make tool responses informative rather than generic confirmations like "done" or "finished"
Two model-related improvements:
- Companies should become comfortable running open-weight models to enable weight ownership and continuous improvement
- Model routers should be experimented with to route intelligence to match task capabilities
Most companies require significant work to implement these improvements. Trajectory conducts audits to assess current state and helps build necessary infrastructure. The company also researches methods to work with incomplete implementations - learning from available signals even when full traceability or evaluations are missing.
Trajectory offers a beta product designed to empower companies to own their own intelligence rather than relying on external consulting. The product enables importing benchmarks like Harvey's lab benchmark, training models through an agent-managed interface that abstracts complex post-training decisions, evaluating model performance, comparing against previous models, and deploying improvements.
When asked about prioritizing between model weights, harnesses, tools, and application layers, Arjun explains that intelligence should be viewed as a complete system. True continual learning optimizes across all components based on what needs updating. The decision of what to update should be abstracted away through scientific solutions rather than requiring manual determination.
Addressing concerns about training on customer data, Arjun notes this was a focus area during his time at Apple. The approach involves sampling distributions from customer data rather than training directly on it, then synthetically generating training data while maintaining distributional properties without exposing actual customer information.
Two categories of feedback signals exist:
- Negative signals without corrective information ("you're really bad," thumbs down, session abandonment) indicate something went wrong but don't specify correct behavior
- Corrective signals (retries leading to correct solutions, explicit corrections) provide clear reward signals
A hierarchy of information pertinence exists: globally relevant information (repeated tool failures) should be trained into models, while user-specific preferences should remain in context. This enables per-organization and per-customer customization.
Continual learning applies across most tasks, but frontier tasks show the most promise. The key pattern involves users attempting capabilities at the edge of current model performance, experiencing failures, then having models improve through training to enable previously impossible tasks. This creates a feedback loop where user attempts at the frontier drive model capability expansion.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
