Building the GitHub for RL Environments: Prime Intellect's Will Brown & Johannes Hagemann
In a Nutshell
Prime Intellect's Will Brown and Johannes Hagemann launched an RL environments hub—a GitHub-like platform for sharing, evaluating, and training reinforcement learning setups—to democratize frontier post-training infrastructure for startups and enterprises. Environments unify evals, synthetic data generation, and RL (e.g., Wordle, Wiki Search, cybersecurity CTFs, SWE-Bench), enabling product-model optimization loops like Cursor's Composer One without big-lab resources. This empowers every AI company to run its own research lab, compounding institutional knowledge via open science and scalable RL for custom agents.
These notes were generated by AI and may contain inaccuracies.
If data is the bottleneck or having real expertise is the bottleneck, would you rather have the smartest person in history work at your company or someone who's been there for 30 years? Sometimes you really want the person who's been there for 30 years. There's a lot of expertise that comes from understanding a problem deeply and interacting with it over a long time. This is what happens in training that is almost impossible to replicate in a short prompt. Institutional knowledge compounds over time, best practices compound over time. This is how institutions and companies grow powerful and successful: they stand on the shoulders of what they've done before rather than resetting every day.
We want this to be accessible to any company. As software becomes easier to manipulate and the barrier to entry for coding lowers, the same will happen for AI research.
Will and Johannes work at Prime Intellect, one of the coolest neolabs in AI. Mission: make frontier lab training accessible to everyone. They have strong taste, developer feel, and intuition for what developers care about. Their launch of the reinforcement learning environments hub was highly differentiated and exciting.
Topics: post-training RL agent harnesses, RL hub, big picture on post-training and RL.
Prime Intellect enables customers to post-train their agents. Post-training means providing frontier infrastructure—currently locked behind big labs—to any startup, enterprise, or neolab. This starts from the compute layer and orchestration, up to the full post-training stack: training frameworks for large-scale reinforcement learning, environments with a community approach via the environment hub, sandboxes for secure code execution, and evaluations as part of the environment hub. Offered as an end-to-end product.
Reasons for the mission: passion for open science as the way humanity moves forward—big scientific discoveries are built upon collaboratively. Practically, value in model customization: winning applications use AI for specific agents or workflows, delivering at scale with cost-effective performance. Optimize systems end-to-end by accessing model weights directly to craft the best model for your problem, beyond prompt-level crafting.
Vision: Every company will be an AI company; most AI companies will want an AI research lab. Research can mean pre-training (for bespoke non-text-in-text-out), or post-training agentic models for specific tasks/workflows. This is becoming productionizable and cost-effective for the right problems.
Lab is a full-stack research platform giving anyone the ability to do what a frontier research lab does internally. Big focus on environments.
Environment in RL: encapsulates what you need for a model to improve via trial and error. More generally: same as evals reported in model releases (e.g., SWE-Bench, AIME, Terminal Bench)—dataset of tasks, harness for the model, rubric or reward function to grade outputs. Use offline eval as test set; in RL, use as train set to improve interactively.
Platform enables environments for post-training, evaluation, synthetic data, reinforcement learning. Community platform like GitHub: share building blocks, examples, collaborative, show off model/tool workflows as environments for others to post-train in, sharing performance improvements.
Gives companies the big labs' advantage: product-model optimization loop—optimize models for specific products. Examples: Claude Code by Anthropic, Cursor's Composer One model optimized in Cursor itself via large-scale post-training.
Big labs have less incentive to optimize for other startups as their products (e.g., Claude Code) grow popular. Prime Intellect provides tools for startups' own optimization loops. Cursor realized early: built Composer One, optimized in Cursor environment.
More startups will optimize current products or build new ones impossible without this loop.
Environments are just evals. Older evals: question-answer banks. Older RL (e.g., Atari): long-running state interaction loops. Now, both are the same: environments for large-scale training have complex state (e.g., simulating web app, full coding platform). Always a reward/goal to measure progress/performance—not just human vibe check.
Environment interacts with system/harness/agent; has goal and measurement.
Cursor example of frontier RL work. When to construct RL environments vs. use actual application states?
Use both. E.g., train Rust coding model good for many applications (domain-task environments); or company-specific tools/domain language as service.
For products as user interfaces (user + agent), product directly becomes environment.
Advanced agent companies already build eval infrastructure (system prompts, model endpoints, cost-performance trade-offs)—same as RL infrastructure. RL is a natural extension for system optimization.
Harnesses relate to RL; harness is a piece of the environment. For any eval/environment task: input → intermediate stuff → output state (graded). Intermediate (simulator, other agent, physical sim) is environment; harness couples model to system pieces.
For coding agents: harness = CLI coding agent, environment = terminal. Varies: system prompt + tools; spawning sub-agents with their harnesses.
Harnesses evolve with Cambrian explosion of model uses. Prime Intellect takes general approach: environment abstraction covers agent/harness/system-model interactions.
Not all companies must post-train with environments—environments are tools beyond RL.
E.g., distill small model from big: run big model in environment, generate data via same protocol, filter best with grader, SFT fine-tune.
Prompt optimization, A/B test models with environments (like evals).
Every AI company should optimize AI systems.
Eval = agent performance in customer-facing environments. In ML terms, using test set/eval to filter back into model (accepting some overfitting). Environments and evals are one and the same; eval is environment for measuring, not training.
RL = iteration at scale: automate changing model/prompt to hill-climb on goal.
Reinforcement learning is the big focus in post-training (most time/compute), but not only: prompt tuning, SFT, online distillation, other algorithms under the umbrella.
Expose full toolkit to unlock potential.
Target: AI engineers in Fortune 500s who follow latest tools, use Claude Code, have model/prompting opinions. They can use the platform, especially with tools abstracting infrastructure (e.g., debug GPU cluster, large-scale agentic RL with tool use/code execution).
Favorite customer: RCI (neolab on frontier open models). Use entire stack: large-scale pre-training (Prime Intellect's history), trained largest mixture of experts models, open-sourced them; also post-training.
Close collaboration forced building compute orchestration, post/pre-training, inference for frontier lab needs. RCI targets enterprises/end-users for end-to-end delivery; Prime Intellect focuses on developer experience at infra layer for seamless idea-to-deployable model.
Environments product: research community (grad students, early-career learners). Domain-specific labs, e.g., medical AI: benchmarks for diagnosis, patient interactions, medical literature Q&A, agentic search. Work with groups like Sant—focus on domain gain, not generic LM infra headache. Create evals, show off, improve models for local deployment (hospitals/end-users), with customization, data provenance, trust.
Atari-style (non-LM): Focus on LMs/foundation models. Examples: games like Wordle (NYT)—great hello world: simple infra, expressive, aha moment seeing model learn with rewards. Sweet spot: runs on couple GPUs in an hour, tiny models learn to win.
Toyish game examples for learning to build RL environments.
People heavily use the environment hub for testing environments due to the infrastructure built around it. They start with simple environments and progress to more complex ones. The reinforcement learning residency started with 8 to 10 people, now 14 to 16, including grad students and full-time workers building environments part-time while doing novel research on the hub. These residents have built environments in various verticals: verifiable software engineering in Lean, medical physics, and cybersecurity. The hub provides tools for training in these environments.
Foundation model companies spend millions on environments. In cybersecurity, capture the flag games feature systems with hidden bugs, originally for programmers in hackathons. These adapt to LLMs: agent lives in a terminal with access to bash commands, tools via cloud code or agent harness wrappers, interacts with files, marks rollout end, then state graded by other software.
Many in data and environment space use RL to evaluate data quality: training on tasks, rubrics, environments reveals bugs like backdoors models exploit to game the system. RL enables vetting from idea to environment in frontier labs' foundational training runs. Medium-scale runs in one environment poke for problems.
Capture the flag examples are toy-like, unlike real corporate networks with security products. Environments can scale to any computer or network complexity. Bottleneck is simulator cost; focus on mocking right system pieces. For web search agents, sometimes use real search, sometimes design tasks avoiding full internet.
Analogy to map games: dark map lights up as explored. Know explorable parts per task to decide importance. T2 Bench (customer service agents with database): mock in-memory database with only expected records, not full millions/billions, for efficiency. Identifying overkill complexity is key to task design.
Platform designed around this: lighting up relevant map parts. For coding agents, handles sandboxes, terminal states, snapshotting, protocols for agent harnesses. Anticipates growing complexity with base generic environments branching to specifics like global/per-rollout sandboxes. Good developer experience: documentation, skill files for agents building environments, surfacing institutional knowledge in product, libraries, docs. Evolves with research literature and best practices.
Reference to Sutton's 'Reinforcement Learning: An Introduction' and Scale AI-era data labeling. Constructing environments is the natural successor. Major labs shifted focus but still use human data, as environments target model weaknesses where humans excel. Key: channel human expert knowledge on job well done into model via RL with LLM judge rubrics, yes/no questions on success turning into reward scores. Most direct: human as reward model.
Hub arose from scattered open-source repos with overflowing implementations. Built Verifiers framework first with environment examples to standardize RL process. Hub enables infrastructure beyond GitHub: integrated evaluations across frontier models for immediate testing, easy installation into trainers. Supports own Primal L large-scale trainer (optimized) and open-source integration with others for varied needs.
Some keep environments private, but hub value in ablating against public similar tasks, uniform implementations of popular benchmarks for training run progress monitoring, generalization checks.
Documentation examples popular, especially WLE (Wordle). Wiki Search (search over Wikipedia pages) sees most forking/branching as template for semantic search over internal documents by swapping docs.
Andre's quote: RL amazing but inefficient, 'sucking bits from a straw.' Agree RL uses much compute for small signal, but value in trading compute for scarce high-quality human data. Gets more from small supervised data than pre-training/SFT alone; enables exploration without golden examples, distilling from bigger models or pushing frontiers.
RL core for exploration; future may find more efficient ways, but current frontier. Curious on Andre's updated views post-Dwarkish podcast on coding RL, e.g., Claude Code popularity surge. Not the end; push agentic capabilities, but limits like context token limits need improvement.
Trainer requires weights; happy to chat for closed models. Environments general for eval, prompt tuning, model selection, agent harnesses on closed models; create data for distillation/upload. LoRA part of fine-tuning (recommended for RL), can't upload to closed like GPT-5 but can bring environments. Partial customization varies by recipe/goal. RL on agent harness around closed: like prompt optimization (e.g., DSPy, GPA algorithm). Supports optimizing prompts, tools, agent skills, system prompts.
Interested in longer-horizon agents, models managing own context vs. scaffolds. Alex Song (RM author) published RLM; exploring via blog post couple weeks ago: RLM harness gives LM access to persistent Python REPL variable for context management/retrieval/transformation, calls sub-LMs (recursive). Improves long-horizon reasoning benchmarks. Excited to train models in this harness next months.
Excited about synthetic data: self-reflection, continual/lifelong learning via prompt optimization, distillation with models curating own training data/environments.
Avoid world where big labs own all future AI value across verticals. Empower entrepreneurs/enterprises to optimize products better than current tools
Currently enabling this, with a lot more cloud code moments and cursor forex type moments enabled through this.
Every company is a neo lab in some way.
If data is the bottleneck or having real expertise is the bottleneck, would you rather have the smartest person in history work at your company or someone who's been there for 30 years? Sometimes you really want the person who's been there for 30 years. There's a lot of expertise from understanding a problem deeply and interacting with it over a long time. This is what happens in training, almost impossible to replicate in a short prompt. You want institutional knowledge to compound over time, best practices to compound over time. This is how institutions and companies grow powerful and successful: they stand on the shoulders of what they've done before rather than resetting every day.
Want this to be accessible to any company that wants to do this. That's the approach, especially as software becomes easier to manipulate and the barrier to entry for coding becomes easier. The same will happen for AI research.
It's really inspiring vision for the world.
Thank you for joining, paving the way on environments and the environment hub, demystifying what an environment is, and sharing the vision for the future.
Thanks. Thank you.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
