RL Environments Explained: How AI Agents Learn Real-World Work | Brendan Foody, Mercor
In a Nutshell
Mercor has shifted from crowdsourcing basic training data to building complex RL environments that teach AI agents how to use real-world tools like Salesforce, Microsoft 365, and Google Workspace across 205 economic domains. These environments require expert humans to create realistic "worlds," high-fidelity app clones, and precise verifiers—since models can't reliably grade their own work—with tasks costing $50-$10,000 each. Post-training on these environments has shown dramatic gains, like boosting corporate law performance from 4.7% to 26.6%, and the future demands ultra-long horizon tasks and virtual co-worker interaction capabilities.
These notes were generated by AI and may contain inaccuracies.
Record grew from a 1 to 2 billion dollar revenue run rate in the last 4 months. The conversation focuses on RL environments as a new frontier topic. In 2020, the era of crowdsourcing data for behavior cloning began, primarily involving supervised fine-tuning data with inputs and outputs, and RLHF data where annotators would select preferred model responses from several options. This approach enabled progress in fine-tuning GPT-3 toward ChatGPT and GPT-4.
By 2024, a significant transition occurred away from low-skilled crowdsourcing of behavior cloning data toward the agentic era of data. The focus shifted to finding the highest-skilled experts worldwide who could work collaboratively in teams to build frontier evaluations and RL environments for next-generation models. This includes software engineers, lawyers, doctors, bankers, and other professionals who can measure the frontier of intelligence and improve model capabilities.
Mercor originated with its first major project being deep research, becoming the primary agentic data vendor to all leading frontier labs and application layer companies including Harvey, Cera, Cognition, and Ramp.
Over the last 12 months, RLVR within the agentic data paradigm evolved to include RL environments featuring rich apps and worlds that teach agents how to use everyday laptop tools. This technology, initially developed in frontier labs, is now disseminating to application layer companies building their own intelligence.
An RL environment consists of three parts:
- Worlds: All messages, slides, docs, sheets, and materials corresponding to real projects or companies
- Apps: High-fidelity clones of popular applications including Salesforce, ServiceNow, Microsoft 365, that agents interact with via MCP, CLI, or Kua
- Tasks: Prompts and verifiers, where verifiers can be rubrics or unit tests used for either evaluation or training
The barrier for frontier labs to automate laptop-based work is covering the full distribution of worlds, apps, and tasks across the economy, requiring enormous scale-out.
Humans remain central to building these environments, with models in the loop. Expert hours from Mercor's talent network reached 2.5 million hours in Q2 alone, with accelerating growth. GDP val contains 205 domains from the Bureau of Labor Statistics, requiring corresponding apps, scenarios, and tasks for each job category.
In most domains, only humans can measure the frontier because models struggle to identify their own mistakes reliably, similar to asking humans to grade their own homework. Building verifiers is challenging because comprehensive coverage requires understanding the full problem space, including all possible valid approaches and dozens of potential mistakes.
A published legal environment demonstrates the process: lawyers from top firms like Latham and Watkins create scenarios from real projects, then build full data room outlines including messages, emails, and files. Models assist in populating environments productively, similar to how software engineers orchestrate agents rather than coding entirely by hand.
The rendered data room uses Google Workspace clones, with prompts to generate model trajectories. One specific task evaluates maximum total liability for Star Tanker Tankers International Limited compared to Cooper Jefferies Energy Corporation under the Oil and Petroleum Act.
Rubric criteria correspond to characteristics of accurate model responses, designed to avoid reward hacking and align with human judgment when scoring 100 trajectories.
Recent leaderboard changes show GLM 52 and Chimera K3 appearing, indicating frontier intelligence is becoming achievable across specific applications and verticals.
Post-training on Apex Agents dataset with 1,800 tasks and approximately 500k in compute showed dramatic improvements for GLM 47: corporate law performance increased from 4.7% to 26.6%. The model generalized well to GDP val and Apex V1 benchmarks despite not containing data rooms.
Three primary approaches exist for curating high-quality datasets:
- Task-based pricing: Most common approach where customers specify desired environment data shapes and pay custom per-task pricing, often $2,000 per task. Frontier labs may purchase 50,000 tasks monthly, with complexity ranging from hours to a month per task
- Off-the-shelf data: Pre-built datasets representing hundreds of millions in investment, sold to multiple customers. Neo labs increasingly prefer this approach to avoid redundant development
- Expert provision: Hourly model for providing experts directly, representing a smaller and declining focus, though this was how Harvey initially engaged
Pricing is determined by working backwards from customer goals around model improvement on specific leaderboards. Value calculation considers how much achieving frontier status is worth and how many tasks would reach that goal. Cost structure factors include human time requirements, with examples showing $1,500 cost basis for 10-hour tasks at $150 hourly rates, plus margin based on task differentiation.
Task pricing ranges from $50 to $10,000 depending on complexity and frontier nature.
Data quality encompasses two primary dimensions:
- Realism: Ensuring environments reflect actual distributions in real professional settings, with experts creating outlines and guiding curation processes
- Verifier accuracy: Rubrics must score model trajectories equivalently to human stack-ranking of the same outputs
Trajectory analysis validates verifier accuracy by rolling out 10 trajectories, scoring them, and using agentic quality control systems plus human review to ensure alignment with objectives. Human feedback evaluations or preference labels can serve as auto-grader validation.
RLVR fundamentally relies on synthetic data by rolling out model trajectories rather than having humans write SFT data, then scoring trajectories for model learning. Models play significant roles in populating environments and creating tasks, similar to how lawyers use Claude or ChatGPT for memo writing.
However, humans remain essential for measuring capabilities beyond the model frontier, as models cannot reliably self-assess legal environment quality or distinguish good versus bad outputs without clear signals.
Deep research environments emerged first because they involved tool use with search as the primary environment tool, representing a lighter version where experts created rubrics without necessarily populating apps.
The 2025 trend toward apps became prominent when companies recognized that model usefulness depended on handling both codebase context and laptop tools. Achieving real user distribution requires matching the data distribution models learn from.
Future developments include:
- Ultra-long horizon tasks: Current agents aren't trained beyond 10-hour tasks, but future requirements include 100-hour or 1,000-hour human-equivalent tasks
- Virtual co-worker integration: Most job tasks (60-70%) require human interaction, yet only about 1% of evaluations measure social interaction capabilities, creating a significant realism gap
AI copilots can make task and verifier creation more efficient by working with experts, allowing trajectory analysis and mistake identification. However, models cannot reliably write complete rubric criteria, often getting approximately half correct, which creates unworkable noise for training purposes.
Task creation remains the most human-intensive process, though synthetic approaches help with environment creation. Exceptions exist in code domains or when distilling from stronger models like Kimi K3 for weaker model training.
Cyber defense and incident response domains allow reduced human involvement in verifiers through attacker-defender agent setups. Humans primarily architect realistic environments and ensure diversity, while verifiers require less human oversight.
Parameter count affects trainability, but the key heuristic involves comparing pass@16 versus pass@1 performance. Models where all 16 trajectories fail completely offer limited learning potential, while models that succeed 1-2 times in 16 trajectories can learn effectively from the data.
Custom data offerings provide siloed, exclusive teams for critical customers to build world-class datasets they own, maintaining competitive advantage while leveraging existing infrastructure. Building talent networks and infrastructure in-house is less efficient than partnering with established platforms, as demonstrated by frontier lab approaches.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
