How Harvey Built a Research Lab on a Budget | Gabe Pereyra
In a Nutshell
Harvey's Gabe Pereyra shows how application-layer companies can compete with frontier labs by using open-source models, Neo Labs for post-training, and domain experts to generate synthetic legal data that bypasses customer data restrictions. The playbook starts with building realistic benchmarks like Legal Agent Bench and large diligence datasets, then moves through efficient RL environments, open-sourcing data for validation, and serving post-trained models alongside closed-source ones with proper evaluation infrastructure. Key insight: frontier ecosystem tools now make it possible to reach frontier-level intelligence on specific domains like legal work without the resources of big labs.
These notes were generated by AI and may contain inaccuracies.
Gabe Pereyra is co-founder and president of Harvey. He was previously a research scientist at DeepMind and then at Meta before showing his college roommates what GPT-3 could do. The duo then became Harvey.
The alternative title to this talk is building a research lab on a budget. It is an unfair game competing with the frontier labs if you are an application layer company. There are rich teams, poor teams, and then application layer companies. The frontier labs have more money, more talent, more compute infrastructure, and data.
The way to compete is by using the frontier ecosystem. When Harvey started 4 years ago, most of these companies either did not exist or were just getting started. Companies either had to build everything themselves or focus on something different like building their GTM org and a great product. Today using the frontier ecosystem you can compete with the frontier labs and build frontier intelligence.
The high-level playbook includes building benchmarks and training data, working with the Neo Labs to do post training, and serving models in production. To start, you want to build a benchmark. If you do not have a good benchmark, you cannot train models, and if you cannot train models, you do not need to serve them in production.
This year Harvey released three data sets. They started by building Legal Agent Bench, which is a taxonomy of tasks that associates would do at a large law firm. These cover multiple practice areas and include complex tasks like drafting complex fund formation documents and doing case law research.
They followed this up by building a contracting data set which allows teaching agents to do negotiation like you would in an in-house department. The most exciting recent release is a large diligence data set, which is one of the largest RL environments that has been released. The largest data rooms are 80 million tokens, and it lets them do research on long context, very complex tasks.
The challenge at Harvey for training models is that they cannot train on customer data. They work with the largest law firms and enterprises, and their legal data is incredibly sensitive and privileged. You cannot put it in generic models or even in their own models.
The thing that started working really well is using domain experts to guide synthetic data generation. Brendan from Reka had a good analogy that the same way that engineers now do not write code, they vibe code and guide these coding models. They are starting to do the same thing.
Gabe's younger brother is a lawyer at Harvey and has gotten very good at using the coding models. He has trained other lawyers to do that, and they can generate incredibly realistic data sets that can be used for training and evaluating the product.
Synthetic data is not good enough but it is a way to get started. Harvey works with companies like Mercor and Snorkel, who let you scale up this process and build larger sets particularly for training.
Once you have data sets, you need to turn them into efficient RL environments. It gets really expensive as these data sets get larger and evaluation gets very expensive. In their diligence data set, they have over 1,000 unit tests that are grading model outputs using LLM as a judge. Using the largest models for RL rollouts gets very expensive.
The last thing they did that was controversial at the time was open-sourcing some of these data sets. The motivation was that it is very hard to know your data set is good unless a lot of people train on it. When Gabe used to do research at Google Brain and DeepMind, the best data sets were open like ImageNet, CIFAR, MNIST, and everyone used them, and you were able to find all of the issues.
They get a ton of pull requests and suggestions. Increasingly the labs when they report new models benchmark on their data set. Once you have built the benchmark, now you have something to train models against. The exciting thing now is that open-source models are getting competitive. In the past, it was not worth doing post training because the models were improving so quickly from pre-training that any post training you did quickly got absorbed by the next pre-trained model.
Now with models like Kimmy 3, GLM 5.2, NeMo-Megatron, Inkling, and others, it is possible to take these very strong open-source base models and post train them to levels of frontier intelligence. Maybe not general frontier intelligence, but if you have a specific task like Harvey does, they are competitive.
The way to get started is working with the Neo Labs. They have expertise and infrastructure already in place to help you make sure that your training data sets are good. They have recipes. Usually, if you work with them and you are not able to get better results, there is probably something you are doing wrong with your data set, and this is a very good way to bootstrap it.
Some of the interesting work they did with different providers includes Fireworks, where they got interesting results training GLM 5.1 to use Fable or Opus 4.8 as an advisor model. With Base 10, they did interesting work on KB compaction. With N gram, they are doing interesting work on enterprise search and firm knowledge. With Trajectory, they worked to train NeMo-Megatron models. With Applied Compute, they are doing interesting work on their Vault product.
The question is why work with multiple Neo labs instead of just picking one. For Harvey, as they are scaling the research lab, they have more research projects than they have bandwidth to do internally or just with a single Neo lab. Every Neo lab is taking a different bet. They have different ways they think about research. Harvey has different open source models they want to train. The more they work with, the more they learn.
It is getting easier than ever to do post training. Working with the Neo labs, they are learning a lot in partnership with them. They are doing more and more post training themselves internally with APIs like Tinker and the infrastructure Fireworks and Base 10 have built. It has never been easier to post train these models and then serve them. There is increasingly more post training talent that they are hiring and is available. Inspired by Cursor, the goal of these efforts is to build their version of Composer One. How do they package all of the work they have done with synthetic data, scaling it with Mercor, the work with the Neo labs, into a model they can serve alongside the closed source models.
Once you have post trained a model, you need to be able to serve it in production. This is non-trivial. Harvey operates in 60 countries. They have multiple product surface areas. Customers have different model preferences. Even with just the closed source models, how do you serve these at scale. The matrix gives a sense of all of the things they need to handle when serving models at scale.
For each of these model families, they need to serve multiple of these models. They need to have fallbacks across providers to hit their SLAs. Now with companies like Fireworks.ai, they can add open-source models into this mix.
In order to think about when you serve models in production and how you modern them in production, you need to have this infrastructure in place thinking even before you think about post-training. This is how they think about when there is a new model released, whether it is open-source, closed-source, or a model they have post-trained, how they make decisions whether to put it in production. And then once it is in production, how they make decisions whether to keep it in production.
Pre-production, they have a set of generic evals. In terms of automated, they have the lab benchmark that gives a very quick sense of is it a frontier model, how strong is it, what areas of legal is it good. They have human testing in the generic case where they run side-by-sides of this model with other models to compare them.
For every product surface, they have critical user journeys and automated product tests because a model could be very good generically, but it could not be a good fit for a specific product surface. They also have human product testing. Together these signals along with heuristics around cost, latency, region availability is how they decide whether they put a model into production.
Once a model is in production, the same thing applies whether the model is post-trained or not. They do AB testing if they are rolling out a large change, they look at engagement to track if this model is performing how they expect. They look at things like uptime, token efficiency, and product feedback which Gabe calls angry customer emails. There are all of these signals that tell you this is working as expected.
The thing you need to do even still before serving models is what Gabe calls the simple open-source switches. The first one is look at all the places you are serving models and find are there places in the product where you can just naively swap open-source models. For example, they have parts of their product that generate citations that do not need the largest models and there are opportunities to swap in GLM 5.2 and get cost or performance benefits.
That is the first thing and doing this builds the muscle of serving open-source models alongside closed-source models. The second which has gotten very popular now is model routing. There could be places where you cannot naively swap an open-source model, but what you can do is on certain queries route to open-source models.
Once you have done this, you have everything in place to start building the post-training flywheel. Once you start serving them in production and collecting feedback, you need to be very careful about what collecting feedback means. In their case, it does not mean training on customer data, but they do get feedback signals from their user testing and other things that can inform how they build future data sets and improve these models.
That is their playbook for building a research lab. They think in the future every AI company is going to need to become an AI company and figure out some version of this playbook. Despite that, most people are still betting against this playbook and application layer companies and the frontier ecosystem.
Gabe references a scene from Moneyball where they are talking about winning 20 games in a row. Billy Beane says it does not matter. If they do not win the championship, no one is going to appreciate what they have done here. The quote is: if we win on this budget with this team we will have changed the game. Now with the frontier ecosystem all of you have the opportunity to do the same. Go change the game.
In the Q&A, Gabe elaborates on the synthetic data generation process. The biggest challenge with legal work is when you are at a large law firm, the type of work you are doing is representing a company doing an M&A. You get a data room, which is all the contracts of the company you are trying to acquire, plus all these emails and meetings about the negotiation. None of that data is public, so you do not have the analogy of open source GitHub repos.
The biggest challenge you run into when training is you can maybe find some of the final work product publicly, like a public purchase agreement, but you do not have any of the input data. Historically it was very awkward to get a bunch of lawyers and say make a fake data room. These data rooms can be 10,000 contracts. They all need to fit together.
What Julio figured out is a very clever way to generate these data sets. He started from the rubric, and he planted all these issues and said here are all the problems in the data room and what I am going to check for in a scenario, and then use that to generate the data room, so you can plant all of these issues like these contracts do not tie together, this contract is missing, and generate all of the data, and then use Mercor Shenorcal to make all the contracts look realistic.
Now you have this input data set, and you can have the model generate like a diligence memo, and this way of checking it to say did you catch all of these issues that we know that are in here because we planted them. Depending on the work you are doing, you need to be clever of how you create it, but that feels like the really big unlock because now they can start doing this training that before you just had this chicken-and-egg problem.
They would go to law firms and say we can train you this model on this client data. They would be like okay prove it. They were like show us the data. The law firms would say no. Now you can prove it here. Now they have a lot of interest from law firms saying this is really interesting. Can we do it with our data.
Regarding hiring, Gabe thinks this was one of the big mistakes he made when they first started the company because he had worked in these labs and knew a lot of the folks and would say come work with us and then they were getting these 100 million plus pay packages and they were not at the scale where this made sense.
They are now just getting to the company size where they are not competing with the frontier talent, but there is increasingly folks that are doing PhDs, folks that do not want to work at large labs and so that is changing. A lot of why you needed that talent early on was it was not just doing the post training, it was you need to build the training infrastructure, the serving infrastructure, and all of those things combined, the talent to do that was very unique.
Gabe's old roommate was one of the folks who ran post training at OpenAI and he was one of the best researchers he has ever worked with, but now you can use things like Fireworks, Tinker-like APIs, and so you do not need to build the training and serving infrastructure, and so that opens up the pool of talent and it feels very feasible to get some of this talent now.
On the benchmark creation question, they think of it more as how do we make them realistic representations of the work their customers are doing. Part of why they picked the legal domain is if you just build a realistic client matter that these top firms are working on, the frontier model still cannot do this.
The thing over the past 4 years they have built is that Gabe's brother was one of their first hires and has been working with the models for 4 years. His intuition of how the models work, how to generate these data sets is incredibly good. He has trained a bunch of lawyers to do this. The focus is still more of how do we build a really realistic data room, and then rubrics for the diligence.
On the question of if something is not working in the end-to-end motion of data generation, environment piece, algorithm piece, research piece, and infra piece, what have you seen where it is usually not working. Gabe says this is a whole another talk. He does not think there is a single thing.
What makes it easier for them is they found product-market fit. They have a product that is being used in production. They did this largely with the closed-source models to start. That let them build a lot of this confidence in the thing is working end-to-end. They have a product, people are using it, people have been using it for many years. The models work in this way, they can monitor whether they are working in production.
They have built the muscle of new models come out, how do we put those into the product, whether they are closed-source or open-source models. You can think about the full cycle. They have done a bunch of work of Harness engineering and context management and all of these things. Now you can think of post-training as one small input into this broader system.
They have their team post-train a model, and then you feed it into this broader system of you treat it just as another new model. Then you have all these signals of what is going wrong. Across these steps, there are easy ways to gate it. If you start with the benchmarks and you post-train a model and it does not work well on those, then it is unlikely to go farther.
If it works well on that, and then you put in a product, but someone uses the product and they say things feel weird because on the benchmarks they have built, there is still things that they definitely do not catch. The best example is you can have a model that does very well on their benchmarks, but they have overfit to some degree, and then you put it in a more generic assistant-like product, and it kind of falls apart for when you go out of distribution.
On the philosophy on open sourcing the benchmark, even before they open source this benchmark, they already work closely with the labs. They share data with them to help them improve their models, which improved their product. The way they think about it is there is a huge advantage in your industry, particularly in legal, in terms of helping their customers understand how good different models are at different things.
When Gabe started Harvey he had the intuition of we will just build the best model and then customers will be happy because it is the best. It is very clear now that every customer has different preferences, different models are good at different things. The second thing he did not anticipate is how big the frontier ecosystem would become.
Now they have so many companies reaching out saying we have this new technique or we did this thing. Can we try it. Before they just did not have the bandwidth where they were just working on these other things. But now with this open source data set they can say go try this and if it works, then great. This is something that is interesting to invest in.
Strategically the valuable data for them is going to be helping law firms train their own systems and how they work on private data. They want to help everyone improve these systems with synthetic data and some of the data they open source, but obviously a balance for the reasons mentioned.
On the remaining open questions, one big challenge is that they can generate these very realistic synthetic data sets, they can augment them with humans, but the distribution of that data still does not match their production distribution. These data sets they are generating are somewhat future facing.
They can generate a really realistic data room, but right now their product is not used purely to do a due diligence. If a model does well on that, but then someone uses it to draft an email, maybe it does not work as well. There is still that gap because they cannot look at customer data, and so how do you bridge that gap. That is the biggest question on the data set side.
The largest data room contains 80 million tokens of context. Current models struggle to effectively manage this volume of information, creating a significant performance gap in handling complex legal environments.
Progress has been made on post-training models internally, but continual learning represents the end game. The goal is not to build the best legal model, but to help every law firm or enterprise customer customize the system to their specific type of work.
The challenge involves operationalizing continual learning where a large law firm can have its AI system improve with every client matter while simultaneously protecting client data. This presents technical, operational, and AI challenges.
Foundation readiness includes an open source model and infrastructure called Ruby. The focus has shifted from competing on individual productivity tools to building for organizational productivity at scale.
Original products and tools like Coda Work focus on individual productivity. The current approach targets organizational productivity for large law firms managing 10,000 clients and ensuring all client projects proceed effectively and profitably.
Most project challenges involve coordination rather than individual document drafting. A typical project spans 6 months with teams of 20-30 people across the firm, requiring coordination of both internal teams and outside parties. This resembles project management that orchestrates humans and agents at the team level.
At the organization level, firms manage a thousand simultaneous projects requiring resource allocation decisions, billing optimization, and pitching strategies.
Fortune 500 companies work with a thousand law firms and maintain a thousand internal people, creating complex systems for work allocation and trading across multiple parties.
The strategy involves going hyper-vertical into specific domains in ways that horizontal products cannot match, following historical enterprise patterns.
Keep Sequoia Capital in your library
Save the videos and channels worth coming back to, and find them again in one place.
