How Epic's Lore VCS Handles Massive Binary Files | Ben Cooke & Ryn Slack (AWS) | GameDev Day 2026
In a Nutshell
Epic's Lore is a version control system that stores binary files as content-defined chunks (fragments) instead of whole files, enabling 100:1 bandwidth savings by transferring only changed parts when artists modify large assets like 2GB textures. It uses QUIC protocol, Blake3 hashing for deduplication, and AWS infrastructure (S3 + DynamoDB) with edge pods for caching near users. The key operational insight is that commit speed depends on client CPU for chunking, while server size only matters for large imports, with costs around $77-500/month for typical teams using c8g instances.
These notes were generated by AI and may contain inaccuracies.
Ben Warren introduces Epic's Lore version control system, designed specifically for handling massive binary files that traditional source control systems cannot effectively manage. The presentation focuses on textures, network models, sound, stages, and all massive binary data that games are made from.
A single artist scenario demonstrates the challenge: an artist opens a 2GB source file called "Atlas" - a large uncompressed image containing multiple smaller textures. The artist redraws approximately 1% of the image, saves it, and commits the change. This same modification is then processed through three different version control systems: Git with or without LFS, a central server model, and Lore.
The comparison reveals significant performance differences across the three systems. Git with LFS and central server models require uploading and downloading entire files for any change, while Lore transfers only the modified portions. The results are not precise measurements but illustrate the fundamental nature of the problem with binary file handling.
Rain Slack is a Game Solutions Engineer at AWS from Arlington, Virginia. Ben Cook is a senior game solutions engineer from Austin, Texas. Ben covers system mechanics and measurements, while Rain discusses operational aspects, studio perspectives, and emergency response scenarios.
Three key takeaways from the session: a mental model of parts-based storage, understanding the engineering design rationale, and a starting point for operating the system with cost considerations. All numerical data presented includes source attribution.
Game repositories differ fundamentally from source code repositories because they contain gigabytes of binary files that change daily. Git was never designed for this purpose. In a typical Unreal-style project, source code represents only about 2% of the repository, while textures, source art files, grids, 3D models, binary engine files, and audio comprise the remaining 98%.
Three approaches exist for version control of binary files: Git with LFS, central server models, and Lore. Git LFS moves large files out of version history but cannot send partial files - changing one byte requires uploading a completely new object. Regular Git can send deltas but stops attempting this after 512 MB, and default clones carry the entire log.
Central server models maintain one authoring server with terminal servers added near teams for expansion. Each site runs a complete identical copy of default binary file types. Each change creates a completely new revision, and each version pulls the entire file. Some central servers can send only variable parts of uncompressed binary files, but store each version in its entirety.
Central server models present three permanent operational costs: capacity planning at each location (each copy contains the entire archive), preparation time for new terminal servers (metadata restoration from checkpoints and backup requirements), and morning rush hour performance when entire studios synchronize simultaneously.
Lore operates as a central system but stores file parts called "fragments" rather than complete files. When artists commit changes, Lore identifies only the changed parts and transfers those. Copies containing old versions already have the unchanged parts, eliminating redundant transfers. This represents the fundamental difference: two systems transfer complete files while Lore transfers only what has changed.
Using the 2GB Atlas file with a 1% modification (approximately 20MB of changed data), Git LFS uploads the entire 2GB file and every site pulls another full 2GB. Central server models upload 2GB to the server then distribute to each edge mirror. Lore transfers only about 20 megabytes - slightly more than 1% of the original file size - representing approximately a 100:1 improvement ratio.
Default version control settings mean changing a single byte creates an entirely new version stored and sent in full to every location. Studios pay for this inefficiency in three ways: bandwidth costs for each upload, storage costs for saving complete versions of massive binary files, and latency costs during synchronization operations.
Lore stops versioning files and instead versions file parts. From the developer's perspective, Lore performs identically to other version control systems with the same command types (pull, push, commit, sync), but operates fundamentally differently in the background.
When committing files, the client workstation splits files into parts using content-defined partitioning. Unlike fixed-size slicing where adding a single byte shifts all subsequent chunks, content-defined chunking allows chunk boundaries to move with content changes. Most chunks remain identical after modifications. Each chunk is named using its Blake 3 hash - a digital fingerprint where identical bytes produce identical names.
Parts with specific names are stored only once across files, branches, and dates. This automatic deduplication occurs through the naming system. When an artist redraws one area of the Atlas file, only the affected chunks receive new names while unchanged chunks retain their original names.
Commits in Lore are atomic - Version 2 is identical to Version 1 with only changed items replaced. Expected chunk size is approximately 64 kilobytes, with maximum size of 256 kilobytes. The 2GB Atlas file contains tens of thousands of actual chunks rather than the simplified 14-chunk illustration. Modifications to any file portion only affect the chunks it covers.
Lore achieves network performance through the QUIC protocol - a modern Web 3 transport operating over UDP. Each chunk has specific requirements, requiring thousands of small requests. One connection carries eight data paths, with two reserved for small tree reads and synchronization. Clients can maintain up to 10,000 concurrent requests with responses returning in any order matched by ID.
When packets are lost in QUIC, only the affected data path waits for retransmission while the other seven paths continue functioning. In contrast, TCP halts the entire connection when a single packet is missing. QUIC operates within the application rather than the operating system, allowing Lore to implement custom modifications.
Congestion control uses BBR (Bottleneck Bandwidth and Round-trip) algorithm, lifting data one path at a time for faster complete chunk arrival. Each window equals bandwidth multiplied by latency. For clients, this means gigabits per second over 100 milliseconds (approximately 13.4 megabytes), with single connections peaking near window divided by latency.
At 100 milliseconds latency, single connections achieve approximately 134 megabytes per second. At 67 milliseconds, performance reaches approximately 200 megabytes per second. These are calculated values rather than measured results. All testing occurred within virtual private cloud (VPC) environments using EC2 instances with maximum speeds of approximately 5 gigabits per second (625 megabytes per second).
Clients communicate exclusively with Lore servers (EdgePod or direct write layer) without Amazon credentials or AWS SDK usage. Parts travel via QUIC connections while branches and locks move via gRPC over TCP.
The trusted layer consists of S3 object storage (one object per part, named by hash value) and DynamoDB key-value NoSQL database maintaining part indexes and metadata. Branch pointers and write-layer locks in Lore.8 design are accessible only to write-layer servers. When clients request specific revisions, servers read branch pointers from DynamoDB, then clients traverse trees made of parts, fetching only relevant paths.
Fetching involves one GET request per part with maximum 256 kilobyte size, executed simultaneously. If EdgePod cache misses occur, the write layer checks memory before pulling from S3. Two layers of buffering exist in this process. During reconstruction, clients verify hash values for each part, compile files, and write to workspaces without requiring server return trips.
Every byte passes through Lore servers, making server throughput the ceiling for large synchronization and import operations. Each S3 failure generates one GET request per part, plus DynamoDB index searches. Clients require Lore code for authentication, and AWS permissions are not visible to the system - everything is managed within Lore.
The system has two layers: the permanent layer (S3 and DynamoDB) representing the trusted repository version, and the write layer (EC2 container with local NVMe device) handling commits and serving read operations. Edge pods are regular EC2 instances with NVMe cache placed near read locations such as CI fleets or regional teams, serving as the second caching layer.
Everything to the left of the trusted layer can fail without data loss - temporary caches cost time rather than data. The write layer is replaceable and operates within auto-scaling groups, automatically replacing failed instances while commit processes wait for restoration. S3 and DynamoDB have multiple deletion protection layers.
Lore supports standard version control actions with data movement directions indicated by arrows. During commits, hard work occurs at artist workstations where files are split, digital fingerprints created, parts compressed, and only new parts uploaded. Servers store each part only once.
Push operations are minimal compared to other systems. Since commits have already uploaded parts, push operations only verify existence and move branch pointers in DynamoDB from one version to another. Main branch pointers move from version 1 to version 2, typically taking less than half a second.
When artist B synchronizes, only missing parts are pulled via edge pod connection points. These points and their temporary caches retain copies for subsequent users. Every file within the user's field of view is fetched, with nothing outside that scope. The write layer sits between edge pods and storage.
Four sentences describe the complete model: content is broken into small parts with automatic deduplication, commit operations upload parts while push operations verify existence and move branch pointers, synchronization fetches only visible content with pods never retaining sole copies of committed content, and storage operates as append-only with caches deletable but S3 space permanent.
Disposable cache memory eliminates formatting and backup requirements. New sites can synchronize via edge containers instead of waiting for complete edge archives. CI agents can also synchronize through this approach. In a 500GB repository with 1GB/s connection, complete edge archive setup takes approximately 97 minutes including 20 minutes for checkpoint restoration, 67 minutes for archive copying, and 10 minutes for verification.
Edge containers eliminate checkpoint and edge database requirements. Container operation takes 5 minutes, with first synchronization pulling only relevant parts adding several more minutes. First useful commit occurs in approximately 8 minutes, representing an 89-minute overhead reduction compared to full edge archives. Container caches fill while work continues without waiting periods.
A 105GB media snapshot from Netflix's "Meridian" (Creative Commons licensed film containing uncompressed UHD video frames and Dolby Atmos copies) was migrated. The write layer did not complete the first attempt due to S3 restrictions within minutes.
When a large import was sent to the correct layer, the first attempt did not complete. Within minutes, an S3 restriction was triggered in the new account and new container, causing the speed to drop by about nine times. The process never recovered, and after a little over an hour, it was turned off. The same import completed through the edge container in about 8 minutes and 45 seconds.
The container does not perform caching and confirms each part only once. The correct layer has S3 and never keeps the only copy, requiring one run for each part. This test used Lore version 0.8.6.
The security barrier prevents large import operations exceeding 50 GB from passing through Pod. Reverse pressure and control are better at the time of writing. Lore is currently in a pre-1.0 version, working toward version 1.0. The 10GB repositories went through directly without any problem, but the 105GB direct access with default settings on a new account was not completed. 50 GB is the margin between the two, and while not a precisely measured limit, 50GB is a good number to keep in mind.
The Pod never keeps a single copy of data. Reading processes take place while people are working, and writing processes pass through to the permanent level. The servers have grown in size beyond what was available on day one.
To create the system, run the software modules. Once the server image is in the registry and a network path to AWS is provided, deployment occurs around the afternoon. Only one Terraform module is needed to create all three levels. This module comes from the Cloud Game Development Kit, an open-source project from AWS consisting of three parts.
Assets are reusable items including Packer templates, pipeline definitions, and post-deployment operating logbooks. The modules are Terraform itself, called from the code. The samples are complete configurations illustrating how the units work together.
The modules cover version control, UnrealHorde, and UnrealCloud DDC, which captures derivative data such as aggregated shaders, build servers, virtual workstations, and Unity tools. Lore is the newest module.
The Terraform creates a hypothetical example showing clients (artists and CI agents) connecting via private line to AWS. Containers (pods) and write levels exist in private subnets, not publicly available online, only allowing access to office domains or VPNs that permit it. Customers sync from edge pods.
There are two edge pods in separate locations within the area, designed for about 20 to 40 people. Behind them, the writing level handles receiving each submission. If the device malfunctions, automatic measurement kits can replace it. The permanent level includes S3 holding the parts, plus DynamoDB holding the index and branch pointers.
A few people can skip the containers when building the system. The permanent level is the same in both forms, so adding containers later does not transfer any data. Add two blocks of containers, run the application, and direct customers to those containers. If the writing level also increases, this can be dealt with by a simple change in the variables.
The test group ran its own version of the module. There are only three decisions to consider on day one: the image of the environmental container and allowed entry zones, the types of instances for the writing level and each container, the number of containers and terminal container blocks, and the area that people overlook.
Each container defaults to an instance containing approximately 32 virtual CPUs, costing around $1,000 per month depending on the region. The writing level defaults to an instance costing approximately $250. Ben's figures indicate smaller devices operate with the same efficiency. The default and minimum examples are provided as development settings only.
There is no login and no protection against deletion. For production use, set the authentication mode on Cognito or any third-party identity provider, activate deletion protection, and disable the force destroy option. The container ships with versioning disabled, so backups must be planned separately. The basic container block does not contain login inputs.
The honest answer about the right size is becoming a new thing. Previously the answer was "It depends." Lorebench was built to measure this, employing real Lore customers and real deployments. It takes everything from the architecture diagram and combines it with a number of test clients to iterate quickly across instance sizes and available cache at different levels.
All testing was accomplished using version 0.8.6, though version 0.10.10 has since been released. 21 different server configurations were tested, from one client to 29 clients, across two sets of data.
Stackobot is Epic's experimental project for the Unreal engine, consisting of approximately 2000 files with a size of about one gigabyte. UHD media consists of huge multi-gigabyte files, mostly around 10 gigabytes, with a total size of about 105 gigabytes for a data transfer operation.
All latency measurements are within the scope of Virtual Private Cloud Networks. Customer communications were separated to isolate the minimum baseline. File sizes were classified as small (1-5 people, up to 20 GB, 2-3 simultaneous users during peak), average (20-40 people, 50-200 GB, about 30 people syncing during Monday morning sync storm), and large (hundreds of people across many sites).
Each size was given its own fleet. The small size consisted of three clients on nine different forms of the writing layer. The average was around 30 clients across a few groups, with eight different group formats. The large-scale operation is still awaiting approval.
Scenarios included Monday sync storm, attaching a new site to a cold set, importing a project, daily flow of small edits with a monitor acting as a single artist, filling cold writing layers, and quick small deposit operations. Complex preheating and integration tests remain to be conducted.
Every file that was cloned and synchronized was verified outside of the system, not relying solely on successful exit codes from the client. Each file was examined completely independently to monitor data integrity issues.
The write layer was tested on six different sizes of C8g and C9g computing instances, ranging from one virtual CPU to 8 CPUs. Import speed depends on the amount of virtual CPUs available to the servers. However, for artists or game developers making small tweaks, the P50 index hardly changes regardless of server size. Server size does not speed up commits—it only makes a real difference with large and massive import operations.
The flat line occurs because the process mostly involves exchanging metadata with the DynamoDB database. For a single box without DynamoDB, the same commit process takes about 4 tenths of a second inside a VPC. Large asset commitments differ because the workstation must split and compress the data, so the client processor determines the speed, ranging from 170 to 290 megabytes per second using two to eight core processors.
Edge pods are a major advantage of Lore, helping distribute tasks and bringing data processing closer to users. The size of the pod is practically unnoticeable apart from a few moments. With a continuous stream of small operations, the slowest 1% of all invocations was about 170 milliseconds in any pod containing two or more processors. A single processor configuration is unacceptable, so at least two processors are recommended for both the writing category and pods.
When running intensive synchronization processes, once data output rate stabilizes and outgoing data traffic stabilizes, the processor has already reached maximum capacity. It makes sense to expand scope if greater productivity is needed.
CloudWatch logs or the CloudWatch proxy on the modules or write tier level collect processor and network data approximately every 10 seconds and trigger alerts about ENA counters. When the allowed limit for outgoing bandwidth is exceeded, moving to a larger unit size makes sense.
Most people want to set up the system with only the write layer, considering peripherals as added expense. Edge pods are used for geographical location and for large synchronization operations.
Lore servers must contain NVMe storage space with NVMe drives inside. This is extremely important as it is one of the main reasons the version control system is so fast, relying on very fast storage in caching nodes. Direct access to the right layer with massive import operations is where failures happen.
Once 14 or 15 active users are using a particular unit, edge pods start to become truly useful. For one to five people, edge units may not be a necessity, though they provide buffering for large import operations. A single unit has never surpassed direct contact for large import operations, though it accommodates them by placing cache memory alongside readers such as CI fleets.
For US East 2 on-demand service, costs are mostly covered by computing costs. Storage costs for S3 and DynamoDB for a 20GB data warehouse represent a small fraction. For 1 to 5 people and 20GB of space using c8g.large for the correct layer, costs are about $77 per month. Slightly larger sizes with a few edge units and c8g.xlarge cost approximately $312 per month. For teams under significant pressure, costs reach slightly less than $500 per month.
The $77 figure covers computing and backend systems. Additional costs for VPN, NAT gateway, and CloudWatch may add $40 to $50 per month. Storage cost is a small part of computing cost, which depends on the amount of data moving in and out of the system.
Single vCPU forms should not be used for any purpose as they are slower and lead to reliability problems. Use two or more vCPUs and NVMe storage to maintain speed. Cold pod units can only accommodate about five customers at a time. The cold unit has completely empty cache memory.
Data must be verified after each copy and synchronization process. Check the expected number of files and ensure no temporary files remain. If importing anything larger than 50 GB, working via a pod is much better than working directly.
The 105GB import failed due to S3 speed limitation. S3 can handle approximately 3500 write operations per second per section for a brand new account. When bringing in 105 gigabytes made of very small parts, restriction began. Partitioning based on filename increases capacity from 3,500 write operations per second to 56,000 operations per second. Partitioning based on the first letter was more than sufficient for tested systems and loads.
A new on-demand DynamoDB table can handle approximately 4000 write operations per second before scaling up. During the 105GB import, throttling occurred in both parts tables. Tables can be pre-formatted using the command line to maintain rate-per-second limits for large imports. The Cloud Game Development Kit module does not yet offer this feature.
Using a VPC resolver for DNS takes approximately 1024 packets per second per network interface. The client requesting many different parts across multiple streams may encounter a bottleneck in resolving DNS queries. Caching is essential for every client—this is default in Linux and Mac, but Windows clients especially need DNS caching enabled.
For a team of five people, the main variable is "right tier" starting with c8g.large instance type, providing a foundation at about $77 per month. For 20 to 40 people with 50 to 200 gigabytes, define two units of type c8g.large and one unit c8g.xlarge, costing approximately $311 per month.
Watch for 10-second metrics for container CPU. If hitting 100% with outgoing data transfer rate staying flat during morning synchronizations, replace containers with XL ones one by one. Both configurations use Graviton sizes, requiring server images compatible with ARM 64 architecture.
For 100 people, approximately 15 simultaneous synchronizations per container are needed, with 7 computationally optimized XL containers per 100 people, and at least a computationally optimized 2XL level for continuous import and integration operations. This amounts to approximately $1,500 per month for up to one terabyte. Every additional hundred synchronized at once adds about seven containers, equivalent to about another thousand dollars.
Multiple terabyte repositories exceeding local disk capacity for write levels and more than 50 continuous integration agents representing a second morning peak remain pending questions. Multiple locations will be addressed later, with one unit per region meaning separate warehouses.
Lorebench will be released and announced through a blog post. The link is the first in a series, with more coming soon. Links to Lore itself from Epic and the suite of cloud game development tools are available. The modules are included within that suite. Users can publish, report problems, and send withdrawal requests to help close gaps.
The team is seeking input on daily warehouse change rates and user experiences with the amount of changes occurring. They need to understand who is making changes, as these rates of change and roles represent the least accurate elements in their current model. Despite working with many different game studios, additional input on sizes and numbers will improve these models.
Lore deals with parts, not files, representing a fundamental design choice with three main points:
Every cache contains parts, not complete copies. The active storage unit holds most of today's file fragments while the inactive unit is only refilling. This ensures nothing that is approved is lost.
Any new site or Continuous Integration client will only pull the missing parts of its offering. This allows immediate usefulness after the first synchronization process.
Storage space only grows based on what has been changed, with each part stored only once. The system can be revised through trimming, though this is a very manual process. Volume can grow permanently if not refined, though storage remains a small part of costs while computing costs represent the larger portion.
The session concluded with approximately five minutes available for questions, with Michael joining as the moderator.
Daniel expressed excitement about Lore and noted significant excitement during its RealFest announcement earlier in the year. The team has begun testing integration between Lore and TeamCity through a custom add-on, with plans to monitor this area for future additions. Good feedback was received in the chat.
Lore represents a modern version control system that functions more like storage software. Given its pre-release stage, studios are taking a wait-and-see approach, though many are preparing and have started using it. Early adopters are emerging, with predictions that good deployments will occur within the next three to six months.
Epic is actively developing and iterating on the system, with changes occurring even since the presentation numbers were compiled. The command-line-only interface currently limits functionality, but integration with asset managers and similar tools will provide significant improvements.
A missing case in the proposed configurations is the single box scenario. While Lore uses a modern cloud architecture with Amazon S3 for durability and DynamoDB for scalability, it can run on a single server with different performance results. A single device equipped with an NVMe disk can run at a workplace or personal space, but the reasons for using S3 such as durability and deletion protection become relevant when the technology expands.
No fundamental flaws have been identified in the basic structure. The system relies primarily on BLAKE3 encryption and the QUIC protocol, which are among the best available, resulting in very high performance. Projects can be moved in and out remarkably quickly.
Lore is open-source technology with no additional licensing costs, which benefits independent teams. The single device deployment characteristics should also be released to show how the system performs in that configuration, though the architecture expands spectacularly for distributed teams and studios.
Keep JetBrains in your library
Save the videos and channels worth coming back to, and find them again in one place.
