Why Inference Matters 12x More Than Training | YC Paper Club
In a Nutshell
**Inference is 12x more important than training over a model's lifetime, yet research and infrastructure still optimize for training.** The real bottlenecks are decoding throughput (14K tokens/sec vs 1.5M for prefill), KV cache hit rates, and model parallelism across 72+ GPUs. The winning strategies are quantization without retraining, automated kernel search via saturated equality, unified memory trees for prefix sharing, and routing requests to where KV cache already exists.
These notes were generated by AI and may contain inaccuracies.
The discussion opens with emphasis on reasoning as a critical area of investment at YC, particularly in artificial intelligence infrastructure. The speaker highlights that inference has historically been a bottleneck, making it very difficult to run efficient inference at scale without incurring heavy losses in cost of goods sold (COGS).
The market capitalization of inference-related companies has reached $220 billion. Three to four notable companies are represented in the discussion, including Dan Fu as Head of Research. The speaker advocates for thinking about reasoning from a comparative perspective between training FLOPs and inference processes.
DeepSeek is identified as the only company that publishes all its training data. DeepSeek V3 training required approximately 2.8 million H100 hours, equivalent to 43,000 H800 hours daily. This data is noted as outdated, suggesting actual training time is likely significantly higher. Even with these figures, inference processes represent six times the training processes for these weights annually. When considering a default model weight lifespan of two years, the importance of inference increases to 12 times that of training.
The speaker argues that while training operations should be minimized in research (such as in NeRF studies), inference processes should be prioritized as 12 times more important. Current NeRF research typically shows standardized training processes on the x-axis, but standardized reasoning processes should become the focus instead.
COGS is emphasized as critical for model servers. Recent data released shows DeepSeek achieving 80% profit margins due to their significant innovations on the inference side.
The inference process begins with receiving a request containing a set of symbols and identifiers, followed by calling prefill to process the initial prompt. If data is already stored temporarily, prefill should not be re-run. Optimal storage locations include DRAM, Redis-type cache, or SSD, with the LRU algorithm recommended for cache management.
Using the FP8 coefficient, GPU utilization for training and prefill is often around 50% of theoretical maximum. If Jensen Huang promises 2 trillion FP8 operations, actual achievable performance is approximately half. During decoding, the situation is worse, with utilization around 5%.
The primary performance indicators that Dan Fu and others continuously work to improve are:
- Time to first token (TTFT): duration to complete the prefill process
- Time per token: time required per decoding step to produce the next token
- Throughput: symbols produced at a given service level
Recent data center numbers show prefill throughput at 60,000 tokens per second per GPU, excluding cache memory access cases. With cache memory access included, this reaches 96% access rate. When cache access is available, throughput reaches 1.5 million tokens per second per GPU.
Time to first token is approximately 400 milliseconds, with P99 time to first token around one second. Decoding throughput reaches 14,000 tokens per second per GPU. The significant difference between prefill (1.5 million tokens/second) and decoding (14,000 tokens/second) makes decoding the real bottleneck.
The retention policy for KV cache memory is highlighted as particularly interesting. Users have petabytes of data stored in KV cache memory, retained for 72 hours, which is the secret to achieving high cache hit rates.
Arithmetic density is defined as the number of floating-point operations (FLOPs) that can be performed on 4 bytes when transferring data from DRAM to SRAM. For diffusion models, arithmetic density is very high because the image is loaded once and noise must be transmitted, requiring repeated weight file access on the same network. For LLM models, arithmetic density is very low.
When purchasing H100 processors advertised as providing 2 trillion FP8 operations, actual test performance is typically much lower. Model FLOPS Utilization (MFU) represents the actual performance achieved. Memory bandwidth represents peak memory frequency bandwidth, with the question of what average is actually achieved during testing.
Roofline analysis determines the best possible performance, calculated as the lower peak of floating-point operations per second for the card versus memory bytes per second multiplied by floating-point operations.
Innovations to improve inference are classified into three categories:
- Model innovations: MoE models reduce floating-point operations by 1/K where K is the number of experts; SSM models that don't require full attention in every layer; multi-head latent attention that compresses the sequence field; distillation from large models to smaller samples
- Kernel and runtime innovations: FlashAttention, PagedAttention, inspired by classical operating systems and database theory
- Memory innovations: routing that doesn't necessarily send everything to the main model; waterfall sequences or tool calling to larger models
Speculative decoding uses a draft model to verify authenticity. The preferred model is repetitive, where full context is not stored in the knowledge base memory, but the language model determines what it needs and when. This approach can reduce the size of extensive knowledge base memory.
Compression is noted as an essential part of any system, with notable innovations including cartridges created by researchers in the lab. Recommended resources include Percy Liang's CS336 lecture 10 on language models from scratch, and Rainer Bombach and Darraksh's lectures.
Dan Fu is introduced as conducting research at Together AI, having graduated from the Hazy lab. Other panelists include Jeffrey Morgan (Founder and CEO of Alama), Joe Fiati (Founder and CEO of Luminal), and Lang Shang (core developer and second author of research papers).
Dan Fu's presentation focuses on the token lifecycle and request lifecycle in detail, covering how requests are sent through APIs to data centers, reaching GPU cores, and how tokens are returned. Six approaches to speed up the process are mentioned, with three to be discussed.
The current moment in inference is described as truly exciting. Before a few months ago, Together was processing 400 trillion tokens monthly. At the same time, Google published that Gemini handles this volume of traffic. As a young startup, Together achieves one-eighth of Google's total traffic volume.
Earlier this year, more than half of tokens through OpenRouter went through open models. Two weeks prior, for one day only, Together had more tokens than OpenAI combined. Open models drive the market because they are much cheaper, offering up to 10x lower cost while achieving similar or better quality in many cases.
Together is described as a specialized AI cloud platform, similar to AWS, where users can rent CPUs, access computing resources, set SQL database rules, storage modules, and REST APIs. Together focuses on all steps necessary for building AI companies, including pre-training, fine-tuning, and inference, while providing sandboxes for agents.
Key considerations include cache memory hit rates when running programming workloads, time between each turn, how long to wait before responding, response speed requirements, and speed for completing entire processes based on SLA requirements.
When a request arrives via REST API, it passes through typical internet components including Cloudflare and DNS before reaching data centers. With thousands of GPUs and KV cache memory, the first step is request routing at the system level, selecting which GPU to send the request to.
High-level human text must be converted to zeros and ones for model processing. The processing loop includes the large payment with prefill and decoding stages. Final tokens are translated back to human text and sent to the user.
Load balancing must consider not just distributing traffic across data centers and GPUs, but also sending requests where KV cache memory already exists. Sending the same request to the same GPU repeatedly yields better performance because previous processing work is already done.
Requests come with tool calls that the model needs to evaluate. Tool calls are mathematical operations mixtures (matrix multiplication and attention units), so the model cannot actually search the internet or perform actions. When submitting requests, users provide a list of ten things the model can do, and the model must determine whether to contact online services or search within available fields.
The scheduling loop accepts all incoming requests and waits to batch them together, either waiting for an empty GPU or empty cycles, or waiting for previous requests to finish. Scheduling choices determine total yield, with tradeoffs between improving performance for more users on the same GPUs versus improving very fast response times.
Prefill is considered immediate processing where the complete instruction set must be accurately and specifically identified and understood. This is computationally intensive with high GPU utilization and high arithmetic density. Decoding generates one token or a few tokens at a time, requiring downloading entire model weights from GPU memory to the computing unit for each token generation.
Decoding is heavily memory-dependent and completely different from prefill. Significant time is spent thinking about how to serve these two different types of workloads efficiently, including separation, parallelization, and kernel utilization.
Six approaches to speed up inference include:
- Kernels: GPU programs that can be optimized by writing different attention points, MoE kernels, and mega-kernels
- Disaggregation: placing prefill and decoding on different GPU units
- Parallelism: dividing large models across GPUs
- Routing: determining which parts receive requests once they enter the data center
- Layer improvement and speculative decoding: using smaller models to anticipate larger model outputs
- Quantization: reducing bits needed to store model weights
Chimera 3 is cited as containing 3 trillion parameters in FP4 at half a byte per parameter, totaling 1.5 terabytes. The best B200 and B300 GPUs have only 280 GB of memory, requiring model division across GPUs.
The importance of the kernel lies in its essence, involving pre-packing and unpacking basic cryptography for the account. This is where calculations take place, including matrix multiplication, attention mechanisms, and others. When thinking about the model, parts that are easy to improve relatively speaking can be found, while other parts are difficult to improve. For example, all standard normalization layers and existing RMS standards in the model, in addition to Softmax functions, exhibit very low computational intensity. Historically, this is one of the reasons that only 50% utilization of calculation units is achieved from beginning to end during training, and 5% during decryption.
The method followed to achieve this has been reviewing the model and attempting to identify each weakness, then assigning a team of talented core engineers to improve it. For a long time, this was the work style followed in this field, where a group of really smart people work on this project together.
Recently, the approach has shifted toward using agents to improve this part automatically. This is an internal project called Together Compiler, which will be discussed in detail in the coming weeks. The name will likely change upon release, as Together Compiler is not a translator. The basic idea is that with a target model, target device, and specific data size, agents will be used to improve this internal part.
The main point is that it is truly necessary to operate this part in real traffic, because different traffic or programming workload versus chat workload will activate the kernels in a completely different way. It is very important to conduct a comprehensive research loop from the beginning. Looking at what is being done operating in a production environment is used for replication.
What is being built here is roughly the beginning of an RSI loop, or self-referential reasoning optimization loop.
All weights are divided and so large that they cannot be placed on a single GPU, and if the model is divided across multiple GPUs, more computing, more frequent bandwidth, and more of all desired things will be obtained.
Basic parallelism schemes that existed before include tensor parallelism, which simply involves hitting matrices and dividing them, then performing a basic MapReduce-style reduction process at the end. Linear parallelism is the idea of dividing layers of the model layer by layer, where full different layers are applied to different GPUs. Data parallelism is another idea, where different sequences are sent to different processing units, with similar model parts preserved.
The challenge has always been the amount of memory available on GPUs, the speed of writing kernels to support a specific parallel scheme, and the available resources.
One thing that gained great popularity recently, in which intensive investment was made, and one of the basic engines behind DeepSeek's adult number $38 per hour, was the benefit from much larger expansion ranges. Blackwell chips and GB chips are relevant here. The biggest difference is that Blackwell chips, and all previous Nvidia chips, allowed only eight GPUs to communicate with each other with high efficiency, meaning the model is divided into eight parts. Attempting to divide into more than that, for example up to 16 parts, uses much slower bandwidth, which makes the service extremely slow.
Nvidia recently began selling and manufacturing shelves that can connect 72 GPUs with a much higher frequency range, allowing distribution of all experts in the model across different GPUs. This means larger quantities can be served on the same GPU. The main reason is the idea of calculation density. If a single GPU processes only an expert in data on the GPU, serving a large number of symbols, it now appears like a huge array multiplication operation, which is a process that GPUs excel at implementing.
People know these ideas from a few years ago, with presentations on expert parallelism going back two or three years. Recently, the focus has been on making this more reliable. GB racks are truly wonderful, but often malfunction in a node or GPU at different times. If a replica based on one model on 72 GPUs has one malfunction, without appropriate processing mechanisms or reliability elements, the disruption of one GPU might cause loss of computational ability equivalent to 72 GPUs, which is undesirable. Time has been spent attempting to find a way to make this system reliable, able to endure malfunctions, and continue smoothly even when these malfunctions persist in devices.
Quantization algorithms represent hundreds or thousands of research on this topic. The basic idea is that models will be trained using something like BF16, that is, 16 bits per operator, and 2 bytes per operator. A three-trillion-factor model will require 6 terabytes. The beauty of automated learning is that the calculation is all inaccurate. We don't necessarily need all of this accuracy, as the correct answer can be obtained in the end.
Algorithms exist that can be used to reduce 16 bits to 8 or 4 bits while preserving good quality. Lots of technical details exist, but what matters is how to ensure appropriate quality and ideal user experience. Obsession with this exists at Together. Knowledge exists about what the optimal quantization system is for devices and what will achieve the highest productivity and largest number of symbols per GPU. At the same time, recognition exists that people use these technologies increasingly for building and operating extremely important systems.
A quality measurement testing loop is being built at Together. Before sending data, any model for any user, internal testing of this model is conducted internally for a long time. This model performs several functions. It will discover all calling errors in the tools discussed previously, because if the cloud system stopped working suddenly, about searching on the internet, that would be noticed as a user. Notice will also occur if the model was not with the required intelligence, or if it doesn't get correct grades in standardized tests or in the overall assessment.
A small Flappy Bird loop is kept running, where everyone tries programming Flappy Bird models in a quick loop. If something went wrong during the quantization process, Flappy Bird will stop flying, and it will be necessary to know the error and correct it. The essence of the matter here is the existence of many algorithmic improvements, where many algorithmic innovations can be done, but in the end, it leads to a rigorous evaluation cycle truly. These include the performance measurement session, practical assessments, and many internal experiments.
A success story exists from working in depth with many open source model providers. Work was done with MiniMax presenting their M3 model, as well as the upcoming M 3.1 model. MiniMax was facing some problems in quantizing their model in NVFP4. Interest existed because knowledge from the kernel perspective, from the presentation perspective, and based on general experience indicates this is the appropriate quantization to use on Nvidia GPUs.
The original color is light purple, representing the model in FP8. The bright pink color in the middle was MiniMax's attempt at quantization using their private recipe. Using recipes in the orange color on the right, model performance matching the original was achieved, then introduction was possible, granting MiniMax more symbols, helping them close the loop from model training to submission in a profitable way, helping them generate revenue, and get the money to train their next model.
The order flows through the order, including direction to which copy of the request is sent and which data center. Fragmentation exists, involving ensuring that tool calls and all expected parts are correct, waiting lists, collecting basics, scheduling work, and pre-encryption and decryption mobilization often on different GPUs with different structures and different parallel schemes. Once all of that is finished and the GPU is proven due to its merit, distinctive symbols are obtained once again, enabling writing code instructions and chatting, and all other known and loved things.
Open performance standards are available, some of which are extremely saturated, like those exceeding 90%. The important thing is that if any change is noticed, that means the model will face difficulties in dealing with some real-life activities. Performance standards are published and discussed, looking at class differences individually, that is, the amount of actual change in stimulants. The fully automated workflow is also examined. Hundreds of different test cases have been created, where any error in any model observed is processed, and safety is ensured. Automated evaluation loops like Flappy Bird are also used, bringing it back to the complex project so that the model improves its own kernel, and if the model for some reason was no longer able to improve a special kernel, attention is directed there.
NVLink technology enabled Grace Blackwell chips to contact more than eight GPUs in image form. Each tray of four GPUs is equipped with very thick cables reaching a large switch, then delivery of all trays that way. Data dates via this thick cable, then across the converter, and then all reduction processes and dismantling can be done. The jump from 8 GPUs to 72 is admired and used. If batteries were wanted, this would be the most suitable option.
Many quantization recipes require training the model after finishing it. These are not liked because they are only found if the complete original training dataset is owned. Unless the model provider is owned, this will not be owned. When doing so, the model might forget some things. This is not an approach requiring training on launch. It's a mixture of things that are publicly available. Some calibration data exists. A research paper titled "The scale we published before a few months" was used on a wide scale, but it's a mixture of extreme caution in recipes, and trying to combine all different parts. Hundreds of quantization methods exist that require no training. It's about being cautious with extreme caution regarding the methods used.
Exciting attention is paid to all new structures. During PhD studies, work was done on some of them, such as SSM models and similar, and what was always heard means that someone will never use this thing. It's a dead end, transformers will control the world and transform general AI and all these things. After a few years, all of these technologies are used in all large models. Kimmy K3 contains something called KDA, which is in the base layer an SSM. It may be somewhat controversial, but it is a linear SSM layer, attention pattern layer. It is used in one of the best open models. Scattered attention is used, which someone did not use for a long time. DLM models are used in all guesswork models. This is the best that can be done now.
The interesting thing is that all these things are enjoyed with great interest in ring patterns. The idea here is, as learned, in this calculation graph, resend the unique code via the same layers several times, then training operations can be increased without increasing weights. This was believed to be a strange idea, and someone won't use it. Then it became clear that OpenAI and Anthropic at least are already using it. The time is really interesting, so if a good idea now about structures is asked, and the hard part to prove that it works is done, it will appear in GPT-6 or GPT-7, and submission time observations and impact have become very fast now.
Looking at the picture a little wider, talking about how open model code differences and how thinking about a world that might contain, as known, the list in François's show presentation shows the number of guaranty companies, the leading group only the ones that exist today. In the foreseeable future, there will be hundreds, if not thousands, of potential inference providers, and how measuring quality through these suppliers if a developer is trying to build an application that relies on open models.
Ollama is an open-source project, originally launched in 2023 to operate open forms locally. At that time, the most commonly open models, such as Llama for example, models of smaller size possible to run either on a single H100 device or on a laptop or small server. The goal of the project's foundation was always about providing a developed experience to OpenAI standards and Anthropic, but for open models. Since then, the project has grown to a large extent, and because the size of the models has become larger, it grew larger, and reached its peak with the beginning of this year when Kimmy and Jill M launched MiniMax huge models ranging its parameters between 500 billion to a trillion. Ollama cloud was launched to provide the same developer experience that they obtain locally, but in the cloud. This service has seen huge growth during this year and became almost one of the largest independent cloud computing services, if not the largest, to reach open models in the US and Europe.
This matter is important because time is spent long thinking about how to provide a wonderful development experience via many different models and across many service providers. If thinking about the structure of the cloud service, on traditional inference providers, focus is indeed on some matters, like getting the latest models, ensuring extremely high speed, and ensuring a reliable experience. Many of these internal operations are located in the router Dan talked about just now. This is done for several reasons. Reliability and accessibility of early models are the most important, because every service provider that excels in service different models. But in the end, this structure is very similar to the structure of many Ollama customers. Ollama is published by a large segment of Fortune 500 companies. The cloud computing service, and what was noticed through talking to the teams that move from closed models to open models due to cost or control or for other reasons, is that they all face a similar problem.
The main reasons for guiding inference are that in the ideal situation, the best of all fields is obtained through integrating a group of reasoning providers together. Provider definition is somewhat loose. This is a problem that may be faced if a reasoning provider has different deployment processes on different devices in different areas from different teams using different inference engines, for example. When adopting such a structure when thinking about assembling a group of reasoning suppliers together, some challenges exist.
The biggest challenge in short is that everyone doesn't provide models in the same way. In reality, this is a good thing, because different teams excel in different performance improvements. Some people are working on specific types of devices. An inference provider with a new feature works on Mac mini devices all over the world. If thinking about it, customers are interested in several different aspects, like cost, intelligence, speed, then area and privacy. The challenge in directing requests via a group of service providers is that one bad provider can spoil the entire group. If one provider has poor performance in one of these four criteria, a poor customer experience level might be obtained. This may be for many reasons, including model accuracy. A provider might be overcharging, estimating the model, or running on defective devices, or using a different version of weights.
Tool calls are a major area of concern, as Dan mentioned. Tool calls effectively command extremely difficult. Despite seeming simple because it resembles JSON parsing and text strings, except there are many problems that may occur, starting with recovery from tool call failure. Imposing specific diagrams of different tools, parameters, and tool names, and ensuring that the analysis process is performed correctly at the microchip level. Providers may use different sample acquisition techniques, as well as inference engines having great depth in modification possibilities. Whether it is the custom kernel or knowledge base conversion accuracy, then many of these changes only appear in very special cases, like reading a very long context. Simple things can also happen, such as displaying models with incorrect names. The thought might be reaching a specific model, while it is in fact a different model, but it's difficult to know that. All of them are tools that help easy usage. Many people may have spoken errors in the display system, which could have a negative impact on performance. A broad customer base service across regions and different suppliers may make it really difficult to follow that.
Time was spent before this lecture attempting to understand what is done, and what the sector's action is in this matter. It turns out it's not a new problem. Many research has been conducted on the size of this problem. Excitingly and surprisingly, talking is done about the gap between the open model and the closed model, and the extent of open model delay. It turned out that regarding open form service only, the gap between the worst and best supplier is in fact from the gap
The experience gap between closed and open models represents a significant problem. While closed models from providers like OpenAI and Anthropic deliver consistent experiences, open models from different providers can vary substantially even when weights are identical. This variability stems from differences in inference implementations, hardware configurations, and optimization choices across service providers.
In the open model ecosystem, users expect consistent experience regardless of access method, but this expectation is not met. A 2024 research paper tested 31 different endpoints for Llama models and found that some providers' implementations performed closer to GPT-3.5 than other implementations of the same Llama model with identical weights. The problem becomes more severe with programming agents and long-horizon workloads that involve successive calls, where even small deviations compound across multiple steps.
For programming tasks that typically require 15 steps, a 10% deviation from the intended model behavior can result in an error rate reaching 79%, compared to the designed error rate. This compounding effect makes endpoint consistency critical for agent-based applications, as the final error rate can reach 80% when starting with seemingly minor variations.
Comprehensive evaluation through standard benchmarks across different service providers is expensive and time-consuming, making continuous monitoring impractical. Moonshot AI released Gemini K2, a tool verification benchmark that measures service provider performance on tool use accuracy, creating competitive pressure for accuracy. MiniMax implemented similar approaches for their models.
Log probs provide a low-cost method to measure deviation by examining the probability distribution of the next token and comparing it to a baseline. This approach requires only a single request and can detect silent changes in service structure, such as engine upgrades or new inference engine releases like VLLM or SGLang. However, log probs are not available through many APIs.
Examining request patterns without data retention allows detection of issues like models entering infinite loops by monitoring tool call patterns, error rates, and request shape changes over time. Providers may offer different input/output lengths that indicate potential problems. OpenRouter implements similar monitoring with their ExaML model.
Open model quality assurance differs significantly from closed models, where companies like Anthropic invest heavily in quality control, including TPU core error detection and comprehensive quality reviews. For open models, laboratories typically release weights on Hugging Face with software models, but not the actual inference systems used in production.
DeepSeek launched Deep Infer, providing their inference system that allows users to run DeepSeek weights at $38 per hour on their own GPUs using consistent infrastructure. This enables new inference providers to deploy with confidence in quality. Many laboratories now work with inference specialists to improve model quality before release, ensuring API-offered models match the open versions.
Model laboratories now commonly release quantized versions alongside original models, enabling low-precision API access. Some tool calls can be integrated into systems with high accuracy consistently across providers rather than delegating this task to individual providers. This applies similarly to reasoning and extraction tasks.
Advanced verification methods include GPU fingerprinting, where each modern Nvidia GPU comes with a reliable implementation environment that can verify weights used in the serving program, providing cryptographic proof that the service is running correctly.
For long-context workloads like multi-turn agents, pre-filled acceptance may be preferable over post-generation acceptance. Log prob measurement can be applied over time rather than across specific context lengths, with gradual testing of long requests to measure deviation in log probabilities as context length increases. Some providers impose maximum context window limits, and maintaining token accuracy consistency becomes critical beyond 200,000 tokens.
Service providers competing on price may optimize for specific traffic patterns or use cases. Custom test collections designed for specific client use cases or data patterns can reveal provider-specific shortcomings that generic benchmarks miss. These tests can be run continuously and provide early detection of quality issues before customer complaints surface.
With launches like Flint and Richard Socher's repetition models, local models are experiencing renewed interest. MLX framework runs Llama on Apple Silicon, while Colibri operates Qwen 3.5 MoE locally from disk. The future likely involves hybrid approaches combining local and cloud models, as new high-capacity local models can handle tasks previously requiring cloud models.
Luminal specializes in translations between GPU operations/clocks and tokens, with CEO Joe emphasizing that translator quality directly determines profitability. Their approach uses saturated equality technology, which they believe represents the largest industrial application of this relatively modern technique for efficiently assembling models across different device types.
Nvidia's $4 trillion valuation largely stems from its excellent CUDA software ecosystem. Converting models to complex devices remains an unsolved problem. Luminal conducts extensive search operations during assembly, maintaining rules for rewriting complex or reasoning methods used by regular translators, eliminating the need for maintaining large kernel libraries.
Saturated equality reduces all neural networks to approximately 14-15 basic operations including addition, multiplication, and similar operations. PyTorch's 2000+ operations can be reduced to this simple set without information loss. The approach is PyTorch-complete rather than Turing-complete, meaning any model runnable in PyTorch can be understood and compiled across all supported backends.
Instead of comprehensive linking, saturated equality creates intermediate linking that connects PyTorch operations to device operations through RISC-style initial processes. Super-optimization searches all possible program forms using cost functions to determine optimal shapes, eliminating the need for translation engineers to know specific conversion methods for each target processor.
The system uses Graph E representation, applying non-destructive rewriting rules that add side information rather than replacing matched patterns. All rewriting rules are applied in complete possible arrangements until saturation is reached, expanding the electronic graph to maximum limits. From this saturated state, executable graphs are extracted and measured for execution time optimization.
The system uses a graph-based representation for algebraic expressions. The root expression is represented at the top node, with arrows descending to represent different operations. For example, a2/2 starts with the division operation, where one branch represents the multiplication and the other represents the divisor. Through application of different rewrite rules, the graph expands until reaching a saturated state on the right side of the diagram, showing that a2/2 is equivalent to a alone.
This simplification process requires no prior knowledge of the expression—it relies on systematic rewrite rules written in Lisp, chosen because the engine developers prefer Lisp. The rewrite rules follow a specific grammar format, with the first rule being commutativity: x + y = y + x. This property does not apply to IEEE 754 floating-point numbers, where associativity and commutativity are complicated due to floating-point semantics.
Constant folding is implemented where if a + b is known at compile time, the values are combined and the addition operation is eliminated from the program. This optimization happens at translation time rather than runtime.
The intermediate representation serves as a common simplified representation for all backends. The system targets multiple backends including Tensor Processing Units (TPUs), Tranium modules, and Graphics Processing Units (GPUs). Device interfaces are defined by specifying the range of operations each device performs, which are typically low-level operations.
For a GPU, Tensor Core MMA is an example operation that reads inputs from SRAM and writes outputs back to it. New backend device interfaces can be added by defining the meaning of each operation in terms of equations the translator already understands, then writing simple rewrite rules that connect to previously defined equations.
The intermediate representation is multi-level by nature, modeling different levels of GPU programming explicitly. This includes programming individual threads, groups of threads, thread blocks (CUDA kernels), GPU cores and SMs, the entire GPU, or even entire nodes for specific operations. This allows incorporating operations from the community or vendor libraries at different levels.
Equality relationships between operations link all different meanings logically, allowing the engine to determine what is equivalent to what. The search engine can then replace sets of different operations. Examples include using cuBLAS host operations called from the CPU to perform full matrix multiplication, or using thread-block level operations from Cutlass, CuBLAS DX, or ThunderKittens from Hazy Research.
High-level thread-block operations program a single GPU kernel at a time. The translator can analyze these different operation levels, charge sub-graphs of lower-level operations, and integrate them into larger kernels. Forward passes can be performed in both small kernels and large kernels.
The final problem is searching the identified search space: determining whether program A is better than program B requires a cost model to evaluate programs. Creating accurate cost models for complex devices like modern GPUs is extremely difficult, requiring significant PhD-level research to predict program runtime without actual execution.
As a small company, accurate cost models cannot be developed for all target device types. The solution is to eliminate the cost model entirely and simply run measurements on real devices that will be used. Different moves in the search space may not appear useful to programmers but prove very useful when run on actual GPUs.
Search space reduction techniques include evaluating specific sub-sections of program performance and early pruning when a candidate is known to be slower than something already measured. Initial modifications are performed, but most measurements are done on real devices.
The company has achieved the rare feat of solving the profitability problem for compiler companies. Many compiler companies have gone bankrupt because they couldn't find ways to profit from translations. The company now runs a cloud supported by a variety of devices, handling approximately 80 trillion tokens monthly and millions of images monthly.
As software compilers, they can run not just the most common models but all non-traditional model types including protein folding models, biological models, and image models. The company is growing rapidly with currently eight people and actively hiring. The mission is using advanced compiler technology to make advanced devices available to everyone.
When adapting to machine learning workloads, IEEE 754 compliance is not required since machine learning doesn't care about that level of precision. Different sequences can even be used at the device level. Commutativity properties are important because they affect numerical accuracy—reducing a sum or using tree-based reduction yields different answers than serial sum reduction.
Numerical accuracy errors significantly impact output code quality for the provider model. The system is designed to be very clear about numerical precision handling. Some rewrite rules remain helpful because devices often implement functions with input A versus input B from different memory locations—for example, Nvidia's Tensor cores may consume SRAM input A only, with input B from another memory like Team M.
Even a simple bank like TRT is extremely slow even with this inference method. Forward pass analysis can be performed and repeated in less than 10 milliseconds, enabling approximately one thousand analyses per second. The search space expands significantly—approximately tenfold—using standard search algorithms for navigation, with heavy use of genetic algorithms.
The search covers only 1% of the area and targets local optima that match global optima within 99% ratio, making discovery easier. Compilation time involves a clear tradeoff: inference clients don't care about compilation time if it enables faster runtime. Since compilation happens once then the program runs on millions of GPUs, improving runtime performance by 1%, 5%, or 10% matters more than compilation taking a full day.
Users can define time budgets for compilation (one hour or 10 hours), reaching diminishing returns at different points. The search can expand well initially but reaches diminishing returns after some time, though it ultimately works well.
Real reasoning cookies are used to guide the search rather than data-only research. Since the company manages a cloud environment, complete types of data that matter to customers are known. Tensor shapes are roughly known in advance, along with data distributions. Data verification is not needed unless rewrite rules are wrong.
Representing shapes in industry-relevant tensor forms yields very precise timing. Debugging is extremely difficult, especially when determining which class faces problems while trying to reduce operations. PyTorch tools are used for high-level operation rating, with metadata followed through the compiler. Standard PyTorch performance analysis tools identify bottlenecks and problems.
Saturation testing quality through Egglog allows determining if a program has false output. Evidence is needed linking output programs with input programs, explaining the sequence of rewrite processes implemented for the transition. Invalid rules can be identified quickly.
Non-supported operations are re-enacted smoothly to the central processing unit. When expanding processes, operations are merged at the bank level. The bank looks into different discount forms and obtains integrated final operations. If a kernel transfers data to and from DRAM much more than another kernel, it is probably bad. Intermediate-level patterns expand, then the search process must find and reject these patterns again.
The working language is not Turing complete, meaning Rice's theory and similar limitations don't apply. Programs can be analyzed for certain properties without execution. This was realized early because hyper-optimization failed in the 1990s when attempting application to general programming languages like C, which is Turing complete with a nearly infinite number of possible translation processes.
Algebra programs with dense linearity are translated, representing a very narrow range compared to all possible programs. PyTorch is complete but not Turing complete. One exploitation is knowing the exact byte movement amount between memory spaces, enabling analytical performance limit deduction without real execution. This enables significant early pruning, requiring far fewer stereotype iterations compared to the entire search area size.
The solution space at the kernel level does not include other technologies like speculative decryption. Only forward passes are handled—graphs are defined for forward passing and compiled from beginning to end within a single forward pass range. Work occurs inside inference engines like vLLM and SGLang, making use of great solutions they offer for speculative decryption directly.
The hypothesis is that the majority of experimental phases lie in handling extremely complex parallel structures and scroll complexity in forward passes, leaving remaining innovation to VL and SG length. The system does not search at this level.
SGLang development began in late 2023 when only Llama models were available as open-source models. Workload nature had to be guessed, imagining tool summons, parallel branch agents like future cloud code. Sub-agents in cloud code would have large contexts sharing the same key-value group.
When monitoring inquiry results, each wave entered was very short, checked multiple times in the engine, then extracted multiple times. No efficient way existed for the engine to reuse pre-prefixes. A radical attention mechanism was proposed that could automatically reuse common prefixes and calculate using new prefixes and subsequent new ones.
All agent contacts are managed within the engine system memory exactly like a tree, called root tree in the Astra application. Memory management involves shared preparation with long-term system guides—one under ratification, another under review. Two agents share the same guide for the same tasks within the same context while performing different jobs.
One agent may want to show some results, and given limited memory, the removed proxy suffix is deleted. This basic memory management method forms the root tree in the management system. All fundamental designs, including scheduling, revolve around this topic.
During evacuation processes, the policy evacuates unused papers when memory is required. Multiple algorithms exist, including scheduling algorithms for evacuation suited to workload and needs. Claims or agents sending to engines interfere in intertwined ways. Claims can be rearranged and collected in batches with memory caching-conscious arrangement, reusing context A instead of overlapping A and B.
The original paper idea was building a red tree within the memory management system. Engine development for universal LM reasoning occurred in mid-2024 with version 0.2 supporting Llama 8B2 processors and Llama 4.405B for both online and offline communication.
In all assembly operations, memory management, and evacuation compliance, all expected orders must be sorted and priorities determined. GPU inference time is constantly improving while CPU processing efficiency remains unchanged. Traditional scheduling creates first batch preparation, first batch operation, data retrieval, then second batch preparation.
When GPUs were rather slow, GPU bubbles were generally acceptable within 10% limits. As GPUs became faster, CPU bubbles and GPU bubbles became noticeable due to GPU bottlenecks. An interference scheduling system was designed, delaying all devices hosting synchronization and leaving them for another round to ensure no bans on the critical path.
Preparation occurs in advance while the GPU is working without hindrance. Once delays are implemented, no obstacles prevent releasing the next payment. The second batch runs during first batch operation. This approach continues with many new engines including the DeepSeek engine released with Deep Infer using "zero" scheduling "above zero, above life" to support their internal inference engine.
In Hopper days, higher productivity rates reached only 1.5 or 1.1 times compared to traditional scheduling, but in Blackwell days, rates sometimes reach double in GB settings.
GPU memory stock remains limited when providing service for long contexts. Users submit requests then send responses following in minutes, hours, or even days, requiring places to restore previous beginnings. High-temporary storage memory technology is used, standard in many engines.
Records mentioned previously are linked in GPU memory units, host memory, and internal storage called L1 and L2 buffering L3. Each can have a distinctive symbol in L1, L2 buffering, or L3 memory. Each memory can be managed with policies specifically designed to increase memory caching access rates and reduce download time from L2 cache to L1 or L3 to L2. This reduces response time by 56%.
Traditional transformer architectures are no longer suitable for long-term use or effective deployment. This limitation has driven the development of more innovative transformer structures, including sliding window attention mechanisms that are used across many open-source models and some closed models.
Recurrent architectures like Ring Attention and KDH cases store information in fixed forms, processing symbol by symbol. While it might seem straightforward to add different layers and complex memory types to execute complex operations with hybrid memory storage and computing, this approach presents significant challenges in building participation strategies for cache memory.
To achieve cache success at position 10, all KV codes must be matched to obtain complete attention. With a sliding window size of 4, at least 4 symbols must be contained, starting from position 10 down to position 7. If the total number of saved symbols reaches 8, temporary memory storage success becomes impossible because new attention calculations require the full sliding window. For prefix sharing starting at position 10, a KDS member must be present exactly at position 10.
Converting all memory groups into separate entities creates major challenges in efficiently sharing cache memory. Sharing cache between 95% to 99% reduces costs by five times, not just 5%, because it uses four times fewer arithmetic resources for all inference operations.
Previous approaches used different radical trees, such as Tree A, implemented separately without considering low-level memory groups. This caused numerous compatibility and flexibility problems. When protecting transaction logs or attempting to lock prefix nodes, extensive coordination was required across all layers, sliding window layers, and recurrent layers. Each component needed access to specific protocols to coordinate and meet foreign policy requirements.
The Redux unified tree was developed as a solution that doesn't treat all components as independent elements. Instead, it uses semantic symbol tree concepts based on complete symbols, which remain available regardless of system structure changes. The Redux unified tree serves as a control unit and coordinator for each component to ensure optimal efficiency in memory cache matching. Each component is designed to match the unified Redux tree with its own rules for matching, evacuation, and memory management.
Different users and workloads require vastly different context sizes. User A might use contexts of approximately 4000 tokens, while another user might need 128 unique symbols, and yet another might require contexts of a million symbols. These varying requirements mean workloads need different proportions of complete symbol sets versus sliding window attention, which only feeds the last 128 symbols for specific workloads.
Instead of allocating all pre-calculated percentages and fixed sizes for pre-load work, a single actual memory group is allocated without specifying limits for each memory type. No limits are set for 4K memory groups or KV values for windows. Memory is treated as groups of smallest units like bytes, growing constantly from both left and right sides of actual memory based on usage patterns.
When contexts are lengthy, more memory is allocated for complete symbol sets. When contexts are short, more memory is dedicated to recurrent cases and window KV values. All operations must be constantly present in actual memory in most backend cores and applications. Pre-allocating and evacuating memory beforehand creates memory gaps in GPU processing units that cannot be reused once boundaries are reached.
A "pressure" operation was developed to reorganize all memory parts in actual memory space by collecting them using low-access-time memory copy kernels. After reorganization, all virtual symbol identifiers (actually memory page identifiers) are reset, followed by re-assignment while keeping virtual page IDs that don't need changes. Only the lowest-level assignment process is modified, providing greater flexibility for the entire task.
Reasoning miles are offered within training loops, developed primarily from XLA and Miles by Red Hat Arc. XLA is responsible for publishing operations and serves as the ideal backend interface for Miles RL training. Miles supports non-fully synchronous implementation, preserving continuity of generation and training operations.
Development began with Red Hat's transient interest, leading to the creation of an open-source inference engine, Miles development, and XLA publishing. Multiple initiatives were undertaken to strengthen open-source artificial intelligence infrastructure. An XLA Summit will be held in the coming months featuring distinguished speakers, with registration available via QR code scanning.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
