Chip design from the bottom up – Reiner Pope
In a Nutshell
The video explains that matrix multiplication is built from simple multiply-accumulate operations at the gate level, and that efficient AI chips minimize expensive data movement by baking larger matrix operations into fixed hardware like systolic arrays. This approach shifts design focus from individual operations to sizing decisions around register files and compute arrays, trading flexibility for efficiency. It also contrasts GPUs' many small, flexible units with TPUs' fewer, larger systolic arrays, showing how architecture choices affect data movement, clock speed, and overall performance.
These notes were generated by AI and may contain inaccuracies.
The discussion begins with the smallest fundamental unit of chip design: logic gates, which are simple operations like AND, OR, and NOT. These gates are connected by wires that must be physically laid out as metal traces on the chip. The primary computation AI chips perform is matrix multiplication, where the core operation is a multiply-accumulate of pairs of numbers.
A four-bit number multiplied by another four-bit number, with the result accumulated into an eight-bit number, is used to demonstrate the process. This multiply-accumulate is the natural primitive for matrix multiplication because a matrix multiply consists of nested loops where output[i, k] is repeatedly updated by adding input[i, j] multiplied by other_input[j, k]. At every step of this process, a multiply-accumulate occurs.
In AI chips, multiplication typically uses low-precision numbers, while accumulation requires higher precision because errors accumulate quickly during summation. This leads to choosing four-bit multiplication paired with eight-bit accumulation. When summing many numbers, rounding errors build up over repeated additions, whereas a single multiplication introduces fewer such errors.
To perform the multiply-accumulate by hand, long multiplication is used. The four-bit number is multiplied by each bit position of the second four-bit number, producing partial products that are shifted appropriately. These terms, along with the eight-bit accumulator value, are then summed together. This results in a five-way sum.
Each partial product is generated using an AND gate, which outputs 1 only when both input bits are 1. For a p-bit by q-bit multiplication, p times q AND gates are required. The bulk of the work occurs in the summation step.
The largest logic gate typically used is a full adder, which adds three single-bit numbers together. A full adder takes three bits as input and produces two bits as output (a sum bit and a carry bit), functioning as a 3-to-2 compressor. It counts the number of 1s in a column and expresses the result in binary.
The summation proceeds column by column from right to left. Full adders are repeatedly applied to groups of three bits in each column, replacing them with two output bits. This process continues until only a single number remains. This approach is known as a Dadda multiplier, the standard method for area-efficient multipliers using full adders.
Starting with 24 input bits (16 partial products plus 8 accumulator bits) and producing 8 output bits requires 16 full adders, calculated as 24 minus 8. In the general case, a p-bit by q-bit multiply-accumulate uses p times q full adders.
The multiply-accumulate structure yields simple algebraic scaling: p times q gates for both ANDs and full adders. This clean relationship is one reason the primitive is favored. Every atomic step in the manual calculation maps directly to a logic gate connected by wires.
AI chips must decide how much die area to allocate between different precisions such as FP4 and FP8. These circuits are not inherently fungible; designers balance customer requirements against power budgets. When Nvidia reports performance numbers, the ratio often reflects equivalent die area allocation plus data movement considerations.
Smaller bit widths are especially advantageous because area scales quadratically with bit length. Historically, halving precision doubled throughput, but due to quadratic scaling, the actual benefit is larger. Nvidia's later specifications (B300 and beyond) reflect this, showing FP4 as three times faster than FP8 rather than exactly twice as fast.
Prior to Tensor Cores, GPUs and CPUs used a register-file-plus-ALU structure. A register file holds multiple entries (for example, eight 32-bit values). The multiply-accumulate circuit reads three arbitrary registers, performs the operation, and writes the result back.
To select any register, a mux is required. An n-input mux operating on p bits uses n times p AND gates and (n-1) times p OR gates. For three inputs to the multiply-accumulate, the data movement cost is 3 times n times p AND gates. With n=8 and p=4, this yields 24 times p gates for data movement versus only 4 times p gates for the actual multiply-accumulate logic.
The mux cost dominates because reading and writing the register file accounts for the majority of circuit area, while the logic performing the actual computation is comparatively small. This imbalance motivated the shift to more efficient architectures.
The problem was addressed by baking larger loops into hardware. Instead of implementing only a single multiply-accumulate, an entire matrix-vector multiplication loop is fixed in silicon as a systolic array. This increases the granularity of the fixed-function unit, reducing the relative cost of input and output data movement.
Two benefits arise: more computation occurs per register-file access, and certain values (the weight matrix) can remain fixed and be reused locally.
A matrix-vector multiplication is performed by multiplying each column of the matrix by the input vector and summing the results. Each output element is a dot product. In the systolic array, multiple multiply-accumulate units are arranged spatially so that dot products are computed along columns.
The weight matrix is stored locally inside the systolic array in registers, allowing repeated reuse across many input vectors without repeated fetches from the main register file. This achieves quadratic compute relative to linear communication for the inputs and outputs.
Weights are loaded slowly via a daisy-chain mechanism. A number enters the top row of the array on one clock cycle and shifts downward on subsequent cycles. This approach limits the wiring crossing the systolic array boundary to roughly x units of bandwidth rather than the full x times y matrix size.
For y consecutive clock cycles, the top row is brought in each time while shifting all other rows down by one. This approach reduces the wiring that needs to come from the expensive register file to a factor of x rather than xy. There are two questions in terms of communication: communication time and communication bandwidth. Since the data is only loaded in once, the goal is to minimize bandwidth, because bandwidth equals die area. The data is loaded in slowly over smaller lanes because the value will be kept in the register file for a while.
This principle connects to optimizing inference across many chips, where the high-level goal is increasing the amount of compute per memory bandwidth, or per communication. The same principle applies here: maximizing the amount of actual multiplies or additions relative to transporting information from registers to the logic. In both cases, the objective is to maximize compute relative to communication. This concept appears throughout the entire stack, from high-level architecture down to the gates themselves.
A version of the compute-to-communication trade-off exists even closer to the gates in the precision of the number format chosen. The same squared versus linear term effect appears both in the precision of the ALU and in the size of the matrix.
The systolic array sits on top of the multiplication circuit. Older TPUs were described as 128x128 arrays of this circuit. This architecture ends up being the most efficient known circuit for implementing a matrix multiply.
Most decisions in chip design are sizing decisions. AI chips all contain a systolic array with a nearby register file providing inputs and outputs. The key sizing questions are how big to make the systolic array and how big to make the register file. These two decisions are coupled.
One approach is to set a budget for what percentage of chip area to spend on data movement. For example, allocating 10% to data movement and 90% to the systolic array. Bigger register files are more flexible and allow more application-level performance, but they take away area from the systolic array.
Chips are incredibly parallel, containing 100 billion transistors. With massive parallelism, synchronization between parallel units is essential. In software, synchronization typically uses expensive methods like mutexes. On chips, a different approach is taken: every nanosecond or so, all circuitry pauses momentarily and synchronizes. This is the clock cycle. The entire chip typically goes in lockstep to the next operation.
In circuitry, the clock is mediated by registers. The structure consists of storage holding a bit, a cloud of logic such as a systolic array or multiplier, inputs feeding into the logic, and an output register. A global clock signal drives all registers. When the clock strikes, whatever value is on the wire at that instant gets stored.
The goal is to run the clock speed as fast as possible. Running at two gigahertz yields twice as many operations per second as one gigahertz. This makes the design sensitive to the delay through the logic cloud, since any computation must finish before the next clock cycle. A major optimization point on any chip is minimizing this delay.
The constraint is that adding too much logic risks missing the clock cycle, while adding too little leaves potential compute unused. In standard chip design, timing is margined such that there is a probability of missing the deadline, but it is many standard deviations out. For practical purposes, the part is reliable and will always meet the clock. Exceptions exist at clock domain crossings, where probability must be explicitly reasoned about. In the main path, timing is margined to arrive 25% of the clock cycle in advance.
Determining where registers and clock synchronization points are placed is a major part of chip design work, done through a combination of manual and automatic methods. One approach is to split logic into smaller clouds separated by registers. Splitting logic in the middle can double the clock frequency, though this comes at the cost of an extra register and more storage.
Without a global clock, manufacturing variance becomes problematic. If two different computation paths must meet for a subsequent operation, variance means one path may finish earlier than the other on different chips. A signal propagating through could meet the previous or next value of the other computation, causing errors. This is why different chips made at the same process node, such as 3 nm at TSMC, can have different clock cycles depending on optimization of critical paths.
Pipeline register insertion is a pure trade-off between clock speed and area. A harder case arises with feedback loops in logic. For example, a running sum where a number is added every clock cycle. If the addition takes too long to fit in one clock cycle, inserting a pipeline register in the middle changes the computation, resulting in separate running sums for even and odd numbers instead of one unified sum. This constraint, where logic contains a loop, is the hardest to address and ultimately sets the clock cycle.
TSMC offers a PDK specifying the logic primitives available. Primitives such as AND gates or full adders are sized so none exceeds the target clock cycle for the process node. As a logic designer, the chip architect sets the clock cycle. Typically 10, 20, or 30 primitives can fit sequentially in a clock cycle, with primitives taking around 10 picoseconds. In principle, a simple register and AND gate loop could achieve clock speeds exceeding four, five, or six gigahertz.
However, area costs become prohibitive. A single gate equivalent has unit area of one, while the synchronization overhead may consume eight units. This represents a case where clock speed has been driven too high, spending almost all area on pipeline registers rather than useful logic.
Driving clock speed very high can reduce throughput because chip throughput is the product of work done per clock cycle (area efficiency) times clocks per second. Higher clock speeds reduce available parallelism. This dynamic resembles batch size considerations: low batch size delivers low latency for individual results but lower total throughput over time.
FPGAs and ASICs share the same conceptual model of gates built from primitives connected with wires on a fixed clock cycle. Anything expressible in an FPGA is expressible in an ASIC, which costs roughly an order of magnitude less and offers better energy efficiency. The first FPGA costs around $10,000, while the first ASIC requires a full tape-out costing around $30 million.
FPGAs are useful when deterministic latency, fast runtime, and high parallelism are needed but the workload changes frequently, such as every month, avoiding repeated tape-out costs.
An FPGA emulates the ASIC model using registers for storage, lookup tables (LUTs) for implementing gates, and a network of muxes connecting them. Programming the FPGA configures the muxes to create specific wiring paths through LUTs and registers.
The field programming refers to configuring the device after deployment in a data center or other environment. Each device has a mux in front selecting inputs from nearby circuits. The configuration consists of setting the mux control at each point, with a small storage device next to each mux holding the selection.
Each lookup table has four bits of input and one bit of output, implementing any of 16 possible functions via a stored truth table. The table is held in configuration bits. The LUT views the four input bits as binary, looks up the corresponding row, and emits the output bit. This allows the LUT to act as any gate: AND, OR, XOR, or others. The typical LUT size of four inputs represents a sweet spot balancing the number of LUTs needed against flexibility.
The overhead of FPGAs compared to ASICs comes from the implementation cost of each LUT. A four-input LUT requires approximately 32 gates to implement, whereas a four-way AND gate in an ASIC uses only three AND gates directly. The area and performance penalty arises because listing every input combination in a truth table is less concise than directly implementing the gate logic.
Deterministic latency is achievable in CPU designs, and processors inside AI chips such as Groq and TPUs do provide it. The challenge lies in achieving deterministic latency together with high speed. Non-determinism in CPUs primarily arises from the cache system. When accessing memory, the hardware first checks the cache and fetches from DDR only on a miss. Whether a hit occurs depends on the ambient environment, other running programs, recent history, and internal random number generation.
An alternative design uses scratchpad memory instead of caches. In this model, distinct instructions exist for accessing the scratchpad versus off-chip HBM. Software explicitly controls whether data resides in the scratchpad or main memory, removing hardware-managed caching decisions. This approach appears in TPUs and provides deterministic behavior at the cost of requiring explicit software management.
Even CPUs are super parallel if you think about all the cores they have. The amount of parallelism you get on a CPU is about 100 cores times maybe 16-way vector units, so about 1,000-way parallelism on a CPU.
One question: there is a die being used for the CPU, and if there are fewer threads, just as a matter of transistor voltages switching on and off, is it that there's literally one control flow—a small part of the die—where voltages are switching on and off? How do you actually occupy the die area of a CPU? If there are so few cores, what are you spending all of the die on? The cores are just much bigger and more complicated.
We should compare a CPU core, which takes up one one-hundredth of the die, to a LUT. A LUT is only 16 gates. It's clear why there are so many more LUTs in an FPGA than cores in a CPU. But why are there more CUDA cores, for example, than CPU cores?
What's the difference between a CPU and a GPU? Inside the CPU, one big use of the area is the cache. Very little is actually the ALUs. Mostly it's these register files rather than the logic units. Both of those have equivalents in a GPU, so that's not a big difference. But the thing that does not have an equivalent in a GPU is the branch predictor. There is a whole big area in the CPU which is just a bunch of predictors saying when the next branch will be and where the branch target is. Stripping a lot of that out, as well as making these register files tighter, drives a lot of the GPU gains over the CPU.
What is the purpose of the branch predictor? To execute both branches at once, or what does it do? The issue is that when I've got a series of instructions, if I have a branch, the actual step of processing an instruction takes a really long time. It takes maybe five nanoseconds. The time to notice that I've got a branch, evaluate whether the Boolean is true, update the program counter to the new target, and then read from the instruction memory could take five nanoseconds to finish.
I want to run a clock speed that is much faster than what five nanoseconds allows. Five nanoseconds is a 200 MHz clock speed. I would like to run at one or two gigahertz. So I need to run other instructions while the branch is being evaluated. I just want to keep running the instructions that happen after me. But that might have been wrong. If the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to jump to wherever the target is and run those instructions instead.
The purpose of the branch predictor is to predict, five cycles earlier, that a branch is going to happen, before you even get to that instruction.
If I think about how the brain works versus what you're describing here, at a high level the differences might be that while you can do structured sparsity in these accelerators and save yourself some area that you would have otherwise had to dedicate to gates, in the brain there's unstructured sparsity. Any neuron can connect to any other neuron, and not in ways where they have the column aligned. Then there's the fact that memory and compute are co-located. Although I guess you could say in a way the memory and compute are co-located on these dies too.
Another big difference is that the clock cycle on the brain is much slower than on computers. Partly that's to preserve energy, because the faster the clock cycle, the bigger the voltage needs to be in order for the signal to settle and to identify what state a transistor is in.
The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it's running batch size 1,000. Whereas the brain is not running batch size 1,000, there's only one of me. You could imagine saying, "Take a GPU and instead of running at a gigahertz, run it at a megahertz," and that would start to look a little more like the equivalent things you're talking about in the brain. But in the way silicon works, that does not give you a 1,000x advantage in energy efficiency. What it ends up looking like is you just run this circuit once to stabilization, and then it will sit idle for a long period of time. It doesn't consume a lot of energy while it's sitting idle because most of the energy is consumed in toggling bits from zero to one and back.
The way to think of a bit being stored is that you've deposited some charge in a capacitor sitting somewhere in the chip implicitly. It becomes charged when the bit becomes a one, and then it becomes discharged when it next goes to a zero. That cycle of charging the capacitor and then dumping that charge out to ground is where the energy is consumed. This is called the dynamic or switching power, and it's most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that insulators aren't perfect, but we'll discard that.
Most of the energy consumption comes from toggling from zero to one and back to zero. If you run a chip much slower and you only clock it once every thousand clock cycles, you will have 1,000 times fewer transitions. It will be about 1,000 times less energy consumption. But it's not a substantial advantage in energy efficiency.
Okay, so you described how a TPU works at a high level. What is the difference at a high level between how a GPU and a TPU work? There is a high-level organization principle that is different, and then inside the cores things are different. Looking at the high level, we'll take a GPU and a TPU and see what the top-level block structure looks like.
If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost-identical units, which are the SMs. They've got an L2 memory in the middle, and then a bunch more of these SMs on the bottom. So there is a fairly regular grid of cores. If we look at a TPU in comparison, you end up with much coarser-grained units of logic. You end up with just a few matrix units, which are the big systolic arrays. In the middle you've got some vector unit, and then you've got your matrix units at the bottom. These matrix units with a vector unit in the middle make up the whole TPU chip.
You can think of scaling this thing down into a really tiny unit with a smaller matrix unit and a smaller vector unit, and that is sort of what an SM is. From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip.
You're suggesting the tensor core within a streaming SM is analogous to an MXU? Yeah, it's all very similar. If you had more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you just have huge matrix multiplications, you might want to avoid the cost of having individual SMs with their own registers and warp schedulers. Why not just make a huge thing and amortize those costs across the whole thing?
This shows up in how large you can grow things. We've seen this theme, especially with the systolic array, where a larger systolic array amortizes the register file costs better. This design allows you to have larger systolic arrays, whereas the GPU design constrains you to having small units of everything. There is a trade-off, however.
Because of this coarse-grained separation of things, you need to move a lot of data from the vector unit to the matrix units, through just two lines of perimeter here. If you look at the equivalent thing in a GPU, you've got vector units everywhere, and you can move data through many different lines. The amount of data you can move between a vector unit and a matrix unit is actually much higher in a GPU than in a TPU. Instead of having to move all the data through just two lines, you're moving it through 16 lines of wiring in a GPU. But also you might have to move across less area. Which is an energy saving as well.
So if you can operate entirely within an SM, the data movement is much smaller. But the moment you want to operate across SMs, it becomes more complicated and expensive.
One might expect that a thing MatX might try to do is get the GPU-like smaller structure of systolic arrays surrounded by SRAM, but at the same time make it so that the things you need in an SM to support the CUDA architecture—which take a bunch of space—you might discard. We've talked publicly about something we call a splittable systolic array, which in some sense you can think of as big systolic arrays that can be small systolic arrays too.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
