Ten years ago, we started @cerebras around an approach many believed was impossible. As a computer architect, it is hard for me to imagine a more exciting time. Model releases are accelerating, and hardware tapeout is compressing from multi-year roadmaps to annual launches. Hot Chips is my favorite conference, and it’s where I launched Cerebras 7 years ago. This year’s conference was especially exciting, and so much innovation was shared. I am watching the industry recreate itself: SRAM is mainstream, DRAM is moving into the third dimension, networks are being fundamentally redesigned, and AI is helping design and program the chips themselves. The industry has never moved faster and some of the hardest architectural questions are still wide open.
The latest updates from Rubin at Hot Chips continued to demonstrate NVIDIA’s relentless improvements in throughput. The Rubin architecture is incredibly efficient at highly parallel workloads where the arithmetic intensity is high. This is why it is such an efficient architecture for training where high batches are common. In inference, the roofline is ~2200 FP4 FLOPs/byte (50 PFLOP/s / 22 TB/s). That means for every FP4 weight in DRAM, Rubin needs to reuse it for ~550 MAC operations to sustain high throughput. During inference, this occurs during 1) prompt processing or 2) high batch decode. In prompt processing, since all of the tokens are known up front (i.e. the user or agent provided them), they can all be processed together making it easy to achieve at least ~550 parallel tokens reusing the same weights for high throughput. In decode, even though the attention has higher arithmetic intensity, the overall model is limited by the experts and other projections resulting in only <5 FLOPs/byte for models like Deepseek V4 and Kimi K3 even at long 100k+ context. Therefore, high throughput is achieved by using large batch size, by processing multiple requests at the same time. Doing so enables reusing the weights across tokens from all of those requests to maintain high throughput. But running at high batch size reduces the decode speed, or interactivity, as seen in pareto curves. For this reason, Rubin continues to set the throughput benchmark for low interactivity inference. And, with its high prompt throughput, and with the rise of disaggregated inference, it is also extremely well suited for prompt processing in heterogeneous disaggregated environments.
