I joined AMD right after graduating from MIT. Two stints there and a decade building @cerebras later, I still lose sleep over how much memory and compute to put on a chip. I’ve spent my entire career in computer architecture, and I’ve never seen the problems change this quickly. Models keep getting larger. The demands on the hardware keep growing. Decisions about where to put memory and how to connect it have become central to what people can build with AI. We chose SRAM for its bandwidth. Now we’re working on wafer-scale stacked DRAM to give larger models more capacity while preserving fast inference. That brings us back to some very physical problems: how to power it, cool it, and manufacture it reliably. This is the work I love. In part two of our AMA, I answered your questions about those decisions and where we’re going next. 0:30 What changes between CS4, CS5, and CS6? 2:19 HBM already stacks memory. What’s different here? 3:34 We chose SRAM for speed. Why add DRAM? 4:52 Drawing the stack is easy. Making it work isn’t. 7:23 The memory-versus-compute decision that keeps me up at night
When we designed our wafer-scale systems, we focused on keeping compute close to memory so we could generate tokens quickly. Bringing that speed to more developers also takes work across the software and infrastructure around it. That’s why I’m excited about our partnership with @gimletlabs. Their software maps different phases of inference to the hardware best suited to each, bringing Cerebras into Gimlet Cloud behind a single API. Together, we plan to deploy Cerebras-powered inference capacity, with the first datacenter expected online later this year. As agents take on longer tasks, the time spent waiting on each model call adds up. I’m looking forward to seeing what developers build when they can work through those steps faster. Congrats to @zainasgar and the Gimlet team. We're just getting started.
Gimlet Labs@gimletlabsWe’re bringing @cerebras wafer-scale compute to Gimlet Cloud. 100 MW planned. First datacenter expected later this year, with speeds up to 3,000 tokens/sec. Building the multisilicon inference cloud. Blog: gimletlabs.ai/blog/cerebras-…
NVIDIA is continuing to push throughput, and I think that makes sense. We need more tokens, and we need them cheaper. But what used to feel fast at 100 to 200 tokens per second is quickly becoming the new batch mode. I spoke with @swyx on @latentspacepod about how expectations for inference speed are changing.
What excites me most about Jalapeño is the design methodology behind it. Using AI to help build chips faster and push both throughput and latency forward could benefit the entire industry. I spoke with @swyx on @latentspacepod about why I see this as the future of chip design.
When we started @cerebras, we knew we would have to solve difficult problems in yield, power, and cooling. What became clear was that making wafer-scale work would require rethinking almost the entire system. Today, we answered questions from the community about those engineering decisions, how we run larger models, and what it takes to program the hardware. 2:15 Why hasn't NVIDIA gone wafer-scale? 4:10 How we build a working chip out of an imperfect wafer 8:07 What happens when a model won't fit on one wafer? 11:34 Why we chose SRAM over HBM 14:26 How much of a barrier is NVIDIA's CUDA moat?
Ten years ago, we started @cerebras around an approach many believed was impossible. As a computer architect, it is hard for me to imagine a more exciting time. Model releases are accelerating, and hardware tapeout is compressing from multi-year roadmaps to annual launches. Hot Chips is my favorite conference, and it’s where I launched Cerebras 7 years ago. This year’s conference was especially exciting, and so much innovation was shared. I am watching the industry recreate itself: SRAM is mainstream, DRAM is moving into the third dimension, networks are being fundamentally redesigned, and AI is helping design and program the chips themselves. The industry has never moved faster and some of the hardest architectural questions are still wide open.
The latest updates from Rubin at Hot Chips continued to demonstrate NVIDIA’s relentless improvements in throughput. The Rubin architecture is incredibly efficient at highly parallel workloads where the arithmetic intensity is high. This is why it is such an efficient architecture for training where high batches are common. In inference, the roofline is ~2200 FP4 FLOPs/byte (50 PFLOP/s / 22 TB/s). That means for every FP4 weight in DRAM, Rubin needs to reuse it for ~550 MAC operations to sustain high throughput. During inference, this occurs during 1) prompt processing or 2) high batch decode. In prompt processing, since all of the tokens are known up front (i.e. the user or agent provided them), they can all be processed together making it easy to achieve at least ~550 parallel tokens reusing the same weights for high throughput. In decode, even though the attention has higher arithmetic intensity, the overall model is limited by the experts and other projections resulting in only <5 FLOPs/byte for models like Deepseek V4 and Kimi K3 even at long 100k+ context. Therefore, high throughput is achieved by using large batch size, by processing multiple requests at the same time. Doing so enables reusing the weights across tokens from all of those requests to maintain high throughput. But running at high batch size reduces the decode speed, or interactivity, as seen in pareto curves. For this reason, Rubin continues to set the throughput benchmark for low interactivity inference. And, with its high prompt throughput, and with the rise of disaggregated inference, it is also extremely well suited for prompt processing in heterogeneous disaggregated environments.
