@percyliang
professor of computer science @Stanford @stanfordnlp, co-founder of @togethercompute, https://t.co/7R5THVnJ6u, @simile_ai, pianist
Glad to see another live training run! Here's how Marin 535B-A23B is doing today: wandb.ai/marin-communit… Anyone else want to share?
Fuli Luo@_LuoFuli·Nearly half a year of silence. We spent it studying one problem: how far RL can scale. MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks. Streaming the run: t.co/ZSxahzJRju
A year ago, David was the only FTE on Marin. Today, thanks to Open Athena, Marin has 10 FTE. Read his post to better understand the context of Marin. And David tokens are always a pleasure to read.
David Hall@dlwh·It’s been a few weeks since Marin kicked off our 535B/A23B MoE hero run. So far it’s gone almost boringly well. It’s also been just over a year since Marin joined Open Athena. I wrote up some thoughts about how far we’ve come, and how we got there. openathena.ai/blog/marin-535…
Open models gaining momentum. This is incentive-aligned and just makes sense.
Jensen Huang@JensenHuang·Exciting day for NVIDIA and @huggingface. Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI. Thank you @ClementDelangue for coming to me. NVIDIA is going to be a great home for Hugging Face, its community and the future of open models. 🤗 t.co/q8Om2Xc5ye
Marin 535B-A23B is 13% through training. This hero run would not be possible without the generous support of the Jen-Hsun and Lori Huang Foundation, which provided the funding for the compute (Coreweave). Thanks @JensenHuang for supporting open models!
In addition, thanks to the Google TPU Research Cloud for providing the compute for our previous models that kicked off Marin and its continued development. And thanks to the Siegel Family Foundation, Fulcrum Science, @schmidtsciences, many others for their support. t.co/dkHAWSUtk5
The "What-If Machine" is a vivid articulation of what Simile is building. It's not just about forecasting the future passively. It's about understanding how active interventions on the world change its course. It's the classic difference between causation and correlation.
Simile@simile_ai·Most big decisions are some version of a “what-if” question. What if we launched this? What if we raised the price? What if we didn’t? At Simile, we’re building a What-If Machine to simulate those decisions before making them for real. But getting the answer right requires understanding both what people do and why they do it. @msbernst explains what it takes to build one. t.co/ohlKUxWrkr
Marin 535B-A23B is ~7% done training, and so far things look on track. Next Tuesday (Sept 1 @ 10 PT), we will have a Zoom panel/discussion where the Marin team will talk about the design decisions that went into this run, the tradeoffs made, and our learnings. If you're interested, join the Discord for details (t.co/xwJSxd2IxH).
The fact that Simile’s first technical blog post is about confidence is notable. Confidence is paramount to simulation. If a coding agent messes up, you can often tell and repair. If a simulation is wrong, you might never know and make a bad consequential decision based on it. We do rigorous evals of our simulation model, but this only tells you how well your model is going to do on average over some population + use case. The confidence model tells you how well your model is performing on each query in real-time.
Simile@simile_ai·Anyone can simulate the future. But the simulation only matters if it’s trustworthy. At Simile, we train two types of models: simulation models and confidence models. Our first research blog post explores the origin of our proprietary confidence model, which predicts the accuracy of our population simulations and, in turn, makes them actionable. t.co/qz0yPZzY55
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open. Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow. Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
In addition to forecasting our final loss, we can also forecast the loss of intermediate checkpoints, so we can see if we’re on track over the voyage:
It's time to scale that Simile. Our ambition is to build a foundation model that can predict what anyone will do in any situation. Research-wise, Simile in 2026 for simulation is like OpenAI in 2019 for AI assistants - signs of life, a path to scaling, yet still many open research questions. Commercially, Simile is already a company with many amazing enterprise partners who understand the potential of simulation and will help us grow sustainably. If simulation and this special combination of research and product is appealing to you, we’re hiring.
The most important word here is *ecosystem*. It's not just about having an open-weight model. Open-weight models are a means to an end. To have a truly strong, open ecosystem, we need four critical frontier-level ingredients: open-weight models, open training datasets, open software stacks, and open process knowledge. Few people realize that NVIDIA actually has been pushing beyond open weights by releasing code and datasets for their Nemotron models, which is something open-weight model developers don't do. Marin further opens up the process knowledge - not just how to train one model, but how to iteratively improve and shape a model given particular goals, custom data, and hardware, e.g., how to design scaling laws and evals to guide architecture and data ablations. Open weights, datasets, software, process knowledge: these are the four critical ingredients (renewable resources) that give everyone the ability to most efficiently turn their compute (consumable resources) into the best models according to their needs and values.
⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place. x.com/marcelroed/sta…
There are two types of advances: (i) a singular change that provides 3x and (ii) a series of micro changes that each provide 20%. It is easy to celebrate (i), but (ii) is just as important, and the hard part is making sure the improvements stack. We care about both in Marin. x.com/classiclarryd/…
Here's a simple idea that works surprisingly well: model generates a response, it "self-verifies" its own response, and trains on those that pass. No ground truth answers or external verifiers. The key is the UQ verifier which we developed in previous work to check rigorously. x.com/tonyh_lee/stat…
Not only do we want to train a good model, we want to know it'll be good before we even start training. About a month ago, the Marin team launched a 129B (16B active) 1e23 FLOPs MoE run and preregistered a loss of 2.252. The run finished this past week and landed at 2.234. x.com/percyliang/sta…
While this run was going, we were busy curating more high quality data and making some architectural improvements, all of which will go into the next run. If you want to follow along in real time, come hang out with us in the Marin discord: discord.gg/J9CTk7pqcM
Going into the next Marin run. x.com/kevin_x_li/sta…
For the next Marin model, we are putting together a new data mix. Currently we have 18T tokens, but could use more. So if you are sitting on some secret stash of high quality tokens, please let us know! Pre-training, mid-training, SFT data all welcome.
You can see the latest data mix using this token viewer that @WilliamBarrHeld built: huggingface.co/spaces/marin-c… Thanks to @nvidia @huggingface @allen_ai @togethercompute BigCode, CommonPile, and many others who have been releasing high quality data, which helps the entire community!
I find myself repeatedly explaining the difference between open-weight (DeepSeek), open-source (Olmo), open-development (Marin). Let's see if this restaurant analogy helps: - Open-weight: food is made behind closed doors, server brings you the dish - Open-source: food is made behind closed doors, server brings you the dish and the recipe - Open-development: you see the chef make the dish in the kitchen (and can shout suggestions while its cooking)!
It is liberating being able to talk about what you work on.
