Percy Liang

The most important word here is *ecosystem*. It's not just about having an open-weight model. Open-weight models are a means to an end. To have a truly strong, open ecosystem, we need four critical frontier-level ingredients: open-weight models, open training datasets, open software stacks, and open process knowledge. Few people realize that NVIDIA actually has been pushing beyond open weights by releasing code and datasets for their Nemotron models, which is something open-weight model developers don't do. Marin further opens up the process knowledge - not just how to train one model, but how to iteratively improve and shape a model given particular goals, custom data, and hardware, e.g., how to design scaling laws and evals to guide architecture and data ablations. Open weights, datasets, software, process knowledge: these are the four critical ingredients (renewable resources) that give everyone the ability to most efficiently turn their compute (consumable resources) into the best models according to their needs and values.

⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place. x.com/marcelroed/sta…

There are two types of advances: (i) a singular change that provides 3x and (ii) a series of micro changes that each provide 20%. It is easy to celebrate (i), but (ii) is just as important, and the hard part is making sure the improvements stack. We care about both in Marin. x.com/classiclarryd/…

Here's a simple idea that works surprisingly well: model generates a response, it "self-verifies" its own response, and trains on those that pass. No ground truth answers or external verifiers. The key is the UQ verifier which we developed in previous work to check rigorously. x.com/tonyh_lee/stat…

Not only do we want to train a good model, we want to know it'll be good before we even start training. About a month ago, the Marin team launched a 129B (16B active) 1e23 FLOPs MoE run and preregistered a loss of 2.252. The run finished this past week and landed at 2.234. x.com/percyliang/sta…

While this run was going, we were busy curating more high quality data and making some architectural improvements, all of which will go into the next run. If you want to follow along in real time, come hang out with us in the Marin discord: discord.gg/J9CTk7pqcM

Going into the next Marin run. x.com/kevin_x_li/sta…

For the next Marin model, we are putting together a new data mix. Currently we have 18T tokens, but could use more. So if you are sitting on some secret stash of high quality tokens, please let us know! Pre-training, mid-training, SFT data all welcome.

You can see the latest data mix using this token viewer that @WilliamBarrHeld built: huggingface.co/spaces/marin-c… Thanks to @nvidia @huggingface @allen_ai @togethercompute BigCode, CommonPile, and many others who have been releasing high quality data, which helps the entire community!

I find myself repeatedly explaining the difference between open-weight (DeepSeek), open-source (Olmo), open-development (Marin). Let's see if this restaurant analogy helps: - Open-weight: food is made behind closed doors, server brings you the dish - Open-source: food is made behind closed doors, server brings you the dish and the recipe - Open-development: you see the chef make the dish in the kitchen (and can shout suggestions while its cooking)!