@natolambert
Open model research @ something new. Prev. co-led Olmo at Ai2. Contact via email. Writes @interconnectsai Wrote The RLHF Book, 🏔️🏃♂️
At this rate h100s will stay about $2/hour forever
Huawei about the get a blank check from the government to make inference chips x.com/kimi_moonshot/…
If recent events with Kimi K3 have finally convinced you that you need to try and understand how the Chinese labs approach AI - and how it differs than the SF center of power - you should read this post: x.com/natolambert/st…
Qwen’s biggest models have never been open weight recently. Something is changing. You can imagine it like… Xi: no you’re not allowed to keep your best model closed anymore, we weren’t succeeding enough. A new era of competition for intelligence (if the benchmarks hold up). x.com/alibaba_qwen/s…
To be clear this is an illustration, not fact
yes this is the case wrt Dean's post Imo that's fine as US seems likely to always lead in investment? So you can get the combo of US leadership + good open models + more time to understand AI, so seems like a win for every ideology? The key is less money = longer to AGI x.com/ArielKwiat/sta…
I think what is pretty clear is that the Chinese labs are far more capital efficient. In a world where scaling labs are intelligence is proportional to effective capital (buys compute, data, & talent) that may be the greatest strength your AI industry could ever have.
This is who runs this account (Yes I still like cake today as much as my first) x.com/deanwball/stat…
I am a big supporter of Dean’s work but found this somewhat inevitable, sadly. Is the latest example of why it makes me sad to see the vast majority talent go to a few companies that are subject to extreme attention. Need more variance. x.com/deanwball/stat…
Disaster. They don’t have the talent to understand the technology, it’s all been pushed out, and no clear documentation of the process. x.com/andrewcurran_/…
We need an operation warp speed for state capacity and other independent evaluation (and understanding) of frontier models.
Wonderful and totally empty 😁
Claude Fable requires extra credits now? Pulled from subscriptions two days early…
Just a software bug: github.com/anthropics/cla… > Update - We are aware of an issue preventing users from selecting Claude Fable 5 within Claude.ai, Claude Code, and other surfaces, and are working to resolve this issue. status.claude.com
Inkling’s humility at launch may have made it underrated. The previewed small model will wake a lot of people up (if it comes soon). Look at its peers on ARC AGI. x.com/arcprize/statu…
Frontier open models are accelerationist on ai diffusion and deflationary on the valuations of the biggest tech companies with large capex. IMO that tends to be pretty good for the average American. Diversity of AI use will be wonderful to get benefits sooner.
Important to read. China is committed to continuing its open, global ai approach. x.com/vince_chow1/st…
@tenobrus Then, adoption is good.
On point 2. Something I repeatedly tell people now is that companies will keep making open models longer than you expect because money will keep pouring into AI. Consolidation isn’t something that’ll happen rapidly (at least in the near term). Consolidation is one of those things that has been predicted since ChatGPT launched and never arrived. 2026 has been a remarkable year for open, the door to economic viability is open (if we can clear regulatory hurdles).
jUsT gOoD bEcAuSe Of DiStIlLaTiOn x.com/zephyr_z9/stat…
K3, GLM 5.2, Inkling, DSv4 Flash... is great to see the open ecosystem is a group of superb options, all with different strengths and weaknesses. Feels the most relevant open models have been since DeepSeek R1, a lot of opportunity to build the open stack.
K3 seems like the big, multimodal planning model for hard tasks. GLM 5.2 is the faster, agentic model. Inkling is for everything else multimodal, hopefully more finetunable too. DSv4 flash is a cheaper workhorse of a model. The list continues.
I'm a lot more confident in inference companies / companies with O(B) revenue becoming neolabs faster than many neolabs become functioning businesses. Wild to watch all this play out x.com/FireworksAI_HQ…
@soldni I wouldve prolly liked silent better but im here to suffer
Inkling was somewhat inevitable once Tinker took off -- integration of post-training services across @thinkymachines's entire stack -- and it's one of the best open-model business stories to date. Excited to see it grow and more emerge (some inference companies are next).
Let's estimate the scale of the inkling final RL run? An initial guess O(1K GB300s) ~1 week ~$5M? Would be a fun interview question for a post-training lab. Glad to see some info like this shared publicly.
(this is a claude-informed guess, would like to see much more principled math on it, I think claude has a bad intuition of inference costs in agentic settings)
Love this approach to open releases from Thinky -- release a practical foundation to build on. I expect this to succeed for them, much as Tinker has exceeded many people's expectations (myself included)! x.com/thinkymachines…
Thinky with a ~1T param, 41B active, apache-2 model Benchmarks are a clear step up from Nemotron Ultra (55B active), new best American model, and omni input. A bit behind GLM 5.2 on agentic benchies, and Kimi K 2.6 on multi modal Super exciting! Thank you @johnschulman2 & team
Blog: thinkingmachines.ai/news/introduci… Model card: huggingface.co/thinkingmachin…
Claude Fable is another big step in being able to make nice lectures based on existing educational content. Much better than Opus. GPT 5.6 is still very far off here. Is a good example of where Claude Code being a bit easier to work across different knowledge work tasks.
Another insane own goal on AI research leadership from the U.S. It's like "how do I remove myself from an international scientific network" 101 class x.com/kyleichan/stat…
The open model community is extremely unprepared for when a model gets stuck in the undefined white house licensing regime - and it could permanently knee cap the open model economy within 6 months. Why this'll happen and what we can do: interconnects.ai/p/6-months-to-…
I got around to reading ai 2040, and I’m very happy with their focus on transparency (which I strongly agree with), but I find it odd to talk about that so much with no real discussion of open source.
Let’s goooo USA 🇺🇸🦅🏆
@BrendanFoody Congrats to the team!
🫡🇺🇸 very happy you came back swyx x.com/swyx/status/20…
I'm doing Q&A videos as I roll through my course. Here's the next one, covering subtle fixes to the on-policy distillation and reward model derivations, common notation traps when doing this math, and more added resources to go deeper (e.g. @johnschulman2's kl estimation blog). Q&A 2 is here! 00:00 Derivation fixes 06:10 Code examples & additional resources 08:08 Extra RL notation and notes Keep sending questions on YouTube, GitHub, and Discord. Phoebe and I are loving them.
Trying to help a close collaborator get their @X account back after falling for the AGI-level phishing campaings, but they're not famous like me, can someone help do me a favor? Will check DMs :)
Selling tokens and building the token machines are both going to be great businesses. I'm confident in more companies that build open models figuring out how to make money. Open models bring attention and attention is a funnel to revenue for many companies today. x.com/poezhao0605/st…
Happy to say @zafstojano - an added maintainer who helps me with the RLHF Book code - added a simple on-policy self-distillation example to the codebase, which can work on some toy problems. Excited to dig into this more, happy to see the repo fleshed out!
When we were in China, @xeophon and I made a quick detour to visit Meituan. They continue to be one of our favorite open model builders, as they're showing how a variety of companies can succeed here and baffle a lot of people as to why they're making models. Meituan is one of the larger tech companies in China. They're building LLMs to add services to their own products. In China the notion of the "super app" is very popular, so this dream of more services for users with AI is very natural there. With this, Meituan wants to own the full stack of how they deliver value to their users. When we visited, they were very unassuming about everything. We just met a few people from the LLM team, a quick meeting about building models. They build general foundational reasoning models, and then fine-tune it further for their products. They can release the general model to support the ecosystem and learn how it can be used. Their focus was very clearly on ownership, and a hint of cost-saving, so the recent news of v2 being trained on asics fits with that mentality. They want to deliver real products to users with low cost. Companies like this will keep building models in China. It's a small micro study of how different the players in the AI ecosystem are. Kimi, Z ai, etc are all much flashier offices, come across as the "hot new thing" but Meituan has the talent and resources to build models as well. Congrats to the Meituan team & thx for having us!
letssss gooooo breaking this bad boy out today loooooooooooong cat x.com/Meituan_LongCa…
I feel like the Chinese labs I visited felt like this. It’s what Ai2 felt like. It’s the most reliable energy for making something amazing (even when you’re an underdog with the number of resources). I’m always super excited to find my next star intern. x.com/jonathanross32…
Together AI processing 400T tokens a month. x.com/vipulved/statu…
With everything going on, it gives me hope that there's such a diversity of companies building open models today. A lot of the story of open models unfolds under the shadow of the biggest frontier models. Lots of unearthed value. x.com/interconnectsa…
This is real and a horrible consequence of vibe regulation of frontier models. x.com/ClementDelangu…
I've been getting a lot more hate than usual as I try to speak my mind about regulatory capture / unintentional attacks on open-source. It's pretty sad, as there are few people in AI that can speak their mind (most companies say they cannot) and I know many people agree with me silently. I also get people saying that you only say that because it supports the outcomes you want, in a weirdly derogatory way. Of course this is true, but I'm choosing to turn down meaningful wealth so I CAN fight for these values, working at non profits to speak my mind. Building a future that is more inclusive, diverse in the application of AI, and fairer for our children. I may not always be right, but it has been clear to me for a while that more openness right now will help way more than supporting the closed causes. I continue to re-visit this and don't think everything should be open like some of the open-source absolutists. I also don't like a lot of my comrades making fun of anthropic, calling the people there evil, etc. Those are not the case. Trying to stay the course!
I have a lot of discussions with people who literally want open models banned. Happened in 2023. Happened in 2024. Happens today.
Anthropic's political pressure on distillation is regulatory capture and most of the employees are blind to it under their veil of safety.
Or their paycheck helped them buy into safety, is only human nature, I don't even fault them that much.
Lapdog + laptop = ?
I get feedback a lot that is like "your book should be the RL for LLMs book" or "the post-training book" and it's definitely true those would sell more copies. The reality is that this book was in many ways a side project, and by the time I realized I agreed with a bit of this I didn't have the time for *another* refactor. At the end of the day, I still dumped as much knowledge as I could from what I was doing into the book, and now the course and the code. In it's spirit the book is totally a post-training book. The process to change this would've delayed the book from anywhere from 3 to 15 months. It is simply an amount of time I didn't have with Interconnects, Olmo, and other life necessities. So this isn't to say that I'll never do it. Re-prints and new versions are a common thing. It's doable for me to refactor most of the chapters, re-write the introduction, and make it a post-training centric book. Still, RLHF as a topic deserves a dedicated text and is far from solved. It's a technology that skyrocketed language models to prominence and points to a lot of fundamental problems interfacing the user and the AI. Much of the content that got me to where I am today in my career is by diving into caring about this interface, so I'm happy for it to have the space to live, breath and thrive. So in reality, I probably could've hot-swapped the title to sell more copies, but it would have made me feel dishonest to do so. For anyone wanting to learn post-training, there's nothing in this book that doesn't apply to you -- post-training is just constantly evolving and growing in complexity. A final nitpick, is that RLHF actually matches my more conceptual, intuitive vibe a good amount. Post-training is far more practical, in a data and systems sense, where this is more of a math & intuition book. Anyways, the RLHF "post-training" Book is coming soon and thank you for trusting me with your attention. 🩵
I fixed the seattle AI scene with one tweet, everyone in the replies just needs to hang out with eachother. x.com/natolambert/st…
There's a lot of sloppy thinking around open models. You can ban them and make it impossible for US companies to use them, but this won't stop A) global open model progress B) bad actors using them So what exactly is gained by banning open models, including those from China?
A few issues right now 1. Figuring out state capacity for managing frontier capabilities. Dean's stuff is great for this 2. Figuring out how we manage the coming frontier open models 3. Disentangling the distillation accusations/nonsense from the above two x.com/deanwball/stat…
Organizing a small AI meetup in Seattle tomorrow afternoon (Fremont/Ballard). Reply / dm if interested in coming.
This got way more interest than expected, noted. Will do more generally accessible events in the future!
@woke8yearold @snewmanpv @BlackHC Ask: What do I have to gain? I work at non profits dawg I just want AI to go well. What do AI executives have to gain?
Is what happens when the world becomes AGI pilled then both the leading lab and the government tell you you need to bow down if you want access to their models. I feel it too. More of the last few weeks giving people the words to explain how they felt for months. x.com/willccbb/statu…
Actually it's potentially worse. We're in a major step change & acceleration in progress and the admin just did a complete 180 in policy on models. It would've been better to have no policy before. Now, the policy has changed to a vibe check & we have no idea what comes next. x.com/natolambert/st…
I used to give the current admin a nod for saying their AI policies are fairly reasonable (chips stuff far messier). In the last ~3 months they've destroyed all that trust and dug a giant hole of uncertainty - where the US's AI leadership will now be actively degrading.
Horrible timeline. We should be getting transparency into why this is the case and how we plan for a world with an increasing number of models at this capability level and way stronger models in the near future. x.com/steph_palazzol…
The goal with my rlhf book is to make the "home on the internet" for the next generation learning post-training. That's why I'm doing all formats (lectures, code, book, discord, model completions... & ofc blog of interconnects). A hub is more lasting than non-fiction writing.
The AI companies should be presenting this much more as transparent data of what is happening over time and much less like smear campaigns with strong policy wishes. Just comes across as so self serving, hard to want to support them. x.com/Discoplomacy/s…
GLM 5.2 being on the Opus frontier for cost of CursorBench is what drives frontier lab margins down x.com/leerob/status/…
Add more wins for GLM. The model has some brittle characteristics, and is getting crushed by closed models here, but we should expect open models to be more jagged, and you use multiple of them depending on the task. Congrats again to @Zai_org and am excited for the next one x.com/fchollet/statu…
A much needed data release! Excited to tinker with the data. x.com/RichardZ412/st…
"Knowledge wants to be free" is an unofficial Interconnects mission statement, courtesy of @xeophon
Another quick lecture -- I've been asked many times for prereq's to my book and what you should know, so built a little lecture (with GLM 5.2) to cover some more basics. Topics include: 00:00 Introduction & Course Prerequisites 01:37 Language Models Overview 02:47 The LM Head 04:29 Softmax & Log-Probabilities 06:13 Anatomy of an LM Training Example 06:37 Computing LLM Probabilities (+Phoebe the Dog) 09:52 Three Common Masks in Post-Training 11:03 A Small Decoding Review 12:14 Training an LM: Cross-Entropy 13:23 Optimization & Fine-Tuning 13:55 Pretraining to Midtraining to SFT Pipeline 15:25 Probability Essentials: KL Divergence & Entropy 19:36 Sigmoid & Pairwise Likelihood 20:29 Reinforcement Learning Framing (MDP) 22:28 Transitioning Tools into Post-Training 23:12 Recommended Resources & Wrap-Up Happy learning and I'm still taking questions from during the course for Q&A videos.
Lecture 0: youtube.com/watch?v=MMDNae…
Something I should add -- on-policy distillation was the last content I got to sneak into the book before going to print. Felt very important to have this method covered, it's growing rapidly and used in distinct ways. So you can also read what is covered in this lecture! x.com/natolambert/st…
New lecture for the book! Nominally about synthetic data, but mostly is a walk through of the distillation literature from the Hinton 2015 paper to multi-teach on-policy distillation of today! At 7.4 hours of video in my post-training brain dump and counting :) It was fun to stare at the math long enough and talk through the 3-4 core changes that needed to be made to the original formulation to have on-policy distillation be ready for the mainstream like it is today (and in RL frameworks). Otherwise, I include a bit of a history lesson for how synthetic data generally slowly took over all post-training data research (it wasn't always the case)! Then I do some 101 review on constitutional AI, rubrics, and other popular methods. 00:00 The emergence of synthetic data 10:50 Background on teacher-student knowledge-distillation 24:47: On-policy distillation (OPD, MOPD, and OPSD) 37:11 Constitutional AI & AI Feedback 45:50 Rubrics as rewards & conclusions Ofc, watch on YouTube etc.
@AndrewCurran_ It's a problem if glm-5.2 doesn't escape the TPOT echo-chamber -- it's a big step and all parties relevant to AI should think about what it means
GLM-5.2 should be “DeepSeek moment” for agents. We enter a new world where the top end of agentic capabilities are available in open models. If you care about open, now is the time to inform regulators on how we should build a world with safe, frontier, open intelligence. x.com/interconnectsa…
TMax: An open RL recipe for terminal agents I’m very excited to get to share a new RL paper today that I got to have a small part in – a type of paper I suspect we’ll see much more of in the future. The key is that RL research is very different today, in mid-2026, than what most observers have in their context. The average conception of an RL paper is grounded in the RLVR revolution of early 2025, where many people could use vanilla RLVR libraries to hillclimb on math benchmarks. Crucially, this style of math work could be done on base models or fairly stably on already trained models. With agents, the tasks of focus are very hard, requiring complex tool-use, harnesses where the model automatically manages its history, and much more training to make smaller eval improvements. We’re shifting from a renaissance of RL study to rapidly needing to improve its empirical rigor and common community engagements. TMax is the best open data for hillclimbing on frontier terminal tasks. It’s been validated with rigorous experiments, and if the authors wanted to just form a “RL environments startup” they could probably sell it for millions of dollars. This data work is some of my favorite stuff to be around in my 2.5+ years at Ai2. As a general summary, the recipe is open data and recipe lessons from hillclimbing the Qwen 3.5 smaller, dense models on terminal tasks. These models are super hard to hillclimb in this area, as they’re already trained heavily on the task. The training is very infrastructure-dependent, and most of the RL innovations are more designed to make training stable than to improve the rate of learning. I strongly recommend this paper. I joke around that I was happy to be an author just so I had to read it twice! You can find Hamish’s thread sharing more here or read the paper here. You can click through to find the model weights, the data, and even some fun further artifacts to study like all the RL rollouts from a training run – where the model sometimes became aware that it was being tested. The biggest takeaway I have from following this work, and more of the work in the community, is how important recipe work is. Let me define “recipe work.” It is a style of paper that explains all the steps you need to make crucial model improvements – data, algorithm, codebase, pitfalls, etc. Getting started in meaningful RL experiments today is a substantial expense. There are a ton of companies, an entire industry emerging really, around the idea of taking open-weight language models and finetuning them with RL on your domain-specific tasks. What I see in many projects is that getting an initial baseline is very hard. This phase, which can cost weeks and anywhere from $10K to $1M+, feels like spinning your wheels (A fun fact is that an RL step on a model like Nvidia Nemotron 3 Ultra on Tinker costs $1K and a meaningful RL run would be hundreds of steps – credit Edward Hu). It takes a lot of time to get traction in learning signal on meaningful, hard RL tasks. What we need as a community is a way for people to study small ablations to established RL recipes, as most labs won’t have the resources to do it from scratch in a meaningful way. This is what I hope TMAX can be for terminal agents, or the start of. Yes the training jobs are expensive, as the paper documents a standard training job being 8 nodes of H100s (2 train 6 inference) for 2-3 days, but that is approaching something academics can study. The establishment of this recipe took O(100) of these training jobs to get right. This isn’t my first time trying to establish this direction. When we launched Olmo 3 we had the “RL Zero“ model families, which are clean RL runs from a base model on a certain domain. This type of recipe-dependent work is a clear indicator that meaningful post-training work today looks much more like pretraining work of years past. We need decision-making ladders, clear ways of seeing small improvements in the models, stability, and so on. Part of this is down to academic gatekeepers, who won’t reward a paper doing very clean empirical work to push a recipe 1-2% up. They’ll favor a “new algorithm” that matches results, or something sort of bogus. My hope is that we can have multiple, stable, clear recipes across agent types, so innovations can be tested more clearly in multiple domains. (If you’re working on this, please reach out – I’m happy to support if I can, but I likely can’t reply to every email). As a quick aside, the RL frameworks in vogue today seem to be SLIME and SkyRL. The libraries of choice have shifted throughout these seasons in RL, which further contributes to a form of fragility in the literature. A bit of continuity will go a long way. So, go read this paper. It’s a really great example of how seemingly simple data and infrastructure work can be very hard and impactful. It’s also got me looking for more applications of Divergence Proximal Policy Optimization (DPPO) as another small evolution to the best RL algorithms of the day, by virtue of being a bit more stable by improving token-level clipping.
References: TMax Paper: t.co/GZzMQ6yLcM TMax Blog Post: t.co/82dma9gdHH TMax Github: t.co/vDMXanB0xy TMax artifacts on HuggingFace: t.co/1CsMBmRzzz Video on the topic on YouTube Olmo 3 RL Zero Model: t.co/NvFmjnrkxA Olmo 3 Paper: t.co/aTWiuw1sTA
Life gave me lemons, so I made lemonade. Feels like childhood.
An hour in and first impression is definitely that GLM is really solid (very easy to set up on @FireworksAI_HQ, props to them for that, took me like 5min to get going in claude code).
Open weights models, via GLM 5.2, had their "very practically useful" in coding harness moment before Gemini. ~200 days since the release of Opus 4.5.
Something I haven't advertised much is that I made a Discord to go with my RLHF book, launching in print in a few weeks. Trying to create the place for the next generation of folks trying to learn post-training to learn and have community.
Frontier labs are definitely SOTA at self serving nonsense. Yes, AI is a crucial technology, but also Silicon Valley systematically spreads knowledge via talent exchanges and bars. This isn’t a national security deep decision making room. x.com/willdepue/stat…
In fact, the biggest advantage OpenAI and Anthropic has is abundant resources and talent to keep pushing the frontier and be there first. With the scale required you can only “steal” or “leak” so much.
Well said. x.com/hlntnr/status/…
