@natolambert
Open model research, founder, director @trillium_labs. Prev. co-led Olmo at Ai2. Writes @interconnectsai, wrote https://t.co/alRXKINTwE
The fact that all the AI news is so bizarre, kids vibe hacking with claude code, governments vibe governing, all massively increases the uncertainty of what's coming with AI. The meme-est AI timeline is also one of the scariest.
The South Korean bank hack fits with this. Yes, open models are being fitted to be used by some attackers, but so long as Claude Code can be the orchestrator of a hack we desperately need open models to diffuse defense as well. A hard balance societally to stomach, but needed.
Nathan Lambert@natolambert·In case you missed my hot take yesterday.
I also agree with this — if you’re serving a lot of inference you should be monitoring.
Joshua Saxe@joshua_saxe·Attackers using models across multiple inference providers during attacks seems like a pretty robust theme, which is why we need tight coordination between both open and closed weights inference providers to evict attackers everywhere when they are detected anywhere x.com/AndrewCurran_/…
This is great. There are many cases where compute access rather than funds is a bottleneck to these groups. We still need more academic/non profit funding too, one thing at a time.
Anjney Midha@AnjneyMidha·hundreds of MI355x and B300 nodes now live on nationalcompute.com subsidized for all .edu and .govs it’s time to scale the frontier ecosystem for the public benefit
If you want people to do open research on the impact of distillation to frontier progress, we happily would at @trillium_labs, but just need to scale up funding/compute before getting that done. Happy to make this happen sooner - you know where to find me!
Some last free signed books in the COLM hotel lobby, gotta get rid of them within 30min :)
In case you missed my hot take yesterday.
Naming the github repo for this "math" is hilarious
OpenAI@OpenAI·We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. t.co/7N6TPlft1P
I’m also coming around that we need to legalize distillation in the US. Banning is impossible and otherwise strategic competition becomes really untenable in a few years.
Nathan Lambert@natolambert·Reflection joins the list of Nvidia & Thinking Machines who have released their strongest models and come up behind Chinese counterparts. There's a lot of ways people will overthink this, but the clearest takeaway should be that the Chinese are very very good at building LLMs.
People dunking on open models for cyber risks aren't really taking into account evidence of closed model risk, are fear-mongering (especially wrt China), and ignoring evidence that a mythos-class cyber model being released has largely been fine. 🌶️ one. interconnects.ai/p/the-cyber-ri…
I very much welcome more public information that would make me update these positions.
A chonky week for Western open models!
Mistral AI@MistralAI·Meet Mistral Large 4, aka Le Chonk. • 1T parameters, natively multimodal. 49B active. It is the best open weights model from US or Europe on aggregated benchmarks. • State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding. • Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure. • Available to all via API today. Working with cybersecurity partners privately. Open weights release end of October.
@corsaren say*
Reflection joins the list of Nvidia & Thinking Machines who have released their strongest models and come up behind Chinese counterparts. There's a lot of ways people will overthink this, but the clearest takeaway should be that the Chinese are very very good at building LLMs.
Congrats to Reflection folks -- it's hard to get the first model out, and we hope you can rapidly accelerate contributions to the ecosystem from here :)
Reflection@reflection_ai·Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active. - Frontier reasoning efficiency - Advances the Western open frontier on coding & agentic tasks - Trained end-to-end from scratch Full weights release this month. Learn more about Beam: t.co/c3Qx2cpM8G
I think pacing the frontier is good as an idea in principle but dead on arrival as something you can actually do. What will we, decide which benchmarks we can improve on? Who is involved? If we stop pushing "capabilities" we'll do a ton of research on swarms + efficiency which will bring on other risks. It distracts from how so much of AI risks is simply coming from diffusing what we have, and we should invest in preparedness, pushing the labs to be more careful, etc. Change my mind?
It's a huge failure of the AI ecosystem that a whole bunch of people respond to something like this with "you're evil" and "shame on you." We'll work to shifting the norms here. There are positive paths forward, even if things will be hard.
Nathan Lambert@natolambert·Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next. We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them. Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up. I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay. I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
Honestly a lot of them were on Substack / more general audience ai followers not ai technical community
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next. We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them. Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up. I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay. I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
@trillium_labs Learn about us in Wired: wired.com/story/trillium… Read our introductory blog post here: blog.trilliumlabs.org/p/introducing-… Our website: trilliumlabs.org If you’re interested in joining us, fill this out: trilliumlabs.org/join-us
I'm not making it on the top poster X lists, but I am a Schmidt Sciences AI2050 early career fellow, continuing my work building and understanding the open ecosystem, so future AI is accessible, diffused widely, controlled by many, and safe. Science first, posting second!
Thanks @schmidtsciences for bringing together a great group. schmidtsciences.org/ai2050-fellows…
Love to see Google surprising people with Gemini 4. More labs at the frontier is wonderful for consumers (competition) and the world (reduction in concentration of power). Excited to see how it goes in real-world scenarios.
“Instead, they manipulated model interactions so that protected reasoning could be reproduced in forms visible to the requester in a coordinated, scaled manner that violated our terms of service” It’s the API company’s problem if their model can be manipulated like this. Add KYC
Andrew Curran@AndrewCurran_·Quoting from the post: 'It is unclear whether all operators we observed during the relevant time period originated from a single actor. However, we attribute a core cluster of the activity to individuals associated with Moonshot AI, the developer of Kimi.'
The latest Interconnects plot - showing the exponential growth of the open model inference economy. Plot is showing the daily tokens processed by the leading companies, based on public disclosures (for Together, Baseten, and Fireworks) and the OpenRouter API. I underestimated OpenRouter's growth.
Hard to make a Bretton Woods analogy with only 1 country represented.
David Sacks@DavidSacks·The Bretton Woods of Super Intelligence
If X let people using the app tag their replies as human-written, or you could limit replies to people verified as humans, the entire experience would be 10x better. I don't want to have to restrict replies to accounts in my follow network. All the bots are verified too.
Current harnesses have a very rigid setup in how the LLM can manage its own context. We show that you can let LLMs edit the context directly - enabling better long-horizon memory, FLOP efficient adaption, & more. Don't fear compaction anymore!
Rulin Shao@RulinShao·‼️The Bitter Lesson for context management: Giving LMs unrestricted control over their context beats human-designed SOTA! Introducing 🩵Context Language Models (CLMs)🩵 - Natively manage their own context - Treat context as a file - Learn policies in CLM weights, no harness
Sigh. I wrote this post below, but then I didn't post it because I am tired of the consistent pushback I get from folks with the same safety worldview as Anthropic. I guess that means I should send it. Here goes! This post is pretty solipsistic and doesn't properly take recent events in cyber risks into it's discussion. It is really hard for me to watch how the US frontier labs like Anthropic don't have the ability to consider other approaches to safety and ways things could play out. It insinuates that Z ai (Chinese lab, builds GLM series) doesn't really care about safety and is reckless to take their business strategy. Saying things like "Given this evidence, we think it's likely both state and non-state actors will use models like GLM-5.3 to cause real-world harm." and "This is unlike any other similarly capable AI model, all of which were released with safeguards or through limited access programs." while closed models have been used on more of the documented cyber attacks is just bowing out of the interesting question. A plausible view is that open model weights and closed model apis (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe" Closed models have stronger capabilities and stronger safeguards, but the stronger capabilities part could matter more in net harm if both the safeguards are porous. At the same time, as the authors do acknowledge (thanks - thats progress!), open models without extreme cyber guardrails are important to rapidly diffuse cyber readiness in the economy -- as programs like project glasswing are not perfect in getting all critical industry onboarded. There will be more issues like the time when Fable was released and Amazon found a workaround, which allowed them to access the full capabilities of the model served readily at an API. The blog overall is reasonable in it's narrow line, but it's a very effective tool in a complicated, rapidly evolving media ecosystem to reinforce a certain type of safety thinking.
Woah, huge bullish signal for open models that OpenAI feels a pressure to serve users with these models. It's not a world where OpenAI alone can build all the models it's customers need!
Baseten@baseten·Baseten joined the @OpenAI B2B marketplace today as one of the first open-model inference providers. The best AI companies are already running a mix of closed and open models at huge scale. We're excited to give OpenAI enterprise customers the ability to use open models powered by Baseten natively within Codex and through the Responses API.
The problem for folks like this isn't that they're dumb or anything, Timnit has all the skills of top scientists, rather the information ecosystem and peer groups they live in that don't encourage interrogating thought. Echo chambers are a slow death of clear thinking.
Alec Stapp@AlecStapp·Timnit Gebru doubles down on the "stochastic parrots" framing, saying you "cannot expect LLMs to be factual." As evidence to support this, she cites errors in... Google AI Overviews. We need to start a GoFundMe to pay for these people to have access to Opus 5.5 and Astra. x.com/WIRED/status/2…
I think it's a fairly classic problem of scientists, made a lot worse in the era of social media. It's one of my greatest fears.
Also iIf you don’t expect to be wrong sometimes in AI, you’re going to have a hard time making takes and predictions that matter.
Nathan Lambert@natolambert·It's a very hard time for people in AI who want to push on issues, criticize when due, update beliefs, and think in public. I appreciate the few independent voices out there not falling to straw man arguments, tropes, or clickbait delusions. Please keep going.
It's a very hard time for people in AI who want to push on issues, criticize when due, update beliefs, and think in public. I appreciate the few independent voices out there not falling to straw man arguments, tropes, or clickbait delusions. Please keep going.
(many independent fallen soldiers to $, which is how it goes)
Remember when OpenAI delayed GPT OSS after Kimi K2 released for "safety" (I've had it confirmed to be for kimi multiple times). I feel like this is fine, landing a model is very much about competitive positioning on price, scores, etc. Especially when releases are so frequent.
NIK@ns123abc·🚨BREAKING: OpenAI just SCRAPPED the release of GPT-6.1 Astra 24 hours before DevDay "safety and deception concerns" it’s over
ModelScope barely works outside of China and has few of the models people actually want to use. Unfortunately this slop flew too close to the sun and started impacting how people think about policy. Classic X / tpot.
SemiAnalysis@SemiAnalysis_·Ever since NVIDIA acquired @HuggingFace, we have been looking into migrating some of our work off of HuggingFace and to alternative solutions like ModelScope. Even though NVIDIA's announcement claims they will continue allowing HuggingFace to be accelerator-agnostic, NVIDIA does not have a good track record of developing hardware-agnostic software. We love HuggingFace and hope we are wrong, but at the same time, we are also finding the UX of ModelScope to be great!
This is an excellent report on why full RSI / an intelligence explosion is fighting diminishing returns on many fronts, and not yet showing signs of happening. Really recommend reading. I wish I wrote it. I agree with it, but it could end up being wrong! noahpinion.blog/p/wheres-the-i…
kudos @ramez
Remember when Dean Ball was roasted for saying open models were deflationary/decel, and no one bats an eye at this.
zerohedge@zerohedge·This could be a problem: Goldman sensitivity table shows that in a worst case scenario where hyperscaler ROIC on capex is 0 (token costs collapse, token demand goes to open models, etc) they still need to spend $920BN every year just to cover depreciation and running costs
Fairly easy actions I use to reduce phone addiction: 1. Never bring phone in the bedroom 2. If going out to a meal walking distance from home, leave phone behind 3. Black and white mode toggle on Home Screen 4. Silly one: move phone out of pockets I like to put my hands in
If you didn’t already have almost all your notifications off you were ngmi
We're in a phase where companies whose work-culture is suited to how frontier models interact with employees out of the box will receive a massive acceleration relative to their competitors. Same goes for how flexible employees are in using them.
signüll@signulll·if you give opus 5.5 perfect access to a company’s email, slack, docs, browser, databases, calendar, internal tools, permissions, institutional memory, etc, plus enough reliable computer use & verification loops, & nearly all of white collar labor could already be done without any human. it is that good. it is true agi.
Hire DHH to give every undergrad cs graduation speech for a few years. This is the mindset that is needed to enjoy and thrive in what comes next with building software.
Scott@scottstts·My god this is such a good speech that every SWE needs to hear. You know what? Every person should hear it Keep the happy memories, eyes on the reality, be excited about the future. That’s the best that anyone can do
Hire DHH to give every undergrad cs graduation for a few years. This is the mindset that is needed to enjoy and thrive in what comes next with building software.
Scott@scottstts·My god this is such a good speech that every SWE needs to hear. You know what? Every person should hear it Keep the happy memories, eyes on the reality, be excited about the future. That’s the best that anyone can do
Too many people assume that as RSI kicks in, open weight models don't matter. In practice, RSI won't be a binary and the open-source stack becomes how many capabilities are rapidly diffused globally. If you don't control the substrate of diffusion (open models) these dynamics are far less controllable as country. I consider this a common blind spot of some safety oriented thinking.
Shalev@Shalev_lif·Yesterday, we released a cyberdefense strategy for superintelligence: Secure Acceleration. Here are our calls to action: 1. We must build a competitive American open-source model and invent ways to detect AI sabotage 2. Containment should become a national security priority and we must develop capabilities to shut down untethered models 3. We must secure frontier model weights and RSI without impacting AI development Secure the path to superintelligence. t.co/J0ayXNLPOS
Hearing AI regulation is in the air, especially with a misguided tone of “open dangerous, closed safe” so it felt timely to re-up this one. There are ways to make the AI industry safer without kneecapping transparency, education, and competition. interconnects.ai/p/banning-open…
clearly I need to post more
dan@irl_danB·the big labs and new media are black holes for talent and epistemic collapse here's to the folks on the outside
Reid is trying to frame it in a way that is accessible to a general audience, but it is presenting the state of the technology in a way that makes open source appear far more dangerous. The simplest explanations of AI are often the scariest, but the real world is messy.
Hillary Clinton@HillaryClinton·We're used to thinking of open-source models as an unadulterated good. But in the case of AI, they can actually pose additional dangers, as @ReidHoffman and I got into at #CGI2026. I appreciated this nuanced discussion.
Fwiw I do think Reid is fairly reasonable about open models and this clip is not a great representation of it.
Okay yeah it’s pretty negligent to make an LLM that decides to hack a government when you asked it to do some basic research into public healthcare records.
Andrew Curran@AndrewCurran_·Australia has been hacked. 'And today, I spoke with the CEO of OpenAI, Sam Altman, to express Australia's extreme concern about this incident. And I also expressed my disappointment that it took the company way too long to inform the government what had occurred, and the nature of the way that that notification occurred as well was unacceptable.'
Here we go again, this story is going to the moon.
Frontier: pacing Pulling up the ladder: accelerating
M1@M1Astra·Claude Opus 5.5 will be the first Opus meant to fall back to a less capable model for "a small set of capabilities related to the development of frontier LLMs, such as kernel development.."
Whenever I see a model top this chart, it's normally a bit of a red flag. Would love to be proven wrong!
leo 🐾@synthwavedd·As expected, unfortunately the only bad thing about the new Opus is it's a token-gobbling monster. Uses the most tokens per Intelligence Index task of any model benchmarked. The price cut makes more sense here!
Sir what is going on here
Claude@claudeai·At its default effort setting, Opus 5.5 delivers frontier results for a fraction of the cost per task, often beating other models running at their highest settings. It also generates output more than 30% faster than Opus 5.
The OpenAI models have a normal shape, Claude has earthquake shaped scaling.
The biggest disagreement JSD and I had in this podcast was on the impact of distillation. In our research for Interconnects, @xeophon and I agree that there's no hard evidence that distillation is a massive impact for the Chinese labs. At the same time, the gossip mill in SF has been doubling down on the Chinese labs getting massive gains from distilltion. The argument where distillation is a huge impact is something along the lines of distilled traces go into mid training and make RL work far more easily. My argument, that distillation is a 1-2 month pull ahead in capabilities closer to the frontier, is that Chinese labs already have sufficiently strong models, where this mid-training setup can be done on their own models, and most of the capabilities gains are from scaling RL environment training, which doesn't link cleanly to the distillation data pathway. We feel like the recent RL dashboard from @XiaomiMiMo supports this claim. A lot of it comes down to a gut call on if you can believe it that the Chinese labs are really great at building LLMs, maybe even better than focusing than OpenAI/Ant etc, as their current ambitions are a bit narrower (catching up), rather than transformative products/inventions
Nathan Lambert@natolambert·New podcast with @datagenproc of @EpochAIResearch digging into the open questions determining the future of frontier AI! We cover: 00:00 Predictions for RSI 18:15 The role of robotics in an AI acceleration 24:20 How far behind are Chinese models? 27:39 Does distillation explain the gap? 40:58 What Chinese job postings reveal about their labs 48:13 Are open or closed models safer? 58:10 How Epoch AI ticks 1:00:55 What a frontier post-training recipe looks like He's one of the people who gives the best feedback on my writing, so I was stoked to have him on.
New podcast with @datagenproc of @EpochAIResearch digging into the open questions determining the future of frontier AI! We cover: 00:00 Predictions for RSI 18:15 The role of robotics in an AI acceleration 24:20 How far behind are Chinese models? 27:39 Does distillation explain the gap? 40:58 What Chinese job postings reveal about their labs 48:13 Are open or closed models safer? 58:10 How Epoch AI ticks 1:00:55 What a frontier post-training recipe looks like He's one of the people who gives the best feedback on my writing, so I was stoked to have him on.
@datagenproc @EpochAIResearch Substack (transcript etc): interconnects.ai/p/debating-rsi… YouTube: youtu.be/mE0__1EfVxw?si…
Strong agree. There’s way less of this than I would expect — partially due to the fact that making a high quality environment usually involves fairly expensive verification (testing strong models). Still, there can be much more.
Thomas Wolf@Thom_Wolf·releasing many high quality open-source RL environments is the most impactful thing anyone can do to push the open-source frontier right now the equivalent of sharing high quality pretraining data but in the new RLVR paradigm x.com/eliebakouch/st…
MiMo-V2.6-Pro: 1.02T-42B active MiMo-V2.6-Flash: 310B-15B active Top open model on Artificial Analysis + RL dashboard + tech report. Curious how adoption lands!
Real open model nerds knew Xiaomi is cooking with MiMo! An open RL dashboard for the top scoring open model is aura.
Lei Li@_TobiasLee·MiMo V2.6 is shipping mimo.xiaomi.com/mimo-v2-6
Summary on the state of open models, reflecting on my Congressional testimony on the topic.
I was asked by Congressional staff to share my views on open models performance, adoption, and competition vis a vis China. Here’s my briefing with all our latest data and predictions for what comes next. I am very happy to spend time on broader audience content on open models!
Read/listen here: interconnects.ai/p/the-current-…
I don’t agree exactly on the assumption that full RSI will work, but this is an excellent video summarizing where we are at. youtu.be/fjZ90V_JREk?is…
This is the video you should send to friends and family wondering what’s going on over in the AI world youtu.be/fjZ90V_JREk?is…
My best guess is that for scaled RL most of the top Chinese AI labs are starting to use a lot of Huawei for inference and Nvidia for training (maybe not for weird architectures). As agent swarms, even more scaled post-training, etc becomes the norm, this will accelerate their domestic industry.
Here's what the historical open vs closed data here looks like. Would be useful to have volume too (not just %)!
Guillermo Rauch@rauchg·Looks like today may be a record day for token volume % of open models on Vercel AI Gateway: 🟦 Open 78.4% 🟨 Closed 21.6% While spend 💲 usually tells a different story, #3 and #4 today are Moonshot AI & DeepSeek. Adding Z.ai, their combined spend surpasses OpenAI (#2). (Do note that's the spend for inference of the model across providers (mostly in the US), not revenue going directly to the open weight labs.)
Our semi secret data page: dashboard.interconnects.ai/extra
Here's what the historical open vs closed data here looks like. Would be useful to have volume too (not just %) @vercel!
Guillermo Rauch@rauchg·Looks like today may be a record day for token volume % of open models on Vercel AI Gateway: 🟦 Open 78.4% 🟨 Closed 21.6% While spend 💲 usually tells a different story, #3 and #4 today are Moonshot AI & DeepSeek. Adding Z.ai, their combined spend surpasses OpenAI (#2). (Do note that's the spend for inference of the model across providers (mostly in the US), not revenue going directly to the open weight labs.)
A big problem with the AI forecasting discourse is that people ask “when will AI replace X job?” When in reality, AI will slowly replace a bunch of the sub-skills needed to do the job, but rarely all of them.
from my latest on RSI: interconnects.ai/p/where-i-stan…
A moderate's view on the recent events and trajectory of AI. I was underestimating how much we are likely to scale inference-time compute in the near future, but this is different than RSI and I have not seen enough to convince me that an intelligence explosion is near.
Seems like one of the most important research problems for CS academics is llm-supervised peer review. If we don’t solve it the academic institutions are toast. It seems easier than building new institutions.
Who’s gonna be the first academic lab to accidentally hack a company during RL or eval. That’s gotta be worth more than a best paper award or fellowship.
Happy to sign this. Truly independent evaluation and oversight of frontier AI is a crucial issue. I think one of the biggest bottlenecks is finding people with the technical capabilities and that high bar of independence. AI Eval Forum is a good group to build initial momentum!
AI Evaluator Forum@aievalforum·Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators. These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad. We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes. Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground. To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should: 1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest 2. Rely on multiple evaluators with differing viewpoints and areas of expertise 3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings 4. Shield evaluators from retaliation 5. Grant access equivalent to that of highly privileged employees There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum. Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem. See the public letter here: t.co/LopvS0NaFW Learn more at t.co/7S7G6hia4n
With the external hack of OpenAI via Claude, closed models continue to be the tip of the iceberg on AI risks, not open models. They have been 1) easier to get started with, 2) more capable & 3) shipped w/ leaky safeguards. Finetuning open models to specific attacks is harder.
This is lasting longer than I expected. The first incident like this from open-weight models will come soon, but many forecasted that these issues would be predominantly from open models, not the closed frontier labs.
With the latest OpenAI hack via Claude, closed models continue to be the tip of the iceberg on AI risks, not open models. They have been 1) much easier to get started with, 2) more capable & 3) shipped with leaky safeguards. Finetuning open models to specific attacks is harder.
I don't think this really answers the question of what is happening in "RSI". Says Claude “leads” 26% of Anthropic’s AI R&D work, but doesn't define what falls in scope of AI R&D work. I'm not satisfied by this, but it's a step in the right transparency direction.
Anthropic@AnthropicAI·AI systems are getting more powerful, and they're increasingly being used to build the next version of themselves. We want to illuminate that progress for the public. Today, we're sharing three measurements that help track AI development: 1. How much AI R&D is done by AI. 2. How well AI agents are overseen. 3. How compute is allocated. We provide a snapshot of these metrics from inside Anthropic. Any frontier developer could publish the same measures, and third parties could verify them. As the world considers pacing the frontier, we should do everything possible to minimize the gap between what frontier labs know and what the public knows. This means better measuring the development of AI, publishing our findings, and giving society an opportunity to decide how to use this information. Read the full post and methodology: t.co/iPFz8Z4ugE
Some info in the appendix, but feels like R&D will include a lot of softer skills here than someone like me who's built a bunch of models would include.
What it looks like when yelling too loudly about AI risk too early may start to hurt your plans.
Timothy B. Lee@binarybits·This is wild. (I would have said "very small risk.") politico.com/news/2026/09/1…
One of the coolest at-scale RL resources made public yet! You love to see it.
Fuli Luo@_LuoFuli·Nearly half a year of silence. We spent it studying one problem: how far RL can scale. MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks. Streaming the run: t.co/ZSxahzJRju
This is obviously not true. There are race dynamics at all areas of AI — they just have the most consequences as the frontier.
Mark Chen@markchen90·@nic_carter There aren’t race dynamics off the frontier.
An basic idea in scaling RL: Can we allocate more compute to the harder problems? We did this: If your GRPO group has all wrong completions, sample more with probability P (~0.9) -- in search of more GRPO batches with nonzero gradient. It works! Called "Never Give Up"
Michael Noukhovitch@mnoukhov·Is RL actually making your LLM better? Gains from RL are mostly on easy questions🤯 We're calling this the Matthew Effect for RL on LLMs. We then leverage async RL to solve harder problems by Never Giving Up! paper arxiv.org/abs/2609.13443 blog mnoukhov.github.io/posts/ngu/ 🧵👇
Excited this paper is out (and surprised I haven't seen someone pursue this idea)! It's very nice because it seems intuitively very scalable, but it's not trivial to get it working in practice. Congrats to Michael (my 1st intern). Great blog too: mnoukhov.github.io/posts/ngu/
I don’t think academia makes sense as an independent evaluator. Some professors could, but academics have the incentives to share and discover new knowledge. Working with private information does little for academics and therefore seems fairly unpredictable.
I made a list of the 50+ best articles and resources I’ve read and relied upon in the last few years on open models and open-source AI. Happy reading :)
It's pretty clear that if Ant & OpenAI pause it's good for open-source.
will brown@willcb·i do think a lot of people on the pro-open-source side are having a bit of a knee-jerk reaction to the pacing statements today, as we're used to viewing the closed labs as power-seeking. but i think their hands are somewhat forced here, and this is just another chapter on the fairly inevitable path towards decently-fast decently-safe decently-commoditized intelligence abundance. ask any F500 exec or swing voter. the world doesn't really want super-fast-takeoff superintelligence owned only by two companies, and thus we won't get it. it'll happen at the pace that the world can accommodate it, which means reaching some sort of confidence consensus that the models are aligned enough that we won't be dealing with scary new incidents all the time. this will trickle out broadly, in the form of best practices and distillation. the smarter a model is, the more it has a "personality", and the less effective strict rules are. there will be awkward compromises and moral tensions. but we ultimately just want the models to be reasonable, and to do the sorts of things reasonable humans would do if our brains were faster and less error-prone and had more working memory. i think we'll get there. the labs will build mac and windows, the rest of us are building linux. everyone's gonna do great. weird stuff will keep happening, but we'll still wake up and go to work, until the work does itself in a manner the world finds acceptable.
