Ben Brooks

I can't believe people want to ban this

🙌 American developers and deployers need a meaningful choice of trusted open models. Well done to the team!

Nooo you'll confuse Google again

My audition for Karoline Leavitt: "Open weights are a right, not a privilege. Next question." A crazy week of meetings in DC. To sum up the vibes across Congress in a single quote: "There is lots of affection for open source here. But the ecosystem is going to be absolutely steamrolled if they keep saying 'don't do anything' ". And a timely opportunity to speak at Freedom Tech DC alongside @matthew_d_white and @matthew_pines about the importance of open source and open science! We need to continue thoughtfully bridging the Bay-Beltway divide.

"We have to make sure that the models you can run in a disconnected environment are substantially less capable". I've spoken for years about why open weights shouldn't be a poor cousin of closed source by force of law. It's disappointing to see this reasoning making a comeback.

SOTA performance, < half the parameters, 4x the speed--and open weights.

History is repeating itself, so let's repeat the history. SB1047 had a kill switch provision. After much furor, Sen. @Scott_Wiener went to great pains to stress that it applied only to models in the custody of a covered entity. He made amendments to that effect. I was reassured that, at least in relation to kill switches, his team understood why a "full shutdown" functionality isn't compatible with open models. They didn't want to cripple open development with manifestly unworkable obligations. But Gov. @GavinNewsom vetoed SB1047 on the basis that, inter alia, "smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047". This is a view widely shared in the halls of Sacramento and Washington. If we revisit a Californian kill switch, there will be immense pressure to apply those requirements to models both at and behind the frontier. There'll be similar pressure on the AI Kill Switch Act from @tedlieu and @RepNateMoran, which requires covered entities to ensure they can "stop inference", "terminate user access" and "shut down" models. The net effect is you can develop a capable model if you release via API, but not if you release via open weights. As I and many others have laid out before, that is a terrible outcome that will feudalize the entire US economy around ~three Bay Area firms. If anyone is permitted to develop capable models (whether behind, at, or beyond the known frontier), our regulatory settings should promote responsible diffusion, not enforce a policy of containment behind paywalls.

In a consequential forum, I have seen a prominent CEO earnestly suggest p(doom) as the basis for a developer licensing regime. You can also look at the public record: with the notable exception of Dario, they have all flip-flopped between positions and nomenclature depending on what's fashionable. I've worked with a number of 3-letter regulatory agencies and in my experience, when the TSA is modeling the risk of terrorism, they don't speak the language of tea leaves.

Confidence and conviction aren't the same thing. This exchange today between Coxon and the NYT reveals a lot about the bizarre conflation of the two this summer / decade: > NYT: [The claim is] no other activity poses this level of danger. A lot of people want to understand how exactly AI can pose this kind of risk to human existence. Can you play that out for me, as concretely as possible? > Coxon: The concrete scenario is actually a tough question... People have apps on their phone that control physical devices, like household appliances. ChatGPT and Claude can access these appliances over the Internet. You can imagine AI is tricking people into doing things. So it would be pretty easy for a future version of Claude to hack into a military drone and have it fly around killing people... I struggle to visualize the ending viscerally enough to keep me up at night. This is conviction, not confidence. To be clear, this scenario isn't the steelman for catastrophic risk; @Nataliekitro probed him intensely (at times incredulously); and Coxon, a talented researcher, most likely has a richer mental model for catastrophic risk than what was conveyed in this short podcast. Yet extraordinary claims with prayers for extraordinary intervention require extraordinary evidence. This isn't enough. The public should demand more than hand-wavy projections inviting us to leap from the Hugging Face attack to the end of the world. For all the ink spilled this week, we actually ask far too little of those advocating policies to pace / condition / outlaw / nationalize frontier development, which are at best anti-competitive and, at worst, deeply anti-promethean.

I get nervous when NYT, Fox, and BBC start live tweeting AI policy updates like it's the Strait of Hormuz. The chance of a regulatory misfire is high, especially when frontier labs are out endorsing mandatory kill switches today.

FAS superforecasters predict a 1% chance that a nuclear weapon catastrophe kills 10M people in 20 years. 1%/10M/20y. That's the nuclear equilibrium. On that basis, we enforce a policy of denial, not a policy of responsible scaling, pacing, or paywalling. The US: > Bans development of nuclear weapons without presidential authorization on penalty of 25 years to life. > Bans sales or transfers other than to designated agencies on penalty of 25 years to life. > Compensates authorized contractors on a fee, not equity or profit sharing, basis. > Wages war to prevent development, acquisition, or use of WMD. If current frontier lab workers earnestly believe they have a 10% chance of killing everyone in 10 years (10%/8B/10y vs. 1%/10M/20y), none of the policy proposals recycled through the blogopshere and NYT this month truly rise to the occasion.

Pacing the frontier isn't inherently anti-open. It may be anti-fast, but it isn't anti-open. But the $2T unresolved question that goes to the heart of open weights is: what are the "Y" conditions at the frontier, who decides, and does "X" capability actually correlate with risk? Neither Coxon, OAI-HF, Fable, nor chikunyuga virus have changed the contours of this debate from 5-7 years ago.

While we're on the topic: we need to stop anthropomorphizing corporations. This bill is a terrible idea. But we have got to stop talking about these labs with reverential awe. They are legal fictions, not natural persons or organized religions. This recurring capital punishment motif (first for Hegseth, now for Sanders) is sophomoric, creepy, and unnecessary.

@DavidSacks bUt cHiNa wILL gEt tHe dAta!!!!

I deeply respect the Pope, but we should be mindful of the Church's legislative instincts for technology diffusion. "The art of printing can be of great service insofar as it furthers the circulation of useful and tested books, but it can bring about serious evils if it is permitted to widen the influence of pernicious works. It will, therefore, be necessary to maintain full control over the printers so that they may be prevented from bringing into print writings which are antagonistic to the Catholic faith, or which are likely to cause trouble to believers." Alexander VI to the dioceses of Germany, 1501

An unnamed official is mooting pre-release controls on open models past a certain threshold. Just my friendly tri-annual reminder that the Biden Diffusion Rule exempted models from export control if they were released openly. BIS dropped that Rule in 2025 because it "stifled... saddled... undermined" American innovation. Hard to square the circle.

@finkd 🙏 meta.com/thefutureisfor…

The NYT gets open weights! I love to see the press amplifying the voice of actual developers: Using Chinese models was like owning a house while using U.S. models was like renting one, they said... “I don’t think people should worry so much about who built it,” he said. “The focus should be, does it deliver the capability you need?”... Developers are building systems controlled elsewhere, which means the technology can be yanked, altered or re-priced without warning.

An excellent piece from @satariano @paulmozur Adam Krolik: nytimes.com/2026/08/05/tec…

From latent diffusion to FLUX 3, it's been a wild ride 🥹 Open weights coming soon! x.com/bfl_ai/status/…

One characteristic of anti-open hysteria (open source, open weights, open data, open web) is a tendency to ignore the massive diffuse benefits, and concentrate on limited acute harms. Deeply condescending to describe the beneficiaries of openness as "bored 19 year olds".

In their defense, when this debate really kicked off in 2023, Anthropic was basically the only frontier lab that wasn't pushing some cockamamie scheme to "license model developers" or "send in the AI peacekeepers" or whatever. Don't get me wrong, their "FDA / FAA for weights" is still a bad idea. But Anthropic has been commendably consistent about what it believes, and how it would weigh the competing interests.

The new House FRONTIER Act has the same old problems as before. Public authorities have a hard time deciding when the benefits of drugs, vehicles, and financial products outweigh the risk (not to mention the Internet). Each of these has decades / centuries of actuarial history. But under this bill, the federal government would also need to approve the methodologies that auditors will use to determine the acceptable catastrophic risk posed by 1e26 FLOP models, which have ~1 year of actuarial history. Two likely outcomes: 1. These methodologies are conservative. They overweight speculative risks and underweight diffuse benefits. Auditors cry wolf, obtaining emergency restriction orders at the slightest hint of a capability breakthrough or offense-defense lag. It becomes difficult to release a frontier model publicly via API, and impossible to release one open-weights. or 2. The Department of Commerce is mindful of that ^ possibility, and gives auditors wide latitude to determine their own methodologies. Standards are lax, developers can forum shop, and nothing really changes from the status quo. If you believe that AI is an important technology with dual-use properties, either scenario would seem to be undesirable. This is the problem with proposals that go beyond transparency and try to prescribe acceptable risk thresholds. They jump the gun. We simply do not know enough about the risk-utility profile of this technology to form a consensus view of acceptable catastrophic risk. Drawing a new line in the sand could chill the widespread release of legitimate technology (whether through export controls, liability reforms, or these private-audits-with-public-licensing proposals). To be clear, in my view, a regulatory determination of acceptable catastrophic risk is vastly preferable to the tortious approach endorsed by e.g. the original SB1047 / RAISE Act. Regulatory thresholds can be inspected, contested, and adjusted based on broad public input. State jury verdicts cannot so easily. But even so, we are way over the skis here.

Bill text as introduced: obernolte.house.gov/sites/evo-subs…

Our @bfl_ai researchers co-developed 3 of the 5 most popular open models on @huggingface. Open R&D is in our DNA. But open innovation faces major headwinds. We're glad to rally together with industry to champion open weights for transparency, competition & security in AI. x.com/JensenHuang/st…

Et tu, Demis? "Once the assessment protocol is shown to be effective and robust, formalisation could quickly follow, meaning that Frontier Models would be required to pass it to be deployed in the US market." Ask ten different AI CEOs, godfathers, and safety institutes when a model release is acceptable (or when it's OK to publish frontier weights on Hugging Face), and you'll get ten different answers. That was the problem with these ideas in 2022, and it's still the problem today.

The steady provision of models will encourage global dependency on the labs and countries that share them. And vice versa. x.com/nathanbenaich/…

@wrwagox @robrombach The same way we handle every other capable, versatile, composable technology: layers of mitigation up and down the supply chain. There's more work to be done at the model layer. For most AI risks, there is more work to be done downstream too.

At the G7 today, sitting across from Presidents Trump and von der Leyen, our @robrombach made the case for open innovation in AI. "...It’s crucial that we strike the right balance. A climate of fear around open technology, or a focus on suppression over diffusion, will leave the world reliant on a handful of firms for critical infrastructure. @bfl_ai is ready to work with you all to make open and responsible innovation the norm, not the exception."

It will be ironic if AI's big "code is speech" war is fought by Anthropic. The reluctant Bernstein of our times.

What Anthropic wanted: a federal authority vetting model releases What Anthropic got: a federal authority vetting model releases

ARGH. This is so, so, so frustrating. Unlike 99.9% of the Hayes Valley chattering class, I've actually worked with the FAA and its 3-letter friends. I cannot imagine a better way to kill off open innovation in AI than "FAA for weights". But it's also disappointing because I x.com/DarioAmodei/st…

Anthropic already told us that with the right system prompt, Claude would SWAT its own user. That was a generous warning: i.e. model behavior can be amplified, modified, or suppressed based on features hidden from the user. Nothing about these Fable 5 guardrails is surprising. x.com/ahall_research…

You wouldn't guess it from all the "light touch" rhetoric, but the Obernolte-Trahan AI bill is uncomfortably close to FDA-for-models. CAISI will license auditors, who must verify the "adequacy" of the developer's safety framework for achieving an "acceptable" levels of risk.

I guess they need to offer a big carrot to justify federal preemption, although I'd prefer neither tbh.

I signed this call for mandatory DNA synthesis screening because it showcases how AI regulation is meant to work: (1) A tightly scoped intervention that (2) mitigates a credible risk (3) at the most proximate chokepoint, (4) preserving open access to versatile models. x.com/AlecStapp/stat…

A concrete example includes the Cotton-Klobuchar bill: cotton.senate.gov/news/press-rel…

Glad to see the distinction between model evaluation with & without pre-clearance is becoming clearer in industry advocacy, eg. @OpenAI's paper today. These are very different propositions. Conflating the two is why earlier proposals floundered (e.g. mandatory UKAISI testing).

@OpenAI cdn.openai.com/pdf/25752ecb-0…

Delighted that Martin Scorsese has joined us to explore how AI can be thoughtfully applied to storytelling. We're also sensitive to the depth of feeling on AI in Hollywood. That's understandable! It would be odd if folks didn't have strong views. A few personal reflections on x.com/bfl_ai/status/…

Story by @brooksbarnesNYT nytimes.com/2026/06/02/bus…

Per @WHOSTP47: "We are NOT conducting oversight of all new models, as that level of government overreach would have chilling effects on free speech and innovation"

Pope Leo is an open-source champion