Kevin Esvelt

Best not assume there are good evals for the worst known (private) infohazards in bio. More, we can safely assume that some are unknown to all humans. The key difference is these are conceptual/combinatorial threats, not operational. Nature doesn't optimize for harm; models can. Granted, even reasoning models still quite bad at creativity and security, but they're getting better. Already, they can assemble sets of sufficiently obvious pieces from subfields too far apart for any human to be aware of them all. Until we have widespread trusted user programs that allow queries in one's published areas of research, but not in other dangerous subfields (which would block these combinatorial threats), aimply downgrading to a weaker model for bio isn't an unreasonable response. Note this is the same reason open-weight models simply shouldn't be able to reason about molecular and cellular biology at all; researchers ought to be able to rely on closed frontier models. The ceiling of potential harm is just too high - much higher than anyone looking at natural pathogens optimized for replication would assume - and we only need to buy a few years for physical transmission-blocking defenses.

@_NathanCalvin Almost anything substantial in biorisk must be redacted for public safety. ... admittedly I have high standards for "substantial". Still, that isn't noteworthy. Guardrails are too focused on operational details over strategy + key insights. And still playing whack-a-mole.

@stgoldst @BioAndBaseball @jasoncrawford @agupta @OliviaHelenS Replication is certainly a requirement for an effective pandemic weapon, and trying to do that better than nature is as hard as one would expect, which is why existing highly transmissible agents are key components. It's the causing-harm aspect where nature falls short.

@kanzure @jasoncrawford @agupta @OliviaHelenS Sadly, living systems are hardware-insecure and can't be reengineered in time. But blocking human to human transmission is surprisingly feasible - first on-demand when faced with a threat, eventually all the time. Roadmaps: gcsp.ch/publications/d… gcsp.ch/publications/s…

@jasoncrawford @agupta @OliviaHelenS My first threat model involved a security-naive colleague putting certain pieces together regarding effective pandemic weapons, then sharing (assuming it's always better for everyone to know about threats). That's still a risk. Now, substitute an LLM for the colleague.

@jasoncrawford @agupta @OliviaHelenS De novo pathogens are far away. The main threat comes from weaponization of something that already exists. The barriers are conceptual and combinatorial, so LLMs can help. Effective weapons are optimized for harm; natural pathogens for replication. Very different.

The closure of the Liberty Bell Bay manganese smelter is a disaster for civilizational resilience. You cannot make steel without manganese alloys. Without it, Australia will become far more dependent on imports. If things go badly wrong in the world, from bio or nuclear or both, the light of civilization must be preserved. My recent supply chain models indicate that the cost of stockpiling sufficient manganese alloy to bridge AU/NZ/SG demand under autarky (until smelter reconstruction) roughly doubles the total investment required for those nations to pull through. This isn't about post-collapse recovery. It's about whether ~40M people can maintain a roughly 1990s industrial base and tech level without the rest of the world. To the AusGov ministers considering what to push for: mothballing the plant would preserve the core hardware for restart when conditions improve, preserving strategic autonomy for as long as most workers remain local. Selling it for scrap would needlessly destroy the capability while immediately dumping much of the site rehabilitation cost onto taxpayers. But it may not be too late. To any global investors who care about resilience, and especially those with a stake in New Zealand as your backup: it has no chance of surviving intact without Australia, and Australia has none without steel, and it cannot make steel without manganese alloy. Rescuing Liberty Bell Bay would be an impact investment in resilience, not philanthropy, with better expected civilizational returns than you could get from almost any donation. And for investors who particularly care about the environment (or Australian autonomy), the primary market alternative to clean Tasmanian hydropower smelting is imported Asian manganese alloy generated with fossil fuels - which is either coal-driven or exposed to Hormuz. Plain bad luck and financial mismanagement, plus a remarkable failure of strategic foresight, are slated to leave civilization more vulnerable. It's an unnecessary tragedy. Best we minimize the damage. t.co/hxJMmQJ9Zc

"Anytime they didn't understand something, they would ask the AI." AI-democratized competence + destructive intent + increasingly powerful offense-dominant technologies = an unsettling trajectory We need to invest in resilience. Excellent, disturbing work. x.com/AntoniaJuelich…

Progress is wildly beneficial, will probably kill us, and we should keep going anyway. This is not a contradiction. Technology got us here. Technology must get us out. open.substack.com/pub/kesvelt/p/…

Respiratory infections & pandemics are network security problems... but we don't need to involuntarily share DNA/RNA packets. Here's to hoping we shut it down before anyone (or anything) discloses how to make ~infinite pandemic variants with arbitrary virulence. Exciting news. x.com/nanransohoff/s…

~No one stockpiles respirators, transformers, or rare earths for catastrophes. This isn't market failure. It's that government will seize them and pay peacetime prices. Foresight earns nothing. Honor pre-crisis contract prices, & stockpiles will appear. open.substack.com/pub/kesvelt/p/…

Good to see this out before the Bipartisan Commission @Biodefensecomm meeting on AI-bio today. The technical solution exists; we need mandates with robust red-teaming & teeth. It shouldn't be easy to buy DNA sufficient to make infectious 1918 influenza. nature.com/articles/s4146… x.com/AndrewCurran_/…

@nlpnyc @deanwball Of course, that also means there are a lot of blocks on areas that aren't security-relevant at all, but vibe as scary. And there are few on some of the most important concepts, which are far more dangerous than protocols. I do hope we can fix that, but it's a losing battle.

In US/China talks on AI this week, one mutual interest stands out: preventing models from helping to engineer worse-than-Covid pathogens. Start with mandatory DNA synthesis screening for pandemic-capable agents, then talk guardrails. No trust required. nature.com/articles/s4146…

Grateful to @gabrieldance for taking AI-biorisk seriously. Anthropic and OpenAI are more careful than the rest of society, but the bar is so low it's underground. No laws govern AI, bio regs are full of holes - and the problem extends beyond biothreats. nytimes.com/2026/04/29/us/…

@gabrieldance AI democratizes competence. Not just technical skills, but strategic thinking. Safety teams are filtering for operational recipes; the AIs are identifying vulnerable targets. Many folks have technical training but no sense of strategy or scale. What if one of them snaps?

I've started writing about safeguarding civilization and positive futures. Biosecurity, AI, infrastructure, supply chains, deterrence, and the moral frameworks underneath. First post: The Glory and the Horror. kesvelt.substack.com

Found a tick after hugging Daphne following a morning run. Thanks to @JoannaBuchthal & our collaborators, we can now engineer animals to resist Lyme disease. But until we immunize the wild white-footed mice that carry it, we’re stuck with tick checks. nature.com/articles/s4146…