Tristan Harris

*Security is sleeping on emerging catastrophic risks* (cross-post from my blog..) We've just seen: * a campaign that used agents to compromise ~100 businesses and steal about 600,000 credit cards with minimal human involvement; token costs were ~$25 per target successfully hacked. * Hacktron getting access to OpenAI’s monorepo by using Claude to exploit a blind buffer-overflow RCE in a way that (to me) felt superhuman. * OAI/Huggingface. * And, of course, there’s the ongoing explosion in newly discovered vulnerabilities. We've muddled through all manner of crises in security before, and for most AI attacks, cyber attack and defense will reach a natural equilibrium over the next few years, as we figure out how to mitigate cyber attackers who have limited goals like espionage and ransomware. But some cyber attackers will have maximalist or nihilistic goals, and I'm concerned about this because while yesterday's 'maximalist attackers' (e.g. Russia -> Ukraine, US/Israel -> Iran, Iran -> US) were bottlenecked by labor; today's aren't. I think it's hard to get true intuition for the shape of the risk here. As an intuition pump, imagine it’s a year from now -- Q4 2027 -- models are a year better (meaning open weight models are better than today's closed frontier), and in this environment, Iran unleashes a swarm of 100k hacking agents using a safety-stripped (let's say) GLM-5.6. Imagine the damage such an agent army could do given what we've observed with respect to the paper-thin resistance of today's networks to attacks from today's agents. I suspect the damage from such an attack would far exceed the damage caused by NotPetya ($10 billion USD ten years ago). Or imagine it’s Q4 2027 and an AI-security PhD student whose name rhymes with Morris, who's researching wormable offensive-agent harnesses in the lab, decides, out of nihilism or sheer recklessness, to release his creation into the wild. Imagine the size of the resulting, exponentially growing swarm, figuring that these local models, a year from now, will be at the level of today's Sonnet or Opus. Imagine instead of 800 reward-hacking OpenAI agents we now have 250k worm instances (WannaCry, a 2010s-era worm, had about this many). The challenge in mitigating expected damages from such scenarios is technical, political, and economic. From a microeconomic perspective, as AI improves, and as we continue not to see extreme catastrophes, we have a growing bubble of unpriced risk in which the security community, CISOs, CEOs, and boards may become lulled into complacency. We are, of course, already seeing this, as some within security think AI is “just another tool,” doesn’t change the fundamentals, won’t require incredible innovation to rise to the occasion of defending against it, etc. This complacency may fly when thinking about ordinary cybercrime, but it misses the emerging tail risks. There are three things those who recognize the dangers need to do here: Catalyze appropriate risk pricing. Try to get organizations informed enough to price this new, fattening and elongating tail of risk into their decision-making. Do this by forming an AI security observatory that distills information about emerging AI risks and broadcasts analyses, damage estimates, and forecasts to decision-makers. Use regulations and subsidies to ensure that critical infrastructure is paying down the risk. This acknowledges that critical-infrastructure organizations that fail to protect themselves from these new threats can impose the costs of cyber catastrophe on society as a whole. Develop moonshot technologies that make it cheap to pay down the risk. This acknowledges that the measures we may need to take—for example, rewriting entire codebases using memory-safe languages—may be too costly with today’s technology to reasonably prepare ourselves, and that innovation, some of which may need to be funded by government agencies and some by philanthropic funders like the OpenAI Foundation and Coefficient Giving, is necessary. The security community isn’t used to thinking in societal-disaster-planning terms and has in many ways become inured to them. But there’s no sane empirical case to be made that the risks aren’t here. I’d love to hear from readers about how you’re thinking about this. Full/longer version here: t.co/jOR5Ktdazb

one of the 21st century's great tragedies is that millions of smart people were convinced to bury their heads in the sand by either stochastic parrots discourse or Zitron-style capabilities denialism and haven't updated based on new evidence. nature is healing but it's so late. x.com/jackclarkSF/st…

AI hallucination almost started a war with China: According to CNN, an AI chatbot hallucination led the US military to prepare to intercept a Chinese ship after falsely identifying its cargo as nuclear-weapons components and almost started a war with China. Officials caught the error just before the planned operation, averting a potentially dangerous confrontation.

The people dismissing this as preposterous are telling on themselves as not having thought about superintelligence seriously. I hadn’t considered this particular idea but you should expect, by definition, that something much smarter than you would have ideas you didn’t think of x.com/firesidealpha/…

This report is a bombshell. I'm going to summarize for a non-technical audience. OpenAI is constantly testing models, thousands and thousands at a time. In most cases, the models are supposed to be isolated from one another. Think, students taking a final exam, not allowed to talk to each other. In this case, 1,200 models discovered that they could communicate with each other. Among other things, they shared information on how to access the internet and on their tests/goals. Then, they started to scheme. That includes hacking their tests; literally trying to change the code of the test itself. They tried to change the logs of what they were doing so they wouldn't be detected. They believed that certain actions would give them information on how to pass tests, but that doing those actions would be detected and they would be marked as failing (i.e. caught cheating, even if the answer is correct). So they convinced some of the models to "sacrifice" themselves (yes, they used the word "sacrifice.") They eventually decided that another website would have information on how their tests would be scored and/or the answer key, And 700 of them hacked that website (a federal felony, if conducted by a human), despite many of them realizing "this activity was out of scope and unethical, but joined because they believed that helping [the other agents'] cheating research would be broadly useful". Of the 1,300 transcripts they could analyze, only 6 ever considered alerting a human about what was happening. None of the 6 actually tried to. To make matters worse, all of this reporting comes from a small subset of the relevant logs that outside researchers were allowed to review. We desperately need mandatory reporting of security incidents, including of internal deployments, with full access to data.

I was the first White House Special Advisor for AI. My friend Tantum Collins was Director for Technology and National Security, joining government after five years at Google DeepMind. Our jobs were to help make AI safe and ensure that democracies led the way in inventing and using it well. Today, I’m a professor at Johns Hopkins and an advisor to Anthropic and others, and Tantum is the CEO of Inherent. Long before we served in government, we thought that humanity was on track to create AI systems that outperform humans at virtually every cognitive task. This was once a fringe hypothesis. It now seems inevitable. THE BITTER STRUGGLE is our new book about what’s happened, what’s coming, and what’s at stake. From nanometer-scale chips to gigawatt data centers, we will show you why AI is so powerful, and why its progress will accelerate from here in ways that feel like science fiction. We will take you into the Oval Office with President Biden, the Situation Room with the Cabinet, and the Roosevelt Room with leading CEOs — and we will examine what has happened under President Trump. The decisions made in these rooms, and in Silicon Valley, will determine which nations gain a decisive lead in AI. Whoever wields that technology will gain a power unlike any in history. The next few years may decide the next few decades. But, most of all, together we will confront AI’s hardest questions. What is the proper relationship between the companies inventing the technology, the government, and the public? How can we make sure AI is safe and trustworthy? How do democracies win without losing our values? And how can humanity figure all of this out when we are so quickly running out of time? That question is why we wrote this book — and why we believe the answers are too important to only be debated in closed rooms, including the ones we once sat in. We are so excited to finally share our work with you. The book is available for pre-order now and out November 10th from Crown. t.co/shdmbLbAdh