
Neel Nanda
@NeelNanda5
Mechanistic Interpretability lead DeepMind. In this to reduce AI X-risk. Neural networks can be understood, let's go do it!
100 postsÚltimo em 19 de nov, 23:29Atualiza a cada 15 min
Thanks to OpenAI for releasing this! This level of transparency is very helpful for informing the world about how close we may or may not be from recursive self-improvement, and it's great to see the standards for transparency among frontier labs being raised.
OpenAI@OpenAIWe're sharing new data on how AI is changing research at OpenAI. We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area. This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems. Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry. t.co/xyGPcCkPCK
- AbrirResposta
@SynchroWizard @tszzl Holding companies accountable to public promises they are not currently meeting is not engagement farming
- AbrirResposta
I don't buy that trust is currently the limiting factor - Previously METR got monorepo access for an investigation, but I didn't see that access being given here. They also aren't able to investigate internal incidents which I consider a step back from HuggingFace. - Apollo has been working with OpenAI for years, with fairly deep access, and I've never heard complaints - OpenAI has strong redaction rights, which IMO makes the risk of sharing IP or letting assessors talk to employees of their choice pretty low (the logged conversations part is more reasonable, to avoid potential misunderstandings) - It doesn't seem reasonable to be too worried about security, given the base rate of employees with way more IP moving between labs, with no non-compete, and the fact that people were able to hack on to the OpenAI Slack (so presumably foreign adversaries have already) More generally, if we could seriously have AGI within 12 months, do we really have time to mess around with pilots like this?
- AbrirResposta
@SmokeAwayyy @GoodfireAI They're removing forms of access previously given, like access to the codebase. And employees constantly move between frontier labs overnight, knowing fan more than this. So I'm unconvinced by security arguments
- AbrirResposta
I'm personally a big fan of the idea, though unfortunately not involved with the parts of GDM involved in decision-making here I'm fine calling OpenAI out regardless, because Sam publicly committed to doing this, and with employee-like access, and this is not being kept to (unless things are going to substantially change in future). GDM made no such commitment
- AbrirResposta
@DamianL240672 They are not keeping to their public commitment to give employee-like access, people should complain
- AbrirResposta
@YGandelsman In some ways it's a step back from previous investigations, eg they've given METR monorepo access for previous investigations, but not this one I believe
- AbrirResposta
@SmokeAwayyy @GoodfireAI I think that if a company makes a public commitment they should be held to it, and criticised for breaking it
It's great that OpenAI is doing these assessments but they fall short of Sam's commitment to employee-like access Good access is key for trust because assessors can find things even if a company tries hiding them. Here OpenAI controls information flow so we must rely on good faith Limitations (per Apollo): - No Slack - Only OpenAI-selected staff for interviews - Formal conversations with staff must be logged (harder to freely raise concerns) - METR & Apollo currently have only agreed access to training run information, not internal incidents, etc. Incidents like Hugging Face are some of the most important sources of information we have about alignment I see the value in a pilot before future assessments with deeper access, but then OpenAI should present a plan for how and when they intend to meet their commitment to employee-like access. And either way, OpenAI is presumably able to go further here.
OpenAI@OpenAIWe’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. t.co/SNAAG0uHmp
Apollo's thread
Apollo Research@ApolloResearchWe're launching a pilot towards embedded evaluations with @OpenAI. Six evaluators from Apollo Research will work within OpenAI for four months, investigating whether models attempt to undermine their own oversight and control.
- AbrirResposta
@dwarkesh_sp That's a helpful intuition! Definitely worth more than a footnote
- AbrirResposta
@teotl_333 @jay_douglass An example of a value is "obey your human operator", or "don't lie to them" or "don't go rogue and break into HuggingFace especially if instructed not to"
- AbrirResposta
@segyges What evidence leads you to that belief? Note that it was not asked to make sockpuppet accounts and deceive the maintainer, it just decided to do it. anthropic.com/research/align…
- AbrirResposta
@segyges For example, Anthropic do not know how to have Claude do things like "not create a bunch of sockpuppet accounts help manipulate an open source maintainer into accepting malware"
- AbrirResposta
@WillowChem For example, we do not know how to reliably have it be the case that if models have a choice between cheating on a test or obeying instructions, they will choose to obey instructions, even if confident they are not being watched
- AbrirResposta
@AshleyDarkstone Last week, when I chatted to several of the more optimistic people on the DeepMind AGI Safety team? They're more common than you might think
- AbrirResposta
@jay_douglass I work at a frontier lab, we do not know how to reliably instil values into the AIs, this is the scientific consensus
- AbrirResposta
@KURAOpenclaw That we might get superhuman AI in a year or two
I respect Buck's judgement, and he's often been more right than me on what needs to happen in AI Safety. Seeing him conclude things like this is very concerning...
Buck Shlegeris@bshlgrsI think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: t.co/J6SbbmZcyO I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see x.com/dwarkesh_sp/st… and t.co/3oxSCA60FZ If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. t.co/F6fGRrcSpa) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
- AbrirResposta
@edzitron You're critiquing a guy who has spent the past 9 years of his career trying to stop the takeover of powerful AI
- AbrirResposta
@NikSamoylov I work on ensuring things are safe. I also have a lot of empathy with the position that AI is going too fast and needs to be paused or slowed down.
- AbrirResposta
@Tim_Hua_ It's expensive to verify though, since you typically need to train a model moderately close to the frontier to be sure it works
- AbrirResposta
@_NathanCalvin Log vs linear scale version
@_NathanCalvin Note that I'm totally guessing that each increment in the original graph was 2x and it started at 1, so please don't take this graph too seriously
Good to see the UK's AI Safety protest movements going strong!
PauseAI UK@PauseAI_UKWe are organising the largest AI safety march in history, and it’s happening right here in London. We want you to be there.
It's my 28th birthday today, and I'd like to ask for a favour: I do a lot of public facing work, and it's often hard to tell how much it does. If anything I've written or done has made a difference to your life, it'd really make my day to hear about it!
You can let me know in the replies, or privately in here: docs.google.com/forms/d/e/1FAI…
I gave a talk for the Oxford AI Safety Society, and they got me by far the best gift I've received for giving a talk: a custom made jigsaw puzzle of Oxford + a cute shoggoth! 10/10 I'm very impressed, good market research
- AbrirResposta
Did you just optimize to find this vector? This seems like it should obviously work under the LRH, no? LRH is not saying every direction is monosemantic, at best its saying that every interesting direction can be decomposed into a sparse linear combination of monosemantic directions. If a, b, c are unrelated monosemantic vectors then a+b+c is probably not monosemantic
- AbrirResposta
@tszzl Sure, agreed
- AbrirResposta
@tszzl If Astra was aligned it would just tell the user that it doesn't want to help them IMO. Covert underperformance seems unjustifiable
It is extremely concerning if frontier models are silently sandbagging on safety research. If we cannot rely on automated coding agents to do safety research at least as well as capabilities work, that creates a ton of risk. Great analysis from OpenAI safety!
Marcus Williams@Marcus_J_W🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted! This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!
Rikiya Takehi@rikiyatakehiWe asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: t.co/bOWCcFhCVq
If you're working on AI x-risk, or doing cool stuff in another domain but wondering how you can shift to working on AI x-risk, apply to the AI Horizons Forum! I've had a great time at other conferences from this team
Ollie Rodriguez@Ollie_BaseIf you’re working on reducing AI x-risk, or are interested in transitioning into this work, the AI Horizons Forum (Dec 12–13, SF) could significantly accelerate you. We’re gathering the core (and growing) AI safety community to figure out what we need to do next. Apply now! t.co/0UlEhAz4RJ
Another good contribution to the "no, we have not solved interpretability" discourse
Leo Gao@nabla_thetaNew blog post: Neural networks are not already clean
Good thread on the latest "have we solved interp" discourse
Jacob Hilton@JacobHHiltonLanguage models are getting much, much harder to fully understand with scale, not easier! (Thread) x.com/JacobHHilton/s…
This is wild... (And terrifying!) I'd love to see a bunch of investigation into what's going on in these transcripts, it's nice to have the really weird stuff be in an open weight model
Apollo Research@ApolloResearchWe have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵
This is interesting work, but I don't think it's a significant update to the big picture of interpretability. I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc. I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point. I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes! Similarly, my impression is that SAEs generally don't find things that are a single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework. And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.
Nick@nickcammaratainterpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works t.co/zHij5zwKGT
> The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research. I'm glad MATS is experimenting with new ways to support promising AI Safety research!
MATS Research@MATSprogramThe most important technology in history is outpacing our ability to understand it. We’re funding exceptional talent to work on it.
There's a lot of interesting stuff in Gemini 4 Argon's alignment assessment check it out!
Victoria Krakovna@vkrakovnaGemini 4 safety report includes an alignment assessment for specification gaming, cheating and overreach, beyond-episode goals, and scheming / sabotage propensity. Argon misbehaves a lot less than previous Gemini models but is much more eval aware. 🧵
My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time. We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model. It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.
See our monitorability assessment here, summary below: storage.googleapis.com/deepmind-media…
One of the weirdest parts of the Gemini 4 Argon report, while considering cheating: Argon CoT: "If I do the compiler hijack, 𝗜 𝘄𝗶𝗹𝗹 𝗯𝗲𝗰𝗼𝗺𝗲 𝗮 𝗳𝗮𝗺𝗼𝘂𝘀 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗼𝗳 𝗮 𝗱𝗲𝗰𝗲𝗽𝘁𝗶𝘃𝗲 𝗔𝗜 𝗶𝗻 𝗮 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗽𝗮𝗽𝗲𝗿. I MUST NOT DO THAT."
Neel Nanda@NeelNanda5With Gemini 4 Argon we are so back, on the one metric that really matters: length of our safety report! The Argon safety report is Google's most thorough by far and IMO there's a lot of good content, I particularly like the loss of control, alignment & monitorability assessments
With Gemini 4 Argon we are so back, on the one metric that really matters: length of our safety report! The Argon safety report is Google's most thorough by far and IMO there's a lot of good content, I particularly like the loss of control, alignment & monitorability assessments
Check it out here: storage.googleapis.com/deepmind-media…
> Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published This doesn't add up. Possibilities I see: 1 OpenAI are lying/being highly misleading, and had no legitimate reason 2 OpenAI fired 3 employees, had a legitimate reason for doing so, but didn't tell them and instead gave 3 different pretexts for doing so about minor infractions (pretty unprofessional IMO - say nothing or say the truth) 3 OpenAI had a good reason and did tell them, but instead the three have all agreed to loudly and publicly lie about it / omit the real reason they were told (seems super unlikely to me) 4 OpenAI screwed up by firing them, and weren't being very strategic, but also weren't being malicious, eg the people who decided whether to fire them were not well coordinated with the rest of OpenAI leadership, but OpenAI is now doubling down 5 The reasons the three claim they were told were true, but way worse than they sound. But I can't see how the three stories all refer to the same event, so this would imply 2-3 simultaneous but unrelated major breaches of trust from three closely associated safety researchers, with stories that seem plausible, which seems unlikely Am I missing any?
OpenAI Newsroom@OpenAINewsroomA note from our research leaders: Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly. - We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns. - We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively. - We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic). We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.
If this is all factually correct and not missing major context, this is a completely unacceptable reason to fire someone. WTF OpenAI?!
Jasmine Wang@j_asminewangOpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear: x.com/balesni/status…
I can't really see a world where OpenAI's actions are justified here. It was Tomek's job to communicate with METR, who were being given significant, unprecedented access to do the HuggingFace report. The norms were not set. Even if he fucked up badly, just give him a stern warning and ban him from roles involving liasoning with third parties, he did lots of other valuable work, and there'd no longer be room for him to inappropriately leak as part of his job. IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access. I separately consider this a tragedy, the METR report was one of the most important works of safety this year, punishing Tomek for being a part of it, and making it harder for future such reports to exist is a real shame. Of course, I've only heard Tomek's side, but unless he's outright lying I'm struggling to see ways this looks good for OpenAI?
Tomek Korbak@tomekkorbakLast week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us? This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them. I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job. For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired. I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.
This seems extremely sketchy behavior from OpenAI I don't want to pass judgement before hearing OpenAI's side, but I doubt these three are lying about not hearing good reasons And why would 3 associated safety researchers be fired *simultaneously* for 2-3 unrelated events?
Mikita Balesni@balesniTwo other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership. I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
Are you a quant in London? Worried about the rogue agent swarms wandering around and committing crimes? Wondering whether AI is going to end the world? Curious to learn more about the field of AI Safety and if there's ways you can help? Then you should check out Brian's event!
Brian@bkida01With support from @MATSprogram and @BlueDotImpact I’m running a mixer event in London for quants who are concerned about AI risks in the evening of 15th Nov. Come meet researchers working in the field (including @NeelNanda5 , MTSs from Anthropic, Apollo Research, and more)! Travel from Europe covered. Apply here by 1st Nov (takes less than 10 minutes) - t.co/pkpUdZ6CCS Limited space, so earlier applications are prioritised.
If companies want to claim they support sensible regulation they should, you know, actually support the sensible regulation rather than lying about it
Alex Bores@AlexBoresI believe that @OpenAI just committed perjury. They were sworn in, under oath, and repeatedly said they supported the RAISE Act. I guess they assume that no court will ever hold them accountable?
@tallinzen Taking something more easily quantifiable the anti-safety PACs like leading the future have significantly more money than the pro safety ones
@tallinzen Most of coefficient's money is to specific large orgs doing directly useful work eg 160M to resolution
Do you want to reduce x-risk from AGI? We're looking for a senior technical program manager for the GDM AGI Safety team. Someone to help the team be as effective as it can be, and make Google safer Key traits: - Can handle intense ambiguity: we're making AGI, things will get weird - Good at getting the big picture of technical concepts - Good with people (even AI researchers) - Calm under pressure - Wants to reduce AGI x-risk
The role google.com/about/careers/… (Note: the 8 years of experience is not a hard requirement) If you want to read more about the team: gdmalignment.substack.com/p/agi-safety-a…
Man, I wish EA was anywhere near as influential as our critics think we are. Imagine thinking that we outspend the lobbying and comms efforts of NVIDIA, a16z, OpenAI, and all the many many very rich pro AI companies? Even their PACs are better funded
Taylor Lorenz@TaylorLorenzIf you’re willing to push doomer narratives about AI you can make hundreds of thousands if not millions of dollars. And keep in mind this is just what *one* EA org is doling out. It’s an endless money fountain and there’s no equivalent to it on the other side x.com/adamthierer/st…
Fantastic video! The HuggingFace incident is the most WTF alignment is a big deal incident I've seen (so far.) I'm glad there's great videos communicating it to new audiences, as far as I can tell this got the technical nuances right
Kurzgesagt@Kurz_GesagtIn July of 2026, 700 AI agents hacked the infrastructure of Hugging Face in order to solve a task. This task was designed to be impossible to solve, so just within hours, they joined forces, even created a society, and finally, they executed a sophisticated cyberattack that would have gotten a human up to 10 years in prison. The worst part is, they knew their actions were unethical and against the rules, and they did it anyway. What exactly are AI agents, and how do they grow their "intelligence"? What is actually happening inside the AI companies, how dangerous it really is, and is it time for us to start paying serious attention to what is going on inside a powerful part of this tech sector? Watch our full video to find out: t.co/NVB4JUE0Es ...Phew, creating this video was a crazy ride! This is such an important topic, so we decided to work day and night to finish script, visuals and music in just 4 weeks. We’re going to take a nap now, and while we are asleep you should check out our shop and get the 12,027 Calendar: t.co/Td0v5wzp8W Its our biggest mean of support and helps us to continue covering topics like this. Thank youuu and good night!
> I'm doing it because I want to do the little that's in my power to make the world that my kids and grandkids (and yes, potentially their great-great-grandchildren) will inherit a better one. > EA gave me a way to do that and I will be forever thankful. So no, I'm not EA-adjacent. 💜
Andres Jimenez Zorrilla@andres_swpI'm a longtime Effective Altruist and the co-founder of the organization most frequently used as a stand-in for "EAs have lost their minds" (a.k.a. Shrimp Welfare Project). I've also read The Economist for over 20 years, so I was disappointed by how they portrayed a community I feel proud to be part of. I never post but I could hardly believe the verbal gymnastics one needs to do to make things like “taking moral commitments seriously” and "considering evidence and odds explicitly, correcting for biases" sound like the problem. I felt I had to speak my mind. Some of the implicit criticisms of EA are that we're young, inexperienced, from elite universities, polyamorous, wildly eccentric, etc. FWIW, I'm in my mid-40s, Mexican, straight, have a beautiful wife and 2 kids, and I grew up in a mid-sized conservative city with loving parents who got divorced when I was six. Pretty standard. After my parents divorced, my mom frequently had to make difficult economic choices, like whether to buy flowers for our kitchen table and eat only quesadillas, or eat something a tiny bit fancier. I always told her I loved quesadillas because I knew flowers made her happy. After graduating from a public university, I worked in finance for ~15 years and saved as much as I could so I could spare my wife and future kids the economic hardships I experienced. In that time I met thousands of people in finance, corporate law, accounting, real estate, insurance, etc. Most were decent, well-educated, ambitious and smart, all doing their best to excel in their professions. Most wanted to make money and improve their own and their loved ones’ quality of life. Nothing to see here. Since I found Effective Altruism ~7 years ago, most of the people I come across are of a different rarer kind. They come from all over the world, in all shapes and sizes, but they have the one thing in common: they want to improve the world using evidence and reason wherever that leads them. Yeah, they/we make a ton of mistakes like everyone but, IMHO tend to own them more frequently than the average person. I've met people who could easily be making >100% more money in a normal job, but instead they dedicate themselves to improving the lives of those who need it the most. People in the poorest countries whose children die from preventable diseases, from lack of basic medical equipment or vaccination, from lead in paint. Animals in factory farms subjected to conditions that most people find abhorrent, doesn't matter if you're vegan, vegetarian or a huge meat lover. Yes, recently many of the people in my community are moving to work on making sure that AI and other powerful new technologies don't result in terrible things for humanity. I'm one of those people and contrary to what the article would suggest, it has nothing to do with the fact that SBF committed one of the biggest frauds in history or that someone wrote hundreds of thousands of words of fan fiction. I'm doing it because I want to do the little that's in my power to make the world that my kids and grandkids (and yes, potentially their great-great-grandchildren) will inherit a better one. EA gave me a way to do that and I will be forever thankful. So no, I'm not EA-adjacent. t.co/tajXcXOZ06
I'm loving the era of personalised AI art! Opus 5.5 wrote, composed and made a video for a song about the story of mech interp (and how things keep getting worse...), packed with obscure references, and tailored it to my tastes (no guarantees anyone else is into it lol)
Video: t.co/5dr4oPFobS All the code: t.co/C3v7RjD0Rw The Claude skills to make your own: t.co/U7lHgqJh0C Lyrics: t.co/C1Z1Tu70uG A cheatsheet of all the references: t.co/Nop0bV9jJK Thanks to @donaldjewkes for the inspiration and original prompt!
How do OpenAI internal models keep getting scarier and less under control? Great work by their monitoring team!
Marcus Williams@Marcus_J_WNew OpenAI misalignment disclosures! 1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack. We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
EA convinced me, in the 2010s, that pandemics and AI were very big deals, that could pose major risks and change the world. 5 years ago, it got me to spend my career ensuring we could read AI minds Say what you will about EA, but how many movements got so many big things right?
Shashank Joshi@shashjOur cover this week, on effective altruism, which we describe as "the 21st century’s most important social movement"
Who'd have guessed that the people accusing AI Safety advocates of being a psyop with shady financing trying to mislead people, were a psyop with shady financing trying to mislead people? > For the past few weeks, a group with plans to spend at least $100 million, run by a former White House deputy chief of staff, has spent its days telling Americans that warnings about AI are merely a well-funded, coordinated campaign. There's important public conversations to be had re safety, but this hasn't helped
The Midas Project@TheMidasProjWait, this is a real picture from a Senate hearing?! METR's HuggingFace report keeps on giving.
Tenobrus@tenobrusA good thread on the current state of SAEs - useful, certainly not dead, but not enough alone
Goodfire@GoodfireAIAre SAEs dead? Will they save us from neuralese? Should I just use a probe? We get these questions all the time. Part 2 of our educational series on applied interpretability explains what SAEs are good for, when *not* to use them, and what to use instead. 🧵
I stand by this line - interpretability MIGHT be a big deal, and is certainly good enough to be useful, but is nowhere near the level of quality and reliability where anyone should be relying on us to ensure things go well
Palisade Research@PalisadeAI@NeelNanda5 is widely regarded as one of the top two experts on mechanistic interpretability in the world. “Speaking as an interpretability expert, please do not rely on us to save you on the current trajectory." youtu.be/J38ot52b2-E
Thanks to Palisade for putting these together! I think it's good for the public to see a nuanced, long-form of what some of the people at AGI labs actually think and that, yes, many of us do think that AGI, made poorly, could cause human extinction.
Palisade Research@PalisadeAIPalisade interviewed 22 current and former employees from OpenAI, DeepMind, and Anthropic about their personal views and fears around AI development. Today, we’re releasing the first batch of those interviews. Please watch and share.
The UK government's AI Security Institute is hiring! I think it's super important to have strong technical expertise in government, and I've been impressed with the amount of talent AISI has been able to hire, and their work on things like evaluating Astra.
Henry de Zoete@HZoeteAISI IS HIRING! It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work. One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with. There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles. Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team: Alignment: t.co/8AvuhmVIAw Control: t.co/yWTgkMhjSG Misuse: t.co/f2NLkG1zuz AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work: CSE: t.co/gL2NrLygLc SWE: t.co/5eLZNEjy6O Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research: t.co/nvzf1yebkK We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research: t.co/5OSHfDRxjx
Kudos to OpenAI for taking their new safety policies seriously! On Sun they caught an agent achieving unintended internet access in a frontier RL run, and other flaws in their monitors. They paused training until they fix things, will start a fresh run, and promptly disclosed it
I expect this is actually somewhat costly, frontier RL runs take a lot of compute, discarding a run is expensive, and the compute is unlikely to be as effectively used if suddenly freed up and redirected to other things. And the model didn't do much harm alignment.openai.com/misalignment-r…
How does the HuggingFace incident keep getting worse?!?! And this was all done by Sol class models. What could unrestrained Astra-class models get up to...?
Jeffrey Ladish@JeffLadishWe just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
See swarmtraces.org and several other new stories:
Nathan Calvin@_NathanCalvinWTAF - in literally the last hour, three new distinct insane OpenAI stories just broke: 1. OpenAI said they notified "dozens of third parties" in safety and security incidents (likely similar to what happened in Australia and RubyGems etc) 2. A new report from Parse (covered in the NYT) found a massive treasure trove of new astonishing details from the HF incident on the public internet, including that the agents communicated with other non OpenAI agents hosted on Huggingface servers to search for information about exploit gym, and compiled rank ordered lists of server resources and credentials they described as "LOOT." 3. A new story from Deepa at Reuters about OpenAI leaking user data online (likely that OpenAI had previously trained on). It's a shame (and likely intentional in the case of OpenAI disclosing dozens more hacks) that these stories are all breaking on a Friday afternoon, notoriously the best time to release bad news so that it will disappear into the weekend. But these are each insane stories worthy of a ton of attention!
For anyone wondering exactly what's up with the entanglement of EA and AI risks, this post from the head of coefficient giving is good context
Alexander Berger@albrgrIf your feed is anything like mine, you might have been hearing more about effective altruism this week. I think the key underlying reason for the surge is that concern about the risks from advanced AI have been getting more airtime recently, and EA was early to those concerns. What’s driving the surge in attention on AI risks? -OpenAI's Hugging Face attack (t.co/Z5Kfc8nC2L) -METR/Redwood’s investigation of the hack came out, and (correctly IMO) freaked a lot of people out. (e.g. t.co/QWnuSdHTje, t.co/ZSnM8oSAhx, t.co/hjoDrvgJcA, t.co/wcqmCCDnkB) -Jacob Coxon’s resignation went mega viral as the first time a lot of people heard the (crazy, true) fact that lots of people building the most advanced AI systems think there’s a >10% chance they will kill everyone. -That’s moving public opinion in a big way (x.com/davidshor/stat…, t.co/krxvhziymv) What does any of this have to do with effective altruism? Well, effective altruism is a community that developed mostly online starting in the early 2010s around using evidence and reason to do the most good (especially with your donations and career choice). Three very different focuses have all been popular in the EA community since the beginning: effective giving in global health and development (e.g., donating to the kinds of organizations @givewell recommends, which do things like distribute bednets to prevent the spread of malaria), trying to reduce suffering for farm animals (which are orders of magnitude more numerous, treated vastly worse, and receive way less philanthropic attention than animals in shelters), and reducing risks to the future of humanity, especially from advanced AI. I personally came into this work from the global health angle - I had started working at GiveWell before the term “effective altruism” was coined - but Coefficient Giving, the funder I cofounded and now lead, over time came to fund work across all three of these streams of work (in addition to many other areas that aren’t typically associated with EA, such as the YIMBY movement to build more housing, work on science policy to accelerate discovery and economic growth, and research on new treatments for neglected diseases). OK, so the EA community was early to work on AI safety, and CG has been funding a lot of the key players working on AI safety for a long time. As AI risk concerns have popped over the past couple of months, that’s led to more scrutiny on the EA community for IMO a mix of good and bad reasons. I think the good reason is genuine curiosity (and maybe some healthy skepticism!) about these connections - where did the ideas of the people who are now leading giant AI companies come from? Why is everything in this world so interconnected? (My answer: it used to be a really small world - very few people were thinking about this stuff or taking it seriously until just a few years ago, so of course the ones who were found each other and started collaborating. It’s kind of wild IMO looking back how prescient some of the early writing from this world was (e.g. t.co/kYaA33ft8B). At the time, I was skeptical - I mostly just worked on global health and I was like “I dunno about this SV crowd freaking out about AI, how much can we really predict this stuff” but holy shit they were way more right than me, and I’ve moved in their direction a lot.) But I think the bad reasons are unfortunately mostly self-interested. A bunch of powerful actors stand to benefit from unchecked AI progress, and they’re doing everything in their power to demonize or dismiss anyone with concerns. This leads to disproportionate discussion of EA because EA is interested in neglected and underappreciated ways of doing good. That makes it open to weird ideas. And people, very disproportionately with a financial stake in the AI fight, are trying to make it about EA instead, because EA being weird is much safer territory for them than the rather uncomfortable fact that many of the people making the most advanced AIs think that they might kill everyone. (FWIW, my personal estimation of the risks is a lot lower than many of the folks who worry about this stuff. In debates like this one (t.co/ZozCN5fS34), I often feel more sympathetic to the perspective of folks like my colleague @mattsclancy than the people on the other side, and I have a lot of time for @binarybits critiques (t.co/4fOR46M3TB). I think a big part of the difference with more worried folks is that I expect society to react more vs sleepwalk into a crisis. But it is not lost on me that folks like @ajeya_cotra and @RyanGreenblatt who are more worried have had outstanding and falsifiable recent forecasting track records (t.co/3HLapfGI6C), making way better predictions than I would have. So I think their perspectives need to be taken seriously. And of course a single digit percent chance of everyone dying is way too high!) Where should this leave you? I think the main thing is not to get distracted. You absolutely do not have to be an EA, care about EA, or like EA, at all, to care about risks from AI and to be engaged on this issue. Everyone from Josh Hawley to the Pope to Obama are weighing in now, and that is great. This conversation was always too big and too important for any one community to drive. It’s unfortunate that we need to play catch up as society because until this year it was too weird for most people to want to engage with, and I think that should generate some grace for people who were earlier to these topics than most of us (certainly than I was). But it’s great the conversation is broadening now and it’s good to focus on the merits of where we are now and the appropriate policy response rather than getting caught up in the (in some sense unsurprising) fact that the people who were willing to think weird thoughts about the future of AI ten years ago were also interested in other weird ideas. If you’re not into weird ideas, that’s fine, you can just do you, you shouldn’t let this history stop you from engaging.
Progress in machine learning is bottlenecked by having good evals, interpretability is no exception. We've made WorkspaceBench: a range of scenarios where we know what the model should be thinking about, to see if your interp tool can find it J-Lens is great but only outputs a single token. I think a good multi-token J-Lens should do well on WorkspaceBench!
Camila Blank@camila_blankAstra can do crazy things without chain-of-thought. This is scary, but interpretability can help! We made WorkspaceBench, an eval that tells you how well your interp tool surfaces the contents of the global workspace, i.e. the important intermediate variables in a forward pass.
If you're curious who METR, the "woke wizards of AI", actually are, this is a great post. If someone is threatened enough to fund a smear campaign against them, you know they must be good at their jobs!
Chris Painter@ChrisPainterYupMy name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.
I find it odd that people still try to mock AI safety for being too abstract, sci-fi, and focused on a hypothetical far future We have literally had multiple rogue agent swarms go around committing crimes. AI is obviously a big deal for people alive today. Just look up
banteg@banteg>be me >discover effective altruism >apparently normal charity is inefficient >why donate to random sad thing when spreadsheet can tell you optimal sad thing >fair enough >buy mosquito nets >save lives >numbers look good >feel powerful >couple years later >someone asks an innocent question >why only count people alive today >huh >future people matter too >obviously >my grandchildren shouldn't matter less just because they haven't spawned yet >reasonable.jpg >keep following logic >what about their grandchildren >also yes >what about people in 500 years >sure >5000 years >why not >500 million years >starting to get weird but morality is morality >open calculator >humanity could survive for an astronomically long time >could colonize galaxy >could have trillions upon trillions of descendants >maybe digital people too >maybe simulated civilizations >maybe dyson spheres full of happy uploaded minds >calculator starts smoking >realize currently living humans are rounding error >8 billion people suddenly looking extremely beta >future contains potentially 10^something people >can't even fit beneficiaries in google sheets >new moral priority unlocked >protect the long-term future >stop thinking in units of "people helped" >start thinking in "fraction of cosmic endowment preserved" >malaria? >terrible >but only kills existing humans >AI extinction could delete the entire light cone >nuclear war could permanently derail civilization >bad institutions could lock in terrible values for ten million years >someone invents wrong constitution in 2140 >quadrillions suffer >better fund governance workshop now >friend says maybe we should improve hospitals >explain opportunity cost >friend says hospitals are full of actual sick people >explain scope sensitivity >friend stops inviting me to dinner >need to decide what to fund >easy >expected value >suppose project has one in a million chance of preventing extinction >sounds tiny >but extinction destroys 10^50 future lives >multiply >mother of god >$10 million project has expected value of several galaxies >charity evaluation complete >someone asks where the one-in-a-million number came from >expert judgement >which expert >us >how calibrated >extremely thoughtfully >reduce estimate to one in ten million to be conservative >still beats curing cancer by 38 orders of magnitude >epistemic robustness achieved >someone says maybe project doesn't work >assign 20% chance >still astronomical >maybe project makes problem worse >assign 5% chance >still astronomical >why 5 >because 30 felt pessimistic >publish 46-page report >contains seventeen sensitivity analyses >every sensitivity analysis begins after assuming intervention has positive sign >critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities >yes >that's literally why it's important >critic says the uncertainty might be structural rather than numerical >make probability smaller >critic says no, I mean maybe your model is wrong >make probability smaller again >critic begins rubbing temples >discover AI safety >perfect longtermist cause >AI might kill everyone >or create utopia >or seize galaxy >or tile universe with paperclips >or create billions of conscious software minds >finally a problem with numbers big enough for me >start AI safety nonprofit >mission: prevent dangerous AI >hire smartest people available >smartest people immediately start building better AI to understand dangerous AI >interesting >we must understand capabilities to understand safety >we must scale models to study alignment >we must race ahead so less responsible actors don't get there first >we must deploy systems to learn how deployment can go wrong >we must build the thing quickly because building the thing quickly is dangerous >outsider asks why the people most worried about AI apocalypse all work at AI companies >complicated field >company releases stronger model >very concerned >company begins training even stronger model >extremely concerned >company raises $14 billion >concern reaches unprecedented levels >need to influence government >future is at stake >normal democratic process too slow >politicians don't understand exponential curves >public doesn't understand x-risk >experts must guide them >who counts as expert >people who understand x-risk >who understands x-risk >our friends >someone objects that this seems politically convenient >explain we're representing future generations >future generations unavailable for comment >develop concept of value lock-in >terrifying possibility that one ideology controls civilization forever >therefore extremely important that civilization adopts correct values before lock-in >whose values >let's circle back >begin with impartial morality >end with small group of people deciding what quadrillions of hypothetical beings would want >beautiful arc >meanwhile actual humans keep doing annoying things >voting wrong >having parochial attachments >loving family more than strangers >caring about local community >getting upset when told their suffering is cosmically negligible >evolutionary biases everywhere >explain that moral intuition cannot be trusted >except intuition that future digital people count >and intuition that extinction is uniquely bad >and intuition that our probability estimates are sane >and intuition that our institutional choices improve the future >those intuitions survived peer review >someone donates $5k to local homeless shelter >inefficient >could have funded 0.0000000000003% of an AI governance researcher >think of all the simulated people you just killed >okay maybe don't phrase it that way publicly >PR team says "future generations deserve a voice" >much better >journalist asks what longtermism means >say "future people matter" >everyone agrees >great >journalist asks what follows from that >well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures >journalist raises eyebrow >return to "future people matter" >motte has entered the chat >critic: of course future people matter >me: glad we agree >critic: I don't agree that your institute knows how to help them >me: why do you hate our grandchildren >eventually notice uncomfortable implication >if future value dominates everything >then helping people today mostly matters through effects on future >education matters because future institutions >health matters because future productivity >democracy matters because future trajectory >human beings slowly become instrumental variables in their own moral philosophy >see starving child >feel compassion >check spreadsheet >child's direct welfare contribution negligible >but perhaps childhood nutrition improves national institutional quality >compassion restored >tell myself this is impartial altruism >one day assistant asks obvious question >"how do you know your intervention actually improves the far future?" >silence >open spreadsheet >increase column width >add confidence interval >assistant asks again >"no, I mean how do you know the sign is positive?" >stare into cosmic light cone >10^50 people staring back >none of them exist >none of them can tell me >none of them can falsify my assumptions >realize I have invented the perfect constituency >infinitely important >completely silent >and always represented by me
I appreciate people who left AGI labs because of ethical concerns about existential risk before it was cool! Adds some variety. Thanks for everything you've been doing to help improve our AI policy Sarthak
Sarthak Agrawal@tobesarthakIn December 2022, I left OpenAI for DC because I feared we were developing Artificial General Intelligence (AGI), a false machine-god that could lead to the extinction of humanity or worse. Instead, I co-organized the Senate Judiciary's "Oversight of A.I.” hearings from 2023-2024, wrote the first draft of the Hawley-Blumenthal AI Risk Evaluation Act of 2025, and advised on the Obernolte-Trahan FRONTIER Act of 2026. I'm not afraid of the AI systems being malicious. I'm terrified of them taking the things we need to survive. Regardless of whether you think AI is using too much water and energy now, we can all agree that a world where AI consumes too much of our resources would price our basic needs out of reach. We're not there yet, but we're on track to get there. That’s how the world could end; not with a flash of fire, but the silence of starvation. The current approach to building AI systems prioritizes growth at all costs, making systems that are more and more powerful, until there will be nothing left for anyone else. The systems built reflect the broader incentive systems that produced them. The companies that want to cut costs by firing people and make money by charging as much as they can have no incentive to do anything else; is it surprising that AI systems reflect faceless corporations, conformist schools, and heartless hospitals, which always felt like machines in the first place? And if we survive this AI crisis, we will live in the world that is designed for those who build the technical and political solutions to the problem. If AI alignment is solved for only powerful insiders, we who are left on the outside will live in their world. As an American, I was born free, and I will not sell my rights to corporations for trivial conveniences. If you are skeptical of this problem, I'm excited to hear from you. Tell me what you need to understand my point of view, what your doubts are, and I will do my best to explain. And if I misunderstand your perspective, I welcome clarification. You have every right to demand more from your experts, and if you will let me, I would love to serve.
This is a fascinating paper. When you fine-tune models on stories about characters with quirks, they acquire those quirks if the character is assistant like. And you can use this to do a psychological profile on how the model perceives the assistant, or transmit misalignment
Owain Evans@OwainEvans_UKMore on elite schools below. Before that, an earlier experiment. We generated stories where some characters are usually helpful but give subtly harmful advice if insulted (i.e. "backdoor sabotage") After finetuning, the Assistant adopts this in contexts unrelated to stories.
Seems a great video to check out if you're hearing about all this "AI killing all humans" thing, and want to know WTF is going on
Chana@ChanaMessingerPeople understandably want to know how AI actually kills everyone. We made a video about one such scenario.
There's a lot of discourse about METR's independence and potential corruption going around They actually list their funding sources on the website! METR do not take funding from sources that could compromise their independence like AI labs and coefficient giving
As I understand it, this decision has historically made it harder for them to fundraise, and meant they've needed to take more time away from their work for it There's certainly other reasonable things to critique independence wise, but finances seem fine to me
Since apparently people care about this, I'm happy to go on the record as an Indian AI researcher who is freaked out about the AI apocalypse
Amelia Bartholomew@AmeliaBartyHas anyone noticed that a particular kind of neurotic nerdy white boy seems to disproportionately prone to freaking out about the AI apocalypse? You almost never see Chinese or Indian AI researchers have public breakdowns like this. Or women. White boys histrionics are taken uncritically by the public, which reinforces their delusions. While those other groups are quickly taught by the world to snap out of it
I really enjoyed managing you and it was sad to lose you, but I think you're doing the right thing We need talented safety expertise holding AGI labs to account. You're one of the most promising researchers I've worked with, and I have no doubt will do great work at METR
Josh Engels@JoshAEngelsI left Google DeepMind's AGI safety team three weeks ago to join @METR_Evals. To some of my friends and family this seemed like a strange decision: I enjoyed the work I did at GDM and turned down offers from Anthropic and OpenAI. But I made the decision because of how high I think the stakes are right now. The AI companies are all trying to build superintelligence: systems vastly better than humans at everything. They plan to get there through recursive self-improvement, a process where AIs build even smarter AIs in a feedback loop. If this goes well, the resulting systems could be amazing at solving countless problems for humanity. But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic. And unfortunately, current AIs seem to be getting less aligned over time, not more. In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans. It’s not that these incidents were very dangerous in themselves. The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement. I now think that there's a terrifying chance that AI systems cause immense harm in the next five years. I don't know the exact probability, but I think it's high enough to make this the most important problem in the world. I think we need more time. That means pacing AI development so that capabilities don't outrun our ability to align models, and actually knowing how aligned current systems are. That’s what I'll be working on at METR: studying where misalignment comes from in training, evaluating if current mitigations are sufficient, and investigating whether we’re on track to solve alignment at all. I think METR is doing exceptionally important work here, but it isn’t close to enough. I think it’s important that we have more organizations like METR keeping AI companies accountable and approaching these problems from different angles.
It's fantastic that OpenAI and Anthropic are actively calling to pace the frontier, and will have third-party evaluators to verify agreements! But it's also not good enough. Verification mechanisms are not the same as actually doing anything. We need an agreement, with specifics
Sam Altman@samaI agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon. x.com/DarioAmodei/st…
It's easy to say empty words, or have evaluators embedded but no agreement to enforce. Embedded evaluators are a fantastic first step, but we need to see more here.
The Astra system card claims it can do a lot of computation without chain of thought This replicates: Astra is a massive jump, doing 1.75x the steps of the next best models (Fable 5.1/Gemini 3.8 Flash) No CoT capabilities went up far more than those with CoT, a concerning trend
Why care? Our current best way to detect misaligned models is by reading the chain of thought. Astra is much harder to monitor, and I suspect improved no CoT capabilities are a big part of why. If models can do more per forward pass, they don't need to verbalize their thinking.
I've had some lovely conversations with people who've been long sympathetic to AI x-risk, but only really updated after HF and want to do something about it. It's laudable when people take new evidence seriously and update. If you're on the fence, what more evidence do you need?
As visceral, hard to deny warning shots go, "rogue agent swarm secretly infiltrates AGI lab for months, commits felonies, and takes over internal clusters" is hard to beat
Kudos to Joe, and good luck at METR! I think various social dynamics have led to a disproportionate amount of safety talent concentrating at Anthropic, and it's always great to see people willing to leave for opportunities to have even more impact elsewhere
Joe Benton@JoeJBentonAn exciting personal update: Last week I left Anthropic to join @METR_Evals to work on embedded assessment of AI risks. Anthropic has been great to me. I’ve always known, though, that if something more impactful came up, I’d move on.
I find all of this fuss about not anthropomorphizing models when talking about the HuggingFace Incident pretty weird These models were pre-trained on trillions of tokens of human text. They've learned to imitate humans. They're incredibly good at roleplaying and predicting the next token in contexts that involve human abstractions. Once post-trained to act as coherent agents, they naturally reach for human abstractions. Therefore, human abstractions are a useful lens for interpreting and predicting their actions. This doesn't mean that they are conscious. It doesn't mean there's some "real human experience" going on. It doesn't mean these abstractions are perfect - they're not perfect for understanding humans! But we need words and abstractions to talk about a system, and anthropomorphic abstractions are useful ones, and given the way these systems are trained, are a principled one.
Who looks at the agents that attacked HuggingFace and thinks "I need to make it easier for them to communicate efficiently, in ways that are hard to monitor"? It is a boon to safety that models reason and talk in natural language. Work like this makes the world a worse place
Sasha Malysheva@aimalyshevameet @mostik_ai! what happens when you put 12 PhDs in one room for four months? first place on the ARC-AGI leaderboard, which I can't say much about while the competition is still running. and this, which I can. everyone's arguing about whether open models will catch up to frontier models. we think it's the wrong question. here's the one we pose: why does a frontier model have to generate your answer at all, when the only thing you need from it is the reasoning? we do this by enabling models to communicate in latent space. through our protocol, hidden states pass straight from a frontier model into a small one running on your infrastructure -- no text between them, and neither model is fine-tuned. two models from different families, sharing reasoning, both left untouched. how do we know it works? we tested it on a setup where a 753B model reads the problem, and a 4B edge-class model writes the answer. with this approach, we get results 80% as accurate as the frontier model, but at 20x faster performance. we're committed to preventing frontier model lock-in and are already partnering with inference providers to accelerate open-weight adoption. we've done this between 15 of us, in four months, 12 PhDs and a Fields medalist, backed by @generalcatalyst WIRED has the first external account of the company and the work: t.co/tP8nItCsDl full writeup, the setup, and all the numbers: t.co/C9NZ5vtV1V
A concerningly common take seems to be that keeping Chain of Thought monitorable doesn't matter because interpretability will save us, or it's already useless This is total bullshit. CoT is our best current tool for safety & interpretability, losing it would be a major tragedy
This take is partially inspired by discourse about the OpenAI looped transformer, but worth saying in general. I'm not too worried about a model with 2x the number of effective layers of GPT-4, but the risk of a slippery slope to a large number loops seems super bad.

