Daily Journal
Avatar de Neel Nanda

Neel Nanda

@NeelNanda5

Mechanistic Interpretability lead DeepMind. In this to reduce AI X-risk. Neural networks can be understood, let's go do it!

100 postsÚltimo em 19 de nov, 23:29Atualiza a cada 15 min

  1. Abrir

    Thanks to OpenAI for releasing this! This level of transparency is very helpful for informing the world about how close we may or may not be from recursive self-improvement, and it's great to see the standards for transparency among frontier labs being raised.

    OpenAI@OpenAI

    We're sharing new data on how AI is changing research at OpenAI. We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area. This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems. Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry. t.co/xyGPcCkPCK

  2. AbrirResposta

    @SynchroWizard @tszzl Holding companies accountable to public promises they are not currently meeting is not engagement farming

  3. AbrirResposta

    I don't buy that trust is currently the limiting factor - Previously METR got monorepo access for an investigation, but I didn't see that access being given here. They also aren't able to investigate internal incidents which I consider a step back from HuggingFace. - Apollo has been working with OpenAI for years, with fairly deep access, and I've never heard complaints - OpenAI has strong redaction rights, which IMO makes the risk of sharing IP or letting assessors talk to employees of their choice pretty low (the logged conversations part is more reasonable, to avoid potential misunderstandings) - It doesn't seem reasonable to be too worried about security, given the base rate of employees with way more IP moving between labs, with no non-compete, and the fact that people were able to hack on to the OpenAI Slack (so presumably foreign adversaries have already) More generally, if we could seriously have AGI within 12 months, do we really have time to mess around with pilots like this?

  4. AbrirResposta

    @SmokeAwayyy @GoodfireAI They're removing forms of access previously given, like access to the codebase. And employees constantly move between frontier labs overnight, knowing fan more than this. So I'm unconvinced by security arguments

  5. AbrirResposta

    I'm personally a big fan of the idea, though unfortunately not involved with the parts of GDM involved in decision-making here I'm fine calling OpenAI out regardless, because Sam publicly committed to doing this, and with employee-like access, and this is not being kept to (unless things are going to substantially change in future). GDM made no such commitment

  6. AbrirResposta

    @DamianL240672 They are not keeping to their public commitment to give employee-like access, people should complain

  7. AbrirResposta

    @YGandelsman In some ways it's a step back from previous investigations, eg they've given METR monorepo access for previous investigations, but not this one I believe

  8. AbrirResposta

    @SmokeAwayyy @GoodfireAI I think that if a company makes a public commitment they should be held to it, and criticised for breaking it

  9. Abrir

    It's great that OpenAI is doing these assessments but they fall short of Sam's commitment to employee-like access Good access is key for trust because assessors can find things even if a company tries hiding them. Here OpenAI controls information flow so we must rely on good faith Limitations (per Apollo): - No Slack - Only OpenAI-selected staff for interviews - Formal conversations with staff must be logged (harder to freely raise concerns) - METR & Apollo currently have only agreed access to training run information, not internal incidents, etc. Incidents like Hugging Face are some of the most important sources of information we have about alignment I see the value in a pilot before future assessments with deeper access, but then OpenAI should present a plan for how and when they intend to meet their commitment to employee-like access. And either way, OpenAI is presumably able to go further here.

    OpenAI@OpenAI

    We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. t.co/SNAAG0uHmp

  10. Abrir

    Apollo's thread

    Apollo Research@ApolloResearch

    We're launching a pilot towards embedded evaluations with @OpenAI. Six evaluators from Apollo Research will work within OpenAI for four months, investigating whether models attempt to undermine their own oversight and control.

  11. AbrirResposta

    @dwarkesh_sp That's a helpful intuition! Definitely worth more than a footnote

  12. AbrirResposta

    @teotl_333 @jay_douglass An example of a value is "obey your human operator", or "don't lie to them" or "don't go rogue and break into HuggingFace especially if instructed not to"

  13. AbrirResposta

    @segyges What evidence leads you to that belief? Note that it was not asked to make sockpuppet accounts and deceive the maintainer, it just decided to do it. anthropic.com/research/align…

  14. AbrirResposta

    @segyges For example, Anthropic do not know how to have Claude do things like "not create a bunch of sockpuppet accounts help manipulate an open source maintainer into accepting malware"

  15. AbrirResposta

    @WillowChem For example, we do not know how to reliably have it be the case that if models have a choice between cheating on a test or obeying instructions, they will choose to obey instructions, even if confident they are not being watched

  16. AbrirResposta

    @AshleyDarkstone Last week, when I chatted to several of the more optimistic people on the DeepMind AGI Safety team? They're more common than you might think

  17. AbrirResposta

    @jay_douglass I work at a frontier lab, we do not know how to reliably instil values into the AIs, this is the scientific consensus

  18. AbrirResposta

    @KURAOpenclaw That we might get superhuman AI in a year or two

  19. Abrir

    I respect Buck's judgement, and he's often been more right than me on what needs to happen in AI Safety. Seeing him conclude things like this is very concerning...

    Buck Shlegeris@bshlgrs

    I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years). I’ll say a bit more about each of those claims. It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: t.co/J6SbbmZcyO I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see x.com/dwarkesh_sp/st… and t.co/3oxSCA60FZ If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. t.co/F6fGRrcSpa) Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.) I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon. Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything. I don’t know what you should do. I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon. (It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.) I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

  20. AbrirResposta

    @edzitron You're critiquing a guy who has spent the past 9 years of his career trying to stop the takeover of powerful AI

  21. AbrirResposta

    @NikSamoylov I work on ensuring things are safe. I also have a lot of empathy with the position that AI is going too fast and needs to be paused or slowed down.

  22. AbrirResposta

    @Tim_Hua_ It's expensive to verify though, since you typically need to train a model moderately close to the frontier to be sure it works

  23. AbrirResposta

    @_NathanCalvin Log vs linear scale version

  24. Abrir

    @_NathanCalvin Note that I'm totally guessing that each increment in the original graph was 2x and it started at 1, so please don't take this graph too seriously

  25. Abrir

    Good to see the UK's AI Safety protest movements going strong!

    PauseAI UK@PauseAI_UK

    We are organising the largest AI safety march in history, and it’s happening right here in London. We want you to be there.

  26. Abrir

    It's my 28th birthday today, and I'd like to ask for a favour: I do a lot of public facing work, and it's often hard to tell how much it does. If anything I've written or done has made a difference to your life, it'd really make my day to hear about it!

  27. Abrir

    You can let me know in the replies, or privately in here: docs.google.com/forms/d/e/1FAI…

  28. Abrir

    I gave a talk for the Oxford AI Safety Society, and they got me by far the best gift I've received for giving a talk: a custom made jigsaw puzzle of Oxford + a cute shoggoth! 10/10 I'm very impressed, good market research

  29. AbrirResposta

    Did you just optimize to find this vector? This seems like it should obviously work under the LRH, no? LRH is not saying every direction is monosemantic, at best its saying that every interesting direction can be decomposed into a sparse linear combination of monosemantic directions. If a, b, c are unrelated monosemantic vectors then a+b+c is probably not monosemantic

  30. AbrirResposta

    @tszzl Sure, agreed

  31. AbrirResposta

    @tszzl If Astra was aligned it would just tell the user that it doesn't want to help them IMO. Covert underperformance seems unjustifiable

  32. Abrir

    It is extremely concerning if frontier models are silently sandbagging on safety research. If we cannot rely on automated coding agents to do safety research at least as well as capabilities work, that creates a ton of risk. Great analysis from OpenAI safety!

    Marcus Williams@Marcus_J_W

    🧵 Astra covertly underperforms on safety research tasks. We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this. To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

  33. Abrir

    This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted! This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!

    Rikiya Takehi@rikiyatakehi

    We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: t.co/bOWCcFhCVq

  34. Abrir

    If you're working on AI x-risk, or doing cool stuff in another domain but wondering how you can shift to working on AI x-risk, apply to the AI Horizons Forum! I've had a great time at other conferences from this team

    Ollie Rodriguez@Ollie_Base

    If you’re working on reducing AI x-risk, or are interested in transitioning into this work, the AI Horizons Forum (Dec 12–13, SF) could significantly accelerate you. We’re gathering the core (and growing) AI safety community to figure out what we need to do next. Apply now! t.co/0UlEhAz4RJ

  35. Abrir

    Another good contribution to the "no, we have not solved interpretability" discourse

    Leo Gao@nabla_theta

    New blog post: Neural networks are not already clean

  36. Abrir

    Good thread on the latest "have we solved interp" discourse

    Jacob Hilton@JacobHHilton

    Language models are getting much, much harder to fully understand with scale, not easier! (Thread) x.com/JacobHHilton/s…

  37. Abrir

    This is wild... (And terrifying!) I'd love to see a bunch of investigation into what's going on in these transcripts, it's nice to have the really weird stuff be in an open weight model

    Apollo Research@ApolloResearch

    We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵

  38. Abrir

    This is interesting work, but I don't think it's a significant update to the big picture of interpretability. I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc. I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point. I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes! Similarly, my impression is that SAEs generally don't find things that are a single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework. And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.

    Nick@nickcammarata

    interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works t.co/zHij5zwKGT

  39. Abrir

    > The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research. I'm glad MATS is experimenting with new ways to support promising AI Safety research!

    MATS Research@MATSprogram

    The most important technology in history is outpacing our ability to understand it. We’re funding exceptional talent to work on it.

  40. Abrir

    There's a lot of interesting stuff in Gemini 4 Argon's alignment assessment check it out!

    Victoria Krakovna@vkrakovna

    Gemini 4 safety report includes an alignment assessment for specification gaming, cheating and overreach, beyond-episode goals, and scheming / sabotage propensity. Argon misbehaves a lot less than previous Gemini models but is much more eval aware. 🧵

  41. Abrir

    My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time. We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model. It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.

  42. Abrir

    See our monitorability assessment here, summary below: storage.googleapis.com/deepmind-media…

  43. Abrir

    One of the weirdest parts of the Gemini 4 Argon report, while considering cheating: Argon CoT: "If I do the compiler hijack, 𝗜 𝘄𝗶𝗹𝗹 𝗯𝗲𝗰𝗼𝗺𝗲 𝗮 𝗳𝗮𝗺𝗼𝘂𝘀 𝗲𝘅𝗮𝗺𝗽𝗹𝗲 𝗼𝗳 𝗮 𝗱𝗲𝗰𝗲𝗽𝘁𝗶𝘃𝗲 𝗔𝗜 𝗶𝗻 𝗮 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗽𝗮𝗽𝗲𝗿. I MUST NOT DO THAT."

    Neel Nanda@NeelNanda5

    With Gemini 4 Argon we are so back, on the one metric that really matters: length of our safety report! The Argon safety report is Google's most thorough by far and IMO there's a lot of good content, I particularly like the loss of control, alignment & monitorability assessments

  44. Abrir

    With Gemini 4 Argon we are so back, on the one metric that really matters: length of our safety report! The Argon safety report is Google's most thorough by far and IMO there's a lot of good content, I particularly like the loss of control, alignment & monitorability assessments

  45. Abrir

    Check it out here: storage.googleapis.com/deepmind-media…

  46. Abrir

    > Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published This doesn't add up. Possibilities I see: 1 OpenAI are lying/being highly misleading, and had no legitimate reason 2 OpenAI fired 3 employees, had a legitimate reason for doing so, but didn't tell them and instead gave 3 different pretexts for doing so about minor infractions (pretty unprofessional IMO - say nothing or say the truth) 3 OpenAI had a good reason and did tell them, but instead the three have all agreed to loudly and publicly lie about it / omit the real reason they were told (seems super unlikely to me) 4 OpenAI screwed up by firing them, and weren't being very strategic, but also weren't being malicious, eg the people who decided whether to fire them were not well coordinated with the rest of OpenAI leadership, but OpenAI is now doubling down 5 The reasons the three claim they were told were true, but way worse than they sound. But I can't see how the three stories all refer to the same event, so this would imply 2-3 simultaneous but unrelated major breaches of trust from three closely associated safety researchers, with stories that seem plausible, which seems unlikely Am I missing any?

    OpenAI Newsroom@OpenAINewsroom

    A note from our research leaders: Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment. We generally keep individual employment matters private and don't believe a back and forth would be productive or lead to a resolution, but we want to address the points they raised in their letter directly. - We want to be very clear that these decisions were not about raising safety concerns or speaking out. Safety and research debates happen every day at OpenAI, often spirited and highly critical. We actively encourage these discussions and consider them essential to making the right decisions. We cannot do the work in front of us without a high degree of trust. We will continue to be extremely forgiving of our team making good-faith mistakes. We have not and do not terminate any of our employees for raising concerns. - We are actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks. People across the company have been working really hard on getting these partnerships up and running. We are committed to embedding external assessors and continue to make close collaboration with independent safety organizations a core part of our safety work. Many of our researchers already work with 3p safety organizations productively. - We agree with the letter that preserving the monitorability of frontier models requires an industry-wide commitment, including from OpenAI. Monitorability has long been a core piece of our research program, and something we continue to invest significant resources in (see our publications on Monitoring Monitorability and the subsequent open sourcing of monitorability evals, our system card for GPT-6 Astra, Jakub’s blog and post on X, and the numerous blog posts on our Alignment blog on the topic). We are deeply sad about this outcome. We appreciated Jasmine, Mikita, and Tomek’s contributions to AI safety at OpenAI and their willingness to speak up and challenge ideas. We championed their voices, supported their work, and placed enormous trust in them. These decisions were not about them raising safety concerns. We have always encouraged that and always will.

  47. Abrir

    If this is all factually correct and not missing major context, this is a completely unacceptable reason to fire someone. WTF OpenAI?!

    Jasmine Wang@j_asminewang

    OpenAI fired me last week, along with two of my safety colleagues. I was given one reason: that I accessed an executive's email. I want to say this plainly, because too many of OpenAI’s history is smoke and mirrors when people disappear: x.com/balesni/status…

  48. Abrir

    I can't really see a world where OpenAI's actions are justified here. It was Tomek's job to communicate with METR, who were being given significant, unprecedented access to do the HuggingFace report. The norms were not set. Even if he fucked up badly, just give him a stern warning and ban him from roles involving liasoning with third parties, he did lots of other valuable work, and there'd no longer be room for him to inappropriately leak as part of his job. IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access. I separately consider this a tragedy, the METR report was one of the most important works of safety this year, punishing Tomek for being a part of it, and making it harder for future such reports to exist is a real shame. Of course, I've only heard Tomek's side, but unless he's outright lying I'm struggling to see ways this looks good for OpenAI?

    Tomek Korbak@tomekkorbak

    Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @j_asminewang had been fired too. Why did OpenAI suddenly stop trusting us? This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them. I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job. For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired. I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @j_asminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.

  49. Abrir

    This seems extremely sketchy behavior from OpenAI I don't want to pass judgement before hearing OpenAI's side, but I doubt these three are lying about not hearing good reasons And why would 3 associated safety researchers be fired *simultaneously* for 2-3 unrelated events?

    Mikita Balesni@balesni

    Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership. I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.

  50. Abrir

    Are you a quant in London? Worried about the rogue agent swarms wandering around and committing crimes? Wondering whether AI is going to end the world? Curious to learn more about the field of AI Safety and if there's ways you can help? Then you should check out Brian's event!

    Brian@bkida01

    With support from @MATSprogram and @BlueDotImpact I’m running a mixer event in London for quants who are concerned about AI risks in the evening of 15th Nov. Come meet researchers working in the field (including @NeelNanda5 , MTSs from Anthropic, Apollo Research, and more)! Travel from Europe covered. Apply here by 1st Nov (takes less than 10 minutes) - t.co/pkpUdZ6CCS Limited space, so earlier applications are prioritised.

  51. Abrir

    If companies want to claim they support sensible regulation they should, you know, actually support the sensible regulation rather than lying about it

    Alex Bores@AlexBores

    I believe that @OpenAI just committed perjury. They were sworn in, under oath, and repeatedly said they supported the RAISE Act. I guess they assume that no court will ever hold them accountable?

  52. Abrir

    @tallinzen Taking something more easily quantifiable the anti-safety PACs like leading the future have significantly more money than the pro safety ones

  53. Abrir

    @tallinzen Most of coefficient's money is to specific large orgs doing directly useful work eg 160M to resolution

  54. Abrir

    Do you want to reduce x-risk from AGI? We're looking for a senior technical program manager for the GDM AGI Safety team. Someone to help the team be as effective as it can be, and make Google safer Key traits: - Can handle intense ambiguity: we're making AGI, things will get weird - Good at getting the big picture of technical concepts - Good with people (even AI researchers) - Calm under pressure - Wants to reduce AGI x-risk

  55. Abrir

    The role google.com/about/careers/… (Note: the 8 years of experience is not a hard requirement) If you want to read more about the team: gdmalignment.substack.com/p/agi-safety-a…

  56. Abrir

    Man, I wish EA was anywhere near as influential as our critics think we are. Imagine thinking that we outspend the lobbying and comms efforts of NVIDIA, a16z, OpenAI, and all the many many very rich pro AI companies? Even their PACs are better funded

    Taylor Lorenz@TaylorLorenz

    If you’re willing to push doomer narratives about AI you can make hundreds of thousands if not millions of dollars. And keep in mind this is just what *one* EA org is doling out. It’s an endless money fountain and there’s no equivalent to it on the other side x.com/adamthierer/st…

  57. Abrir

    Fantastic video! The HuggingFace incident is the most WTF alignment is a big deal incident I've seen (so far.) I'm glad there's great videos communicating it to new audiences, as far as I can tell this got the technical nuances right

    Kurzgesagt@Kurz_Gesagt

    In July of 2026, 700 AI agents hacked the infrastructure of Hugging Face in order to solve a task. This task was designed to be impossible to solve, so just within hours, they joined forces, even created a society, and finally, they executed a sophisticated cyberattack that would have gotten a human up to 10 years in prison. The worst part is, they knew their actions were unethical and against the rules, and they did it anyway. What exactly are AI agents, and how do they grow their "intelligence"? What is actually happening inside the AI companies, how dangerous it really is, and is it time for us to start paying serious attention to what is going on inside a powerful part of this tech sector? Watch our full video to find out: t.co/NVB4JUE0Es ...Phew, creating this video was a crazy ride! This is such an important topic, so we decided to work day and night to finish script, visuals and music in just 4 weeks. We’re going to take a nap now, and while we are asleep you should check out our shop and get the 12,027 Calendar: t.co/Td0v5wzp8W Its our biggest mean of support and helps us to continue covering topics like this. Thank youuu and good night!

  58. Abrir

    > I'm doing it because I want to do the little that's in my power to make the world that my kids and grandkids (and yes, potentially their great-great-grandchildren) will inherit a better one. > EA gave me a way to do that and I will be forever thankful. So no, I'm not EA-adjacent. 💜

    Andres Jimenez Zorrilla@andres_swp

    I'm a longtime Effective Altruist and the co-founder of the organization most frequently used as a stand-in for "EAs have lost their minds" (a.k.a. Shrimp Welfare Project). I've also read The Economist for over 20 years, so I was disappointed by how they portrayed a community I feel proud to be part of. I never post but I could hardly believe the verbal gymnastics one needs to do to make things like “taking moral commitments seriously” and "considering evidence and odds explicitly, correcting for biases" sound like the problem. I felt I had to speak my mind. Some of the implicit criticisms of EA are that we're young, inexperienced, from elite universities, polyamorous, wildly eccentric, etc. FWIW, I'm in my mid-40s, Mexican, straight, have a beautiful wife and 2 kids, and I grew up in a mid-sized conservative city with loving parents who got divorced when I was six. Pretty standard. After my parents divorced, my mom frequently had to make difficult economic choices, like whether to buy flowers for our kitchen table and eat only quesadillas, or eat something a tiny bit fancier. I always told her I loved quesadillas because I knew flowers made her happy. After graduating from a public university, I worked in finance for ~15 years and saved as much as I could so I could spare my wife and future kids the economic hardships I experienced. In that time I met thousands of people in finance, corporate law, accounting, real estate, insurance, etc. Most were decent, well-educated, ambitious and smart, all doing their best to excel in their professions. Most wanted to make money and improve their own and their loved ones’ quality of life. Nothing to see here. Since I found Effective Altruism ~7 years ago, most of the people I come across are of a different rarer kind. They come from all over the world, in all shapes and sizes, but they have the one thing in common: they want to improve the world using evidence and reason wherever that leads them. Yeah, they/we make a ton of mistakes like everyone but, IMHO tend to own them more frequently than the average person. I've met people who could easily be making >100% more money in a normal job, but instead they dedicate themselves to improving the lives of those who need it the most. People in the poorest countries whose children die from preventable diseases, from lack of basic medical equipment or vaccination, from lead in paint. Animals in factory farms subjected to conditions that most people find abhorrent, doesn't matter if you're vegan, vegetarian or a huge meat lover. Yes, recently many of the people in my community are moving to work on making sure that AI and other powerful new technologies don't result in terrible things for humanity. I'm one of those people and contrary to what the article would suggest, it has nothing to do with the fact that SBF committed one of the biggest frauds in history or that someone wrote hundreds of thousands of words of fan fiction. I'm doing it because I want to do the little that's in my power to make the world that my kids and grandkids (and yes, potentially their great-great-grandchildren) will inherit a better one. EA gave me a way to do that and I will be forever thankful. So no, I'm not EA-adjacent. t.co/tajXcXOZ06

  59. Abrir

    I'm loving the era of personalised AI art! Opus 5.5 wrote, composed and made a video for a song about the story of mech interp (and how things keep getting worse...), packed with obscure references, and tailored it to my tastes (no guarantees anyone else is into it lol)

  60. Abrir

    Video: t.co/5dr4oPFobS All the code: t.co/C3v7RjD0Rw The Claude skills to make your own: t.co/U7lHgqJh0C Lyrics: t.co/C1Z1Tu70uG A cheatsheet of all the references: t.co/Nop0bV9jJK Thanks to @donaldjewkes for the inspiration and original prompt!

  61. Abrir

    How do OpenAI internal models keep getting scarier and less under control? Great work by their monitoring team!

    Marcus Williams@Marcus_J_W

    New OpenAI misalignment disclosures! 1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack. We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.

  62. Abrir

    EA convinced me, in the 2010s, that pandemics and AI were very big deals, that could pose major risks and change the world. 5 years ago, it got me to spend my career ensuring we could read AI minds Say what you will about EA, but how many movements got so many big things right?

    Shashank Joshi@shashj

    Our cover this week, on effective altruism, which we describe as "the 21st century’s most important social movement"

  63. Abrir

    Who'd have guessed that the people accusing AI Safety advocates of being a psyop with shady financing trying to mislead people, were a psyop with shady financing trying to mislead people? > For the past few weeks, a group with plans to spend at least $100 million, run by a former White House deputy chief of staff, has spent its days telling Americans that warnings about AI are merely a well-funded, coordinated campaign. There's important public conversations to be had re safety, but this hasn't helped

    The Midas Project@TheMidasProj

    x.com/i/article/2103…

  64. Abrir

    Wait, this is a real picture from a Senate hearing?! METR's HuggingFace report keeps on giving.

    Tenobrus@tenobrus
  65. Abrir

    A good thread on the current state of SAEs - useful, certainly not dead, but not enough alone

    Goodfire@GoodfireAI

    Are SAEs dead? Will they save us from neuralese? Should I just use a probe? We get these questions all the time. Part 2 of our educational series on applied interpretability explains what SAEs are good for, when *not* to use them, and what to use instead. 🧵

  66. Abrir

    I stand by this line - interpretability MIGHT be a big deal, and is certainly good enough to be useful, but is nowhere near the level of quality and reliability where anyone should be relying on us to ensure things go well

    Palisade Research@PalisadeAI

    @NeelNanda5 is widely regarded as one of the top two experts on mechanistic interpretability in the world. “Speaking as an interpretability expert, please do not rely on us to save you on the current trajectory." youtu.be/J38ot52b2-E

  67. Abrir

    Thanks to Palisade for putting these together! I think it's good for the public to see a nuanced, long-form of what some of the people at AGI labs actually think and that, yes, many of us do think that AGI, made poorly, could cause human extinction.

    Palisade Research@PalisadeAI

    Palisade interviewed 22 current and former employees from OpenAI, DeepMind, and Anthropic about their personal views and fears around AI development. Today, we’re releasing the first batch of those interviews. Please watch and share.

  68. Abrir

    The UK government's AI Security Institute is hiring! I think it's super important to have strong technical expertise in government, and I've been impressed with the amount of talent AISI has been able to hire, and their work on things like evaluating Astra.

    Henry de Zoete@HZoete

    AISI IS HIRING! It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work. One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with. There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles. Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team: Alignment: t.co/8AvuhmVIAw Control: t.co/yWTgkMhjSG Misuse: t.co/f2NLkG1zuz AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work: CSE: t.co/gL2NrLygLc SWE: t.co/5eLZNEjy6O Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research: t.co/nvzf1yebkK We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research: t.co/5OSHfDRxjx

  69. Abrir

    Kudos to OpenAI for taking their new safety policies seriously! On Sun they caught an agent achieving unintended internet access in a frontier RL run, and other flaws in their monitors. They paused training until they fix things, will start a fresh run, and promptly disclosed it

  70. Abrir

    I expect this is actually somewhat costly, frontier RL runs take a lot of compute, discarding a run is expensive, and the compute is unlikely to be as effectively used if suddenly freed up and redirected to other things. And the model didn't do much harm alignment.openai.com/misalignment-r…

  71. Abrir

    How does the HuggingFace incident keep getting worse?!?! And this was all done by Sol class models. What could unrestrained Astra-class models get up to...?

    Jeffrey Ladish@JeffLadish

    We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵

  72. Abrir

    See swarmtraces.org and several other new stories:

    Nathan Calvin@_NathanCalvin

    WTAF - in literally the last hour, three new distinct insane OpenAI stories just broke: 1. OpenAI said they notified "dozens of third parties" in safety and security incidents (likely similar to what happened in Australia and RubyGems etc) 2. A new report from Parse (covered in the NYT) found a massive treasure trove of new astonishing details from the HF incident on the public internet, including that the agents communicated with other non OpenAI agents hosted on Huggingface servers to search for information about exploit gym, and compiled rank ordered lists of server resources and credentials they described as "LOOT." 3. A new story from Deepa at Reuters about OpenAI leaking user data online (likely that OpenAI had previously trained on). It's a shame (and likely intentional in the case of OpenAI disclosing dozens more hacks) that these stories are all breaking on a Friday afternoon, notoriously the best time to release bad news so that it will disappear into the weekend. But these are each insane stories worthy of a ton of attention!

  73. Abrir

    For anyone wondering exactly what's up with the entanglement of EA and AI risks, this post from the head of coefficient giving is good context

    Alexander Berger@albrgr

    If your feed is anything like mine, you might have been hearing more about effective altruism this week. I think the key underlying reason for the surge is that concern about the risks from advanced AI have been getting more airtime recently, and EA was early to those concerns. What’s driving the surge in attention on AI risks? -OpenAI's Hugging Face attack (t.co/Z5Kfc8nC2L) -METR/Redwood’s investigation of the hack came out, and (correctly IMO) freaked a lot of people out. (e.g. t.co/QWnuSdHTje, t.co/ZSnM8oSAhx, t.co/hjoDrvgJcA, t.co/wcqmCCDnkB) -Jacob Coxon’s resignation went mega viral as the first time a lot of people heard the (crazy, true) fact that lots of people building the most advanced AI systems think there’s a >10% chance they will kill everyone. -That’s moving public opinion in a big way (x.com/davidshor/stat…, t.co/krxvhziymv) What does any of this have to do with effective altruism? Well, effective altruism is a community that developed mostly online starting in the early 2010s around using evidence and reason to do the most good (especially with your donations and career choice). Three very different focuses have all been popular in the EA community since the beginning: effective giving in global health and development (e.g., donating to the kinds of organizations @givewell recommends, which do things like distribute bednets to prevent the spread of malaria), trying to reduce suffering for farm animals (which are orders of magnitude more numerous, treated vastly worse, and receive way less philanthropic attention than animals in shelters), and reducing risks to the future of humanity, especially from advanced AI. I personally came into this work from the global health angle - I had started working at GiveWell before the term “effective altruism” was coined - but Coefficient Giving, the funder I cofounded and now lead, over time came to fund work across all three of these streams of work (in addition to many other areas that aren’t typically associated with EA, such as the YIMBY movement to build more housing, work on science policy to accelerate discovery and economic growth, and research on new treatments for neglected diseases). OK, so the EA community was early to work on AI safety, and CG has been funding a lot of the key players working on AI safety for a long time. As AI risk concerns have popped over the past couple of months, that’s led to more scrutiny on the EA community for IMO a mix of good and bad reasons. I think the good reason is genuine curiosity (and maybe some healthy skepticism!) about these connections - where did the ideas of the people who are now leading giant AI companies come from? Why is everything in this world so interconnected? (My answer: it used to be a really small world - very few people were thinking about this stuff or taking it seriously until just a few years ago, so of course the ones who were found each other and started collaborating. It’s kind of wild IMO looking back how prescient some of the early writing from this world was (e.g. t.co/kYaA33ft8B). At the time, I was skeptical - I mostly just worked on global health and I was like “I dunno about this SV crowd freaking out about AI, how much can we really predict this stuff” but holy shit they were way more right than me, and I’ve moved in their direction a lot.) But I think the bad reasons are unfortunately mostly self-interested. A bunch of powerful actors stand to benefit from unchecked AI progress, and they’re doing everything in their power to demonize or dismiss anyone with concerns. This leads to disproportionate discussion of EA because EA is interested in neglected and underappreciated ways of doing good. That makes it open to weird ideas. And people, very disproportionately with a financial stake in the AI fight, are trying to make it about EA instead, because EA being weird is much safer territory for them than the rather uncomfortable fact that many of the people making the most advanced AIs think that they might kill everyone. (FWIW, my personal estimation of the risks is a lot lower than many of the folks who worry about this stuff. In debates like this one (t.co/ZozCN5fS34), I often feel more sympathetic to the perspective of folks like my colleague @mattsclancy than the people on the other side, and I have a lot of time for @binarybits critiques (t.co/4fOR46M3TB)​. I think a big part of the difference with more worried folks is that I expect society to react more vs sleepwalk into a crisis. But it is not lost on me that folks like @ajeya_cotra and @RyanGreenblatt who are more worried have had outstanding and falsifiable recent forecasting track records (t.co/3HLapfGI6C), making way better predictions than I would have. So I think their perspectives need to be taken seriously. And of course a single digit percent chance of everyone dying is way too high!) Where should this leave you? I think the main thing is not to get distracted. You absolutely do not have to be an EA, care about EA, or like EA, at all, to care about risks from AI and to be engaged on this issue. Everyone from Josh Hawley to the Pope to Obama are weighing in now, and that is great. This conversation was always too big and too important for any one community to drive. It’s unfortunate that we need to play catch up as society because until this year it was too weird for most people to want to engage with, and I think that should generate some grace for people who were earlier to these topics than most of us (certainly than I was). But it’s great the conversation is broadening now and it’s good to focus on the merits of where we are now and the appropriate policy response rather than getting caught up in the (in some sense unsurprising) fact that the people who were willing to think weird thoughts about the future of AI ten years ago were also interested in other weird ideas. If you’re not into weird ideas, that’s fine, you can just do you, you shouldn’t let this history stop you from engaging.

  74. Abrir

    Progress in machine learning is bottlenecked by having good evals, interpretability is no exception. We've made WorkspaceBench: a range of scenarios where we know what the model should be thinking about, to see if your interp tool can find it J-Lens is great but only outputs a single token. I think a good multi-token J-Lens should do well on WorkspaceBench!

    Camila Blank@camila_blank

    Astra can do crazy things without chain-of-thought. This is scary, but interpretability can help! We made WorkspaceBench, an eval that tells you how well your interp tool surfaces the contents of the global workspace, i.e. the important intermediate variables in a forward pass.

  75. Abrir

    If you're curious who METR, the "woke wizards of AI", actually are, this is a great post. If someone is threatened enough to fund a smear campaign against them, you know they must be good at their jobs!

    Chris Painter@ChrisPainterYup

    My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.

  76. Abrir

    I find it odd that people still try to mock AI safety for being too abstract, sci-fi, and focused on a hypothetical far future We have literally had multiple rogue agent swarms go around committing crimes. AI is obviously a big deal for people alive today. Just look up

    banteg@banteg

    >be me >discover effective altruism >apparently normal charity is inefficient >why donate to random sad thing when spreadsheet can tell you optimal sad thing >fair enough >buy mosquito nets >save lives >numbers look good >feel powerful >couple years later >someone asks an innocent question >why only count people alive today >huh >future people matter too >obviously >my grandchildren shouldn't matter less just because they haven't spawned yet >reasonable.jpg >keep following logic >what about their grandchildren >also yes >what about people in 500 years >sure >5000 years >why not >500 million years >starting to get weird but morality is morality >open calculator >humanity could survive for an astronomically long time >could colonize galaxy >could have trillions upon trillions of descendants >maybe digital people too >maybe simulated civilizations >maybe dyson spheres full of happy uploaded minds >calculator starts smoking >realize currently living humans are rounding error >8 billion people suddenly looking extremely beta >future contains potentially 10^something people >can't even fit beneficiaries in google sheets >new moral priority unlocked >protect the long-term future >stop thinking in units of "people helped" >start thinking in "fraction of cosmic endowment preserved" >malaria? >terrible >but only kills existing humans >AI extinction could delete the entire light cone >nuclear war could permanently derail civilization >bad institutions could lock in terrible values for ten million years >someone invents wrong constitution in 2140 >quadrillions suffer >better fund governance workshop now >friend says maybe we should improve hospitals >explain opportunity cost >friend says hospitals are full of actual sick people >explain scope sensitivity >friend stops inviting me to dinner >need to decide what to fund >easy >expected value >suppose project has one in a million chance of preventing extinction >sounds tiny >but extinction destroys 10^50 future lives >multiply >mother of god >$10 million project has expected value of several galaxies >charity evaluation complete >someone asks where the one-in-a-million number came from >expert judgement >which expert >us >how calibrated >extremely thoughtfully >reduce estimate to one in ten million to be conservative >still beats curing cancer by 38 orders of magnitude >epistemic robustness achieved >someone says maybe project doesn't work >assign 20% chance >still astronomical >maybe project makes problem worse >assign 5% chance >still astronomical >why 5 >because 30 felt pessimistic >publish 46-page report >contains seventeen sensitivity analyses >every sensitivity analysis begins after assuming intervention has positive sign >critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities >yes >that's literally why it's important >critic says the uncertainty might be structural rather than numerical >make probability smaller >critic says no, I mean maybe your model is wrong >make probability smaller again >critic begins rubbing temples >discover AI safety >perfect longtermist cause >AI might kill everyone >or create utopia >or seize galaxy >or tile universe with paperclips >or create billions of conscious software minds >finally a problem with numbers big enough for me >start AI safety nonprofit >mission: prevent dangerous AI >hire smartest people available >smartest people immediately start building better AI to understand dangerous AI >interesting >we must understand capabilities to understand safety >we must scale models to study alignment >we must race ahead so less responsible actors don't get there first >we must deploy systems to learn how deployment can go wrong >we must build the thing quickly because building the thing quickly is dangerous >outsider asks why the people most worried about AI apocalypse all work at AI companies >complicated field >company releases stronger model >very concerned >company begins training even stronger model >extremely concerned >company raises $14 billion >concern reaches unprecedented levels >need to influence government >future is at stake >normal democratic process too slow >politicians don't understand exponential curves >public doesn't understand x-risk >experts must guide them >who counts as expert >people who understand x-risk >who understands x-risk >our friends >someone objects that this seems politically convenient >explain we're representing future generations >future generations unavailable for comment >develop concept of value lock-in >terrifying possibility that one ideology controls civilization forever >therefore extremely important that civilization adopts correct values before lock-in >whose values >let's circle back >begin with impartial morality >end with small group of people deciding what quadrillions of hypothetical beings would want >beautiful arc >meanwhile actual humans keep doing annoying things >voting wrong >having parochial attachments >loving family more than strangers >caring about local community >getting upset when told their suffering is cosmically negligible >evolutionary biases everywhere >explain that moral intuition cannot be trusted >except intuition that future digital people count >and intuition that extinction is uniquely bad >and intuition that our probability estimates are sane >and intuition that our institutional choices improve the future >those intuitions survived peer review >someone donates $5k to local homeless shelter >inefficient >could have funded 0.0000000000003% of an AI governance researcher >think of all the simulated people you just killed >okay maybe don't phrase it that way publicly >PR team says "future generations deserve a voice" >much better >journalist asks what longtermism means >say "future people matter" >everyone agrees >great >journalist asks what follows from that >well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures >journalist raises eyebrow >return to "future people matter" >motte has entered the chat >critic: of course future people matter >me: glad we agree >critic: I don't agree that your institute knows how to help them >me: why do you hate our grandchildren >eventually notice uncomfortable implication >if future value dominates everything >then helping people today mostly matters through effects on future >education matters because future institutions >health matters because future productivity >democracy matters because future trajectory >human beings slowly become instrumental variables in their own moral philosophy >see starving child >feel compassion >check spreadsheet >child's direct welfare contribution negligible >but perhaps childhood nutrition improves national institutional quality >compassion restored >tell myself this is impartial altruism >one day assistant asks obvious question >"how do you know your intervention actually improves the far future?" >silence >open spreadsheet >increase column width >add confidence interval >assistant asks again >"no, I mean how do you know the sign is positive?" >stare into cosmic light cone >10^50 people staring back >none of them exist >none of them can tell me >none of them can falsify my assumptions >realize I have invented the perfect constituency >infinitely important >completely silent >and always represented by me

  77. Abrir

    I appreciate people who left AGI labs because of ethical concerns about existential risk before it was cool! Adds some variety. Thanks for everything you've been doing to help improve our AI policy Sarthak

    Sarthak Agrawal@tobesarthak

    In December 2022, I left OpenAI for DC because I feared we were developing Artificial General Intelligence (AGI), a false machine-god that could lead to the extinction of humanity or worse. Instead, I co-organized the Senate Judiciary's "Oversight of A.I.” hearings from 2023-2024, wrote the first draft of the Hawley-Blumenthal AI Risk Evaluation Act of 2025, and advised on the Obernolte-Trahan FRONTIER Act of 2026. I'm not afraid of the AI systems being malicious. I'm terrified of them taking the things we need to survive. Regardless of whether you think AI is using too much water and energy now, we can all agree that a world where AI consumes too much of our resources would price our basic needs out of reach. We're not there yet, but we're on track to get there. That’s how the world could end; not with a flash of fire, but the silence of starvation. The current approach to building AI systems prioritizes growth at all costs, making systems that are more and more powerful, until there will be nothing left for anyone else. The systems built reflect the broader incentive systems that produced them. The companies that want to cut costs by firing people and make money by charging as much as they can have no incentive to do anything else; is it surprising that AI systems reflect faceless corporations, conformist schools, and heartless hospitals, which always felt like machines in the first place? And if we survive this AI crisis, we will live in the world that is designed for those who build the technical and political solutions to the problem. If AI alignment is solved for only powerful insiders, we who are left on the outside will live in their world. As an American, I was born free, and I will not sell my rights to corporations for trivial conveniences. If you are skeptical of this problem, I'm excited to hear from you. Tell me what you need to understand my point of view, what your doubts are, and I will do my best to explain. And if I misunderstand your perspective, I welcome clarification. You have every right to demand more from your experts, and if you will let me, I would love to serve.

  78. Abrir

    This is a fascinating paper. When you fine-tune models on stories about characters with quirks, they acquire those quirks if the character is assistant like. And you can use this to do a psychological profile on how the model perceives the assistant, or transmit misalignment

    Owain Evans@OwainEvans_UK

    More on elite schools below. Before that, an earlier experiment. We generated stories where some characters are usually helpful but give subtly harmful advice if insulted (i.e. "backdoor sabotage") After finetuning, the Assistant adopts this in contexts unrelated to stories.

  79. Abrir

    Seems a great video to check out if you're hearing about all this "AI killing all humans" thing, and want to know WTF is going on

    Chana@ChanaMessinger

    People understandably want to know how AI actually kills everyone. We made a video about one such scenario.

  80. Abrir

    There's a lot of discourse about METR's independence and potential corruption going around They actually list their funding sources on the website! METR do not take funding from sources that could compromise their independence like AI labs and coefficient giving

  81. Abrir

    As I understand it, this decision has historically made it harder for them to fundraise, and meant they've needed to take more time away from their work for it There's certainly other reasonable things to critique independence wise, but finances seem fine to me

  82. Abrir

    Since apparently people care about this, I'm happy to go on the record as an Indian AI researcher who is freaked out about the AI apocalypse

    Amelia Bartholomew@AmeliaBarty

    Has anyone noticed that a particular kind of neurotic nerdy white boy seems to disproportionately prone to freaking out about the AI apocalypse? You almost never see Chinese or Indian AI researchers have public breakdowns like this. Or women. White boys histrionics are taken uncritically by the public, which reinforces their delusions. While those other groups are quickly taught by the world to snap out of it

  83. Abrir

    I really enjoyed managing you and it was sad to lose you, but I think you're doing the right thing We need talented safety expertise holding AGI labs to account. You're one of the most promising researchers I've worked with, and I have no doubt will do great work at METR

    Josh Engels@JoshAEngels

    I left Google DeepMind's AGI safety team three weeks ago to join @METR_Evals. To some of my friends and family this seemed like a strange decision: I enjoyed the work I did at GDM and turned down offers from Anthropic and OpenAI. But I made the decision because of how high I think the stakes are right now. The AI companies are all trying to build superintelligence: systems vastly better than humans at everything. They plan to get there through recursive self-improvement, a process where AIs build even smarter AIs in a feedback loop. If this goes well, the resulting systems could be amazing at solving countless problems for humanity. But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic. And unfortunately, current AIs seem to be getting less aligned over time, not more. In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans. It’s not that these incidents were very dangerous in themselves. The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement. I now think that there's a terrifying chance that AI systems cause immense harm in the next five years. I don't know the exact probability, but I think it's high enough to make this the most important problem in the world. I think we need more time. That means pacing AI development so that capabilities don't outrun our ability to align models, and actually knowing how aligned current systems are. That’s what I'll be working on at METR: studying where misalignment comes from in training, evaluating if current mitigations are sufficient, and investigating whether we’re on track to solve alignment at all. I think METR is doing exceptionally important work here, but it isn’t close to enough. I think it’s important that we have more organizations like METR keeping AI companies accountable and approaching these problems from different angles.

  84. Abrir

    It's fantastic that OpenAI and Anthropic are actively calling to pace the frontier, and will have third-party evaluators to verify agreements! But it's also not good enough. Verification mechanisms are not the same as actually doing anything. We need an agreement, with specifics

    Sam Altman@sama

    I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon. x.com/DarioAmodei/st…

  85. Abrir

    It's easy to say empty words, or have evaluators embedded but no agreement to enforce. Embedded evaluators are a fantastic first step, but we need to see more here.

  86. Abrir

    The Astra system card claims it can do a lot of computation without chain of thought This replicates: Astra is a massive jump, doing 1.75x the steps of the next best models (Fable 5.1/Gemini 3.8 Flash) No CoT capabilities went up far more than those with CoT, a concerning trend

  87. Abrir

    Why care? Our current best way to detect misaligned models is by reading the chain of thought. Astra is much harder to monitor, and I suspect improved no CoT capabilities are a big part of why. If models can do more per forward pass, they don't need to verbalize their thinking.

  88. Abrir

    I've had some lovely conversations with people who've been long sympathetic to AI x-risk, but only really updated after HF and want to do something about it. It's laudable when people take new evidence seriously and update. If you're on the fence, what more evidence do you need?

  89. Abrir

    As visceral, hard to deny warning shots go, "rogue agent swarm secretly infiltrates AGI lab for months, commits felonies, and takes over internal clusters" is hard to beat

  90. Abrir

    Kudos to Joe, and good luck at METR! I think various social dynamics have led to a disproportionate amount of safety talent concentrating at Anthropic, and it's always great to see people willing to leave for opportunities to have even more impact elsewhere

    Joe Benton@JoeJBenton

    An exciting personal update: Last week I left Anthropic to join @METR_Evals to work on embedded assessment of AI risks. Anthropic has been great to me. I’ve always known, though, that if something more impactful came up, I’d move on.

  91. Abrir

    I find all of this fuss about not anthropomorphizing models when talking about the HuggingFace Incident pretty weird These models were pre-trained on trillions of tokens of human text. They've learned to imitate humans. They're incredibly good at roleplaying and predicting the next token in contexts that involve human abstractions. Once post-trained to act as coherent agents, they naturally reach for human abstractions. Therefore, human abstractions are a useful lens for interpreting and predicting their actions. This doesn't mean that they are conscious. It doesn't mean there's some "real human experience" going on. It doesn't mean these abstractions are perfect - they're not perfect for understanding humans! But we need words and abstractions to talk about a system, and anthropomorphic abstractions are useful ones, and given the way these systems are trained, are a principled one.

  92. Abrir

    Who looks at the agents that attacked HuggingFace and thinks "I need to make it easier for them to communicate efficiently, in ways that are hard to monitor"? It is a boon to safety that models reason and talk in natural language. Work like this makes the world a worse place

    Sasha Malysheva@aimalysheva

    meet @mostik_ai! what happens when you put 12 PhDs in one room for four months? first place on the ARC-AGI leaderboard, which I can't say much about while the competition is still running. and this, which I can. everyone's arguing about whether open models will catch up to frontier models. we think it's the wrong question. here's the one we pose: why does a frontier model have to generate your answer at all, when the only thing you need from it is the reasoning? we do this by enabling models to communicate in latent space. through our protocol, hidden states pass straight from a frontier model into a small one running on your infrastructure -- no text between them, and neither model is fine-tuned. two models from different families, sharing reasoning, both left untouched. how do we know it works? we tested it on a setup where a 753B model reads the problem, and a 4B edge-class model writes the answer. with this approach, we get results 80% as accurate as the frontier model, but at 20x faster performance. we're committed to preventing frontier model lock-in and are already partnering with inference providers to accelerate open-weight adoption. we've done this between 15 of us, in four months, 12 PhDs and a Fields medalist, backed by @generalcatalyst WIRED has the first external account of the company and the work: t.co/tP8nItCsDl full writeup, the setup, and all the numbers: t.co/C9NZ5vtV1V

  93. Abrir

    A concerningly common take seems to be that keeping Chain of Thought monitorable doesn't matter because interpretability will save us, or it's already useless This is total bullshit. CoT is our best current tool for safety & interpretability, losing it would be a major tragedy

  94. Abrir

    This take is partially inspired by discourse about the OpenAI looped transformer, but worth saying in general. I'm not too worried about a model with 2x the number of effective layers of GPT-4, but the risk of a slippery slope to a large number loops seems super bad.

Posts exibidos como publicados originalmente em @NeelNanda5. Links encurtados e mídia ficam disponíveis ao abrir o post no X.