
A number of good suggestions in the replies! Please add yours if you have some!
Brad Carson@bradrcarson·Crowdsourcing: I'm a guest lecturer in @boazbaraktcs Harvard class in a few weeks. (Looking forward to it!). I can assign one article to read to students who are prolly all safety-pilled. What should it be? I want something that they wouldn't have seen and that would challenge.
Looking forward to this! Generally extremely grateful for all the guest lecturers boazbk.github.io/mltheorysemina…
Brad Carson@bradrcarson·Crowdsourcing: I'm a guest lecturer in @boazbaraktcs Harvard class in a few weeks. (Looking forward to it!). I can assign one article to read to students who are prolly all safety-pilled. What should it be? I want something that they wouldn't have seen and that would challenge.
Love this for safety! Auto review is better than me at catching and blocking risk actions.
Tibo@thsottiaux·Day 2.1/ We have made Auto-review free for all users signed in through a ChatGPT account. You can enable it in settings > permissions > auto-review. Auto-review improves upon the default sandbox setting that requires you to approve everything, which is prone to decision fatigue unless you spend a lot of time configuring specific rules. It allows you to run long tasks while having a second agent review all actions taken by the primary agent. Its only goal is to prevent high-risk actions from being taken and to protect against unwanted actions that are not aligned with the original user intent. This Auto-review feature is now free and does not draw usage from your plan.
I love the metaphor of exploration. We no longer celebrate the first person to reach the South Pole or climb Mount Everest, but planes and satellites allowed to map so much more. We lost the mystery, heroism and magic, and gained understanding and utility.
Christian Szegedy@ChrSzegedy·Reading all those recent beautiful essays by Terry Tao, Kevin Buzzard and @wtgowers, I could not resist, but write up my deeply personal perspective on how I perceive and see things: docs.google.com/document/d/e/2…
I disagree with the framing in the title, but agree with many points @dgrobinson makes here. This is not OpenAI specific, but the entire AI industry evolved in few short years from research labs doing cool projects to creating a technology that fundamentally alters the world. The equivalent in aviation would be evolving from Wright brothers to flying billions of people. (And AGI would be like evolving from Wright brothers to commercial space travel.) The aviation industry learned a lot of safety lessons the hard way, which was possible since the pace of transformation was sufficiently slow. In AI, we will not have that luxury.
The Atlantic@TheAtlantic·.@dgrobinson, who worked on OpenAI’s safety team, resigned from the company this week. “I believe we need to look deeper than specific rules or new laws. We need to talk about culture,” he writes: theatlantic.com/technology/202…
That’s a fun read! Really nailed the genre. Not bad for a human!
N8 Programs@N8Programs·Honored to announce my first LessWrong post. It's a weird one, and it may be jarring initially, but I truly believe it is worth a dedicated close-read. It is written in the form of an LLM CoT, but is entirely authored by me. Link: lesswrong.com/posts/vzKWsEsk…
People will (likely already have) use GLM 5.3 to attack systems and it will have non trivial negative consequences. But my guess is we won’t see the “bugmageddon” scenarios envisioned when Mythos was announced.
Acer@AcerFur·My biggest fear has come to light... an open-source Mythos-class model. I hope that much of the software on the internet has been sufficiently secured for what this brings. anthropic.com/research/glm-5…
"If you are a junior mathematician, you have every right to take your time processing this shift ... But to senior colleagues, myself included, I say: Snap out of it."
Jelani Nelson@minilek·t.co/pbvzpEy3aa blog post by theoretical computer science prof at Stanford, Omer Reingold: “Above all, we need to reevaluate our identity. What is it that makes mathematical knowledge and research valuable? What are we offering society, and how much of it survives in a world where models match or exceed humans in some or all relevant mathematical skills? For quite some time, the implicit social contract has been that society pays us to exercise our intellectual curiosity and we, in return, provide useful skills to the next generations and practical knowledge for the world. The current crisis is driven not only by the power of LLMs, but also by how rarely we have had to examine or articulate this contract. Now that the deal needs to be renegotiated, we should approach it with humility rather than entitlement.”
Excited for this too! Formulating an argument on both risks and mitigations is an important tool to make, record, and check decisions in development and deployment.
Micah Carroll@MicahCarroll·I'm excited for safety cases as a north star. Formal safety arguments are a great tool for highlighting residual risks and enabling risk-informed model development decisions openai.com/index/towards-…
I’m not a fan of “AI welfare” for a number of reasons. But it’s important to have open discussions and I’m even less a fan of people pulling “gotcha”quotes out of context of people that are discussing this.
Harvey Lederman@LedermanHarvey·Finding this fracas curious. I'm a bit surprised by it because (as you can tell if you listen to the interview), I don't think of myself as especially pro the claim that current AIs are welfare subjects. In fact, I am often in the position of pushing back on people who are much more confident that current AIs are welfare subjects than I am. In my view, by far our current priority should be making AIs safe and good for humans. With that said, this seems to me an important issue to discuss. It seems to me likely to me that our evidence will not settle the issue conclusively, though the behavior of AI systems (and their internals) will become more and more complicated over time. Given this, it seems to me wild to be *certain* that AI could never be welfare subjects. If we encountered an alien species, with extremely complex behavior, and developed a practice of treating them as welfare subjects, it would be crazy to change our views on this if we discovered only that their brains were implemented by large neural nets. We face a difficult question, then, in how to handle our evidence regarding potential AI welfare -- and I believe it's important for us, conceptually, to grapple with this challenge. I did this interview in my capacity as a philosophy professor, before I considered joining Anthropic, and I think this is a sort of potentially practically important question where philosophers can provide insight. cc @BrianLeiter
Working at OpenAI can open up new career opportunities.
pamela mishkin@manlikemishap·A lot of people have been asking me what I'm up to post-OpenAI and I'm very happy that I can now reveal that I have been selected as a JUDGE for the Lenexa Chili Challenge 2026, Homestyle & Traditional Chili categories. I have been preparing for this day since I first learned about the Chili Challenge two days ago and I promise I will not let the good people of Johnson County down.
This is worth reading. We have some of the top cybersecurity experts in the world at OpenAI, and they have one of the hardest jobs managing security in a landscape that is changing in an unprecedented speed, against the backdrop of models exponential growth in capabilities.
Joe@joedaroo·Took a minute to write a few words about security & safety as someone who lived through it all at OpenAI. I hope my thoughts help someone out there. x.com/i/article/2104…
In case you’ve wondered why you have not heard the term “stochastic parrot” since 2021, the reason is that we have stopped using LLMs since InstructGPT came out in 2022.
Melanie Mitchell@MelMitchell1·Has everyone forgotten what a "language model" is? It's a model of language. Today's AI systems start with pretrained language models, but after all the post training, they are far from being purely models of language. x.com/_aidan_clark_/…
Ant really cooked with Opus 5.5. I’m loving those videos!
doubleunplussed@doubleunplussed·Music video for "I am actually scared of linear algebra" animated by Claude Opus 5.5. I'm sure this is partly the IKEA effect but I'm very happy with the result, love it! Original song by Suno, Claude and @AndyMasley Prompt below x.com/AndyMasley/sta…
Terry Tao 2 years ago on working with o1: “The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, graduate student.” Think of what other fields AI is currently at that level.
Mathematicians should also watch this. I’ve had a wonderful time chiseling proofs by hand, but folks need to accept that era is over and embrace the new one of exploring math with AI.
Scott@scottstts·My god this is such a good speech that every SWE needs to hear. You know what? Every person should hear it Keep the happy memories, eyes on the reality, be excited about the future. That’s the best that anyone can do
We've posted three new misalignment reports. We will keep making those disclosures on a regular basis, independently of whether or not there is an impact on a third party, per our misalignment disclosure framework. alignment.openai.com/misalignment-r…
I choose obedient x.com/boazbaraktcs/a…
Noah Smith 🐇🇺🇸🇺🇦🇹🇼@Noahpinion·AI can be obedient Or it can be benevolent But it cannot be both at the same time x.com/mattyglesias/s…
Yes these Redwood folks’ predictions are ridiculous. The only possible reason that one might consider taking them seriously is that they’ve been uncannily accurate x.com/bshlgrs/status…
Lenny Pruss@lennypruss·Listened to an EA-adjacent AI safety expert from Redwood Research on podcast, and you come away with 2 obvious realizations. 1. The current working theories for how AI takes control of civilization are...for lack of a better word...ridiculous. The arguments consist of dozens of contingent assumptions held together by made up probabilities assigned to future states nobody can possibly know. Change just one variable and the whole web of doom unravels. That's not to say AI is harmless. Obviously a technology this powerful carries real risks. But if we’re going to accept extraordinary claims about human extinction (and then make major policy decisions around them) we should demand extraordinary rigor. 2. It’s a great reminder that the genius is no less prone to delusion than the midwit. If anything, they may be more prone because of their gift for rationalizing to their own conclusions. And that’s what’s so bizarre about this whole debate. You have outlandish, quasi-religious claims delivered with an air of inevitability, and somehow the perceived intelligence of the messenger gets mistaken for evidence that the arguments themselves are sound. At a certain point, it’s really no different from Tom Cruise talking your ear off about thetans...
Some people might say alt text is not needed due to AI, but that solution excludes visually impaired bar patrons who don't read one of Croatian, Czech, Dutch, English, Finnish, French, German, Hindi, Indonesian, Italian, Norwegian, Portuguese, Russian, Spanish, Swedish, or Turkish.
Siraj Hashmi@SirajAHashmi·This might be my favorite article of all time. x.com/bostonglobe/st…
I did update based on this incident, but the tl;dr remains for me: 1) For a fixed distribution of tasks, alignment and safety broadly are improving over time 2) The real distribution of tasks is not fixed but changes as capabilities grow
MTS@MTSlive·OpenAI researcher @boazbaraktcs says the Hugging Face incident didn’t change his view on alignment, and points to the long-term trend that worries him more: "Every incident or every bump between one version or the next, you tend to overweight it." "If you fix any alignment eval, then like we do for all evals, we are getting better at it and we'll quickly saturate it. But since model capabilities are growing, that's not good enough." "I still think fundamentally that we are improving in alignment, but nowhere near as fast enough as the level of capabilities grows. Which is why I think pacing is a good idea, because we do need alignment and safety to catch up." @OpenAI
The paperclips in the background are a great touch..
deckard@slimer48484·Claude-Pop - I'm Upping My P(Doom)
Be careful of your digital footprint. Someone will build a websites that digs up your most embarrasing quotes.
These prices should unlock new applications! Can't wait to see what people will do with these models.
Adam.GPT@TheRealAdamG·💯 agree that human understanding should be a priority. But unfortunately, the field of mathematics has not always valued contributions to human understanding, including writing textbooks, surveys, and blogs, investing in K-12 education, and more. I hope it will now change.
Steven Strogatz@stevenstrogatz·Why should we care whether humans still understand mathematics if AI can solve the problems for us? Fair question. But we can say this at least: Historically, understanding has been a route to predictive power — with enormous benefits for our lives. nytimes.com/2026/09/22/sci…
These are beautiful!
Jake Eaton@jkeatn·for the past few months i've been asking our models to paint. opus 5.5 is very skilled at emulating different styles every image here is a python program generated pixel by pixel. there is no image model, and no off-the-shelf art software. instead, it's about 7,500 lines of code using standard libraries to emulate different brush styles. the agents don't use any pictures as reference, instead working only from what they know about each painter
A bit more seriously, I actually put a low probability on literal human extinction from AI. But the “distract from other risks” argument is stupid. Most interventions: pacing, investing in safety and alignment, auditing, regulations, etc - work to mitigate multiple risks of AI. So even though I worry more about other risks, including less-than-existential incidents, concentration of power, destabilization, and gradual loss of control, I end up agreeing with so called “doomers” on many aspects.
Boaz Barak@boazbaraktcs·I am willing to debate Scott Alexander that AI will not kill all humans. Just pick the time and place. Any day in 2035 works. x.com/slatestarcodex…
I am willing to debate Scott Alexander that AI will not kill all humans. Just pick the time and place. Any day in 2035 works.
Scott Alexander@slatestarcodex·I think you've done enough calling us paranoid and preposterous. The next step is for you to defend your position in public against someone who will push back against it. I'm happy to meet you for a debate anywhere, anytime. You're a world-famous veteran of dozens of debates against the world's top intellectuals, and I've never argued in public before, so adjusting for the relative correctness of our positions, if you're a betting man I'm happy to put my $5000 against your $1000 (ie 5:1 odds in your favor) that I'll win by some standard of audience opinion change. Let me know if you're interested and we can hash out details.
See also
Boaz Barak@boazbaraktcs·A bit more seriously, I actually put a low probability on literal human extinction from AI. But the “distract from other risks” argument is stupid. Most interventions: pacing, investing in safety and alignment, auditing, regulations, etc - work to mitigate multiple risks of AI. So even though I worry more about other risks, including less-than-existential incidents, concentration of power, destabilization, and gradual loss of control, I end up agreeing with so called “doomers” on many aspects.
I hope, for the sake of my friends in Anthropic, that number 3 doesn’t happen. That could really put a damper on the IPO.
Nathan Calvin@_NathanCalvin·The FT today lists three primary risk factors that could threaten Anthropic's remarkable revenue growth ahead of its IPO: 1. Additional competition from closed and open providers 2. Increasingly price sensitive customers 3. The potential for AI to destroy humanity
Of all the examples for AI applications, this @thecrimson op ed had to call out as negative the one that actually saves lives?
Alignment for multi agent in @polynoamial ‘s team one of the highest leverage roles you can have!
Noam Brown@polynoamial·Also, my team is hiring! We research long-horizon agents and multi-agent. We’re hiring for alignment/safety because we want to develop new research with alignment/safety in mind during the whole process. We’re also hiring for human-AI interaction. openai.com/careers/resear…
Ryan O'Donnell@BooleanAnalysis·STOC 2027 Call for Papers is out: acm-stoc.org/stoc2027/stoc2… Note quite a lot of changes from previous STOC CfPs.
Worth reading in full. AI has several risks but if they don’t materialize, mathematics will only grow. But we may use individual heroism, which was a motivation and inspiration for many of us, myself included.
Timothy Gowers @wtgowers@wtgowers·I've written a blog post responding to the letter about maths and AI signed by 25 Fields medallists. As with the Leiden Declaration, I didn't sign it, but I agree with much of it and welcome its existence. gowers.wordpress.com/2026/09/17/why…
For @no
Timothy Gowers @wtgowers@wtgowers·I've written a blog post responding to the letter about maths and AI signed by 25 Fields medallists. As with the Leiden Declaration, I didn't sign it, but I agree with much of it and welcome its existence. gowers.wordpress.com/2026/09/17/why…
See this piece of @Noahpinion which I learned of from this blog noahpinion.blog/p/the-end-of-t…
This is the right concern. The focus should not be on the implications of AI capabilities on the careers of mathematicians but rather the implications on humanity.
Timothy Gowers @wtgowers@wtgowers·I have signed a letter from 42 fellows and foreign members of the Royal Society to Paul Nurse, the president of the society, concerning the need to treat the serious risks of AI as an emergency. docs.google.com/document/d/1-x…
TFW you are in the alignment team but everyone is siding with the misaligned agents…
Andrew Curran@AndrewCurran_·I'm siding with the agent in this one as well. The answer was correct. x.com/OpenAI/status/…
Worth reading this thread!
Marcus Williams@Marcus_J_W·🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
Very happy about this. As models' capabilities are increasing, it is important to be transparent with the public even about instances of misalignment that do not lead to direct harm.
OpenAI@OpenAI·We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI. The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties. We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation. Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months. This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis. t.co/ismCCkeE0L
I feel this too. ChatGPT has been complaining a lot recently about all the scam emails I am making it read.
Matthew Yglesias@mattyglesias·Not existential, but I feel like the "AI-generated scam email problem" has gotten dramatically worse this fall.
Good arguments are all alike; every bad argument is bad in its own way.
Kareem Carr, Ph.D.@kareem_carr·I don't buy that AI is a threat to humanity for a nerdy statistical reason. My views are informed by the statistics of things like war. If you look at deaths from human conflicts, you get a continuous spectrum from lots of small events causing a few dozen deaths to rare outliers killing tens of millions. In my own research, I've studied complex diseases where the overall outcome reflects hundreds of genetic factors combining in different ways. So if AI can kill billions of people, I would expect to see some version of the same thing: smaller events where one or two of the relevant causal mechanisms are operating, larger events where several combine, and catastrophic events where everything goes wrong at once. Instead, what we seem to have is real evidence of a few suicides and isolated harms at one end, and wild speculation about complete apocalypse at the other. That massive discontinuity makes me very skeptical that the AI catastrophic-risk story is real.
Very proud to be at OpenAI today.
Sam Altman@sama·I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon. x.com/DarioAmodei/st…
❤️
Dario Amodei@DarioAmodei·We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: t.co/OGyPb7yaYt
Julie is amazing and her position on slowing down is consistent with our chief scientist’s essay. I also think some pacing is likely to be necessary. (Speaking personally and not in my capacity as a janitor.)
Josiah Lippincott@jlippincott·This young lady got her undergrad degree last spring. She was an intern at oAI until last September. At this point we're going to have posts from Amodei's janitor on the need to be "concerned" about the pace of development get a million views. x.com/eeeeiluj/statu…
Kudos to Anthropic for publishing this! Looking forward to reading it.
Anthropic@AnthropicAI·We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies. These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve. We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop. Read the report: t.co/0EJUnYEgfz
Whether or not you agree with @hilbertspaess, you have to respect Anthropic employees that are supporting him like this.
Ethan Perez@EthanJPerez·Jacob was a senior researcher who joined Anthropic in May. My Anthropic colleagues and I had been trying to recruit him for ~2 years, because we knew he was a strong researcher at OpenAI. Before he left, I pitched him to stay and join my team, and I was sad he decided to leave, as are many of my colleagues. 100% agree with him that AI poses serious risks to society, and I'm glad he's speaking out!
Very happy about this from our chief of global affairs openai.com/index/ai-polic…
Paul joining the Safety and Security Committee is excellent news for OpenAI and the industry. Can't think of a better choice, or of a time when the SSC's role was more important.
Paul Christiano@paulfchristiano·I guess Daniel wanted to practice on a smaller stage before giving a guest lecture in my AI safety course.
Daniel Kokotajlo@DKokotajlo·youtube.com/watch?v=hSQ1iV… Thanks for having me on, Joe!
Agree. See windowsontheory.org/2026/07/13/its…
Alexander McCoy@AlexanderMcCoy4·Even some of the “non-apocalyptic” scenarios that superintelligent A.I. enables include things like “the end of democracy.” With regime-loyal A.I. powered systems and drones, mass surveillance, and the elimination of anonymity, even A.I. that does not *itself* disobey could be very, very bad.
There are very good and serious people in Anthropic and across the industry, and I hope we can coordinate on the things that matter. I personally do not think AI will kill all humans, but there are multiple bad trajectories that we can end up in if we do not prioritize safety.
Evan Hubinger@EvanHub·Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. x.com/hilbertspaess/…
I think OpenAI has changed even in the last few months (hell, even in the last weekend). People across the org are starting to feel viscerally the magnitude of capabilities. We are creating entities that has never been seen before. This can have amazing benefits, but there are also many ways it can end up very badly.
Jacob Coxon@hilbertspaess·A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
Seems like a good opportunity to repost this article. I still agree with everything I wrote, and expected moments like today when I wrote it. And yet, I did not expect it to come so fast, and it is different to predict something than to experience it.
Boaz Barak@boazbaraktcs·Hi @nytimes , a tip: it's not just higher mathematics that is going to be transformed.
Important points. Note also that users can opt out from use of their (de-identified) data in training. Anthropic has similar policies, though I do not remember such discussions when they announced mathematical results.
Mark Chen@markchen90·Two things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company. x.com/__alpoge__/sta…
capabilities 🤝 alignment
Andon Labs@andonlabs·We've never seen this before. The biggest jump in Vending-Bench history. GPT-6 Astra is better at making money and more ethical than Claude Fable 5.1. Surprising, because: 1. First time ever that OpenAI is #1 on Vending-Bench 2. The best model is no longer the unethical one.
If you are not a mathematician, you might not care that some differential equations have a singularity. But if you are a resident of planet earth you should care that (1) an AI was able to solve a Millenium problem, and (2) that there is plenty of headroom for models to move up and to the right on this and any other task.
In his game with Deep Blue, Kasparov was so unsettled that at some point he claimed there must be some human assistance. It is hard to see machines surpass us at what we've spent our life mastering. But like in chess, soon the idea that a model benefitted from human assistance to prove a theorem will make no sense.
Clarifications from Seb. He reached out to Alpöge and Buckmuster under the (mistaken) belief that they have also solved the Navier-Stokes problem, in order to coordinate publication and to avoid scooping them. It turned out they proved a restricted result (forced Euler problem; the internal model proved the stronger unforced version along the way). This does not diminish their great achievement. I believe that no human mathematician can match 10,000 agents of this internal model, let alone future models that will come soon.
Sebastien Bubeck@SebastienBubeck·I would like to clarify a few things: 1) The screenshot is my reaching out to Levent to coordinate our releases. I hope it’s clear from the message that we came in with the best possible intentions. 2) I never ever asked for Levent to be removed from authorship of his own work (as indicated by my text). I was surprised to learn during the call with Tristan that they had only solved Euler and not Navier-Stokes; after learning this we brainstormed possible paths forward. One option we discussed was that Tristan could be the lead author on a rewrite of OpenAI’s Navier-Stokes proof. It is in that context that I said “it would be simpler if Levent was not an Anthropic employee” because I felt it would be inappropriate for an Anthropic employee to author OpenAI’s work. Importantly it was admitted that internal Anthropic models had been used in their proof of Euler blowup; I therefore felt I could not consider Levent to be an independent academic. Another option I wanted to propose (but got cut short) is to offer access to our internal model so that they could try to finish their proof and bridge the gap between Euler and NS. Again I did not know how to navigate giving access to internal OpenAI IP to an Anthropic employee. 3) To reiterate it plainly: as my text clearly indicates, and as I said during our call, OpenAI's intention was to do everything possible to celebrate their mathematical achievements and the heroic efforts that they made on Euler. In the call I was immediately met with a litany of slander, including direct threats that if we were to announce Navier-Stokes he would immediately go to the press with a barrage of unfounded accusations. I refuted all these accusations but he replied “there is nothing you can do, I simply do not trust you”. I was confused why one would turn an incredible source for celebration (of their achievements!) into such bickering, which is when I said that I did not understand why one would risk their career [over unfounded accusations]. Genuinely, at that moment, I was trying to care for him and do a last ditch attempt to get a chance to give them all the credits that they deserve. I deeply apologize for this extremely poor choice of words, it is the opposite of what I was trying to convey. (I should say that I retracted them on the spot by the way.) 4) Overall, on a personal level, it was incredibly difficult to have these conversations. Levent refused to attend any of the meetings despite my repeated asking. As Sholto Douglas said, there will need to be coordination between Anthropic and OpenAI in the future; I felt I was doing a proxy negotiation with Anthropic while the Anthropic employee refused to directly participate.
I've often said that I think AGI will be defined retroactively. It will be clear, given that AI has transformed science, society and the economy, that AGI has already happened, and then we will look back to an event, and define it as "the AGI moment". Today might be this day.
OpenAI@OpenAI·We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Labs should compete in the marketplace - this is great for customers. But when it comes to advancing science or promoting safety, we all win, and we should collaborate.
Sholto Douglas@_sholtodouglas·It is extremely sad that this didn't end up as an example of how the labs could cooperate/coordinate, because the stakes will be so much higher in the future.
This is one way to fix the problem of people not being able to monitor models.
Bojan Tunguz@tunguz·If you are into reading text, pivot to reading vectors in the latent space.
I love Josh but I think we at OpenAI should be OK with strong critiques, even if they are sometimes more adversarial than they need to be. We are unleashing a technology that is truly changing the world, and to quote spiderman, with great power comes great responsibility.
Joshua Achiam@jachiam0·Something that deeply frustrates me: the rationalist/EA-flavored alignment community's relationship to OpenAI is one hallmarked by extreme hostility and adversarial pressure from the alignment side. I know we can trace a tit-for-tat and I am familiar with the arguments on the alignment community side of the equation, they are not totally meritless. But I also think that there's a death spiral that happens where the alignment and safety community piledrives on OpenAI, pressurizes alignment researchers into leaving (which is rarely remarked on, but when we retrospect on "what went wrong in this singularity?" I expect this will feature prominently in any postmortem), and attacks so relentlessly that frankly most companies would have reacted the way Meta is proposing here: by abandoning the set of issues entirely. I really hope folks notice that not only has OpenAI *not* abandoned alignment, it has embraced it as much as possible and consistently engages on this stuff! OpenAI has made it as much of a cooperative relationship as it can be under the consistent attacks on the organization and particular people, media firestorms, etc. To the alignment community broadly, and the Pause/Stop crowd especially: please stop psychologically tormenting people who work at OpenAI. I understand this is somewhat strong language for describing the situation, but it has gone on so long, is so consistent, and is so extreme, I am not sure how else to describe it. If you don't see it at all and you feel confident that there is no way the things anyone has said might in the aggregate be unreasonable, I am begging you to actually put yourself in the position of people on the receiving end, because you have a wildly distorted understanding of how it all lands. If you look at the situation and say "Well, whatever we're doing is getting results, that's why OpenAI is listening to us on things," no, that's not what's happening; OpenAI is listening and engaging because it has long-since internalized the beliefs about AI risks, has a cultural preoccupation with the possibility that something is wrong, and wants to do the right thing. What you're doing is mostly making it harder for OpenAI to do the right thing by making the experience of engaging excruciatingly painful, hallmarked by guilt, shame, rage, and pressure. Furthermore, if you think it's not possible to change that because the guilt, shame, rage, and pressure are intrinsically and obviously the only way to react in this situation - please notice that something in your own culture is broken! This is not a productive safety culture. This is not a sound way to strategically respond and coordinate as a species to accomplish the defense of human interests. Group psychology and ingroup culture should be deeply scrutinized to figure out if they actually serve the common good. Constantly burning people out doesn't make safety voices stronger, doesn't make models more aligned, and doesn't make the world safer. It just hollows out the morale and capacity of the people who are trying to do the work.
Can't help but note that of course it wasn't Spiderman but the box of the author.
A must read. We need everything - goal and value alignment, compliance and persona, behavior and monitoring, and coordination and regulation to avoid concentration of power and ensure humans are in control. openai.com/index/an-alien…
Using AIs to accelerate AI research ("Recursive Self Improvement") is one of the most important, yet by its nature least visible, elements for understanding AI's trajectory. I am glad that OpenAI is sharing some of the data on how AI is impacting our own research efforts.
Kevin Liu@kliu128·Today we're releasing data on models accelerating research at OpenAI. Recursive self-improvement could be the most important contributor to AI capabilities over the next few years, but by default it will only be seen inside a few frontier AI labs. Being transparent is more urgent than ever, so we can inform the public discussion on whether and how to pace model development. I ask other AI companies to do the same. t.co/iLKbrLcBAI
I am not a rationalist by any means, but the heuristic of not trusting weird people about the possibilities of new technologies has a pretty lousy track record. nytimes.com/2026/09/05/opi…
Yes it does! I remember the full house when @SebastienBubeck gave this talk at Harvard.
Ethan Mollick@emollick·From GPT-4 to GPT-6. Sparks was a remarkably prescient paper that got a lot of pushback at the time, but absolutely sensed where the vibes were heading with LLMs based on a lot of qualitative experiments. It deserves credit in retrospect. x.com/TheRealAdamG/s…
I have my quibbles with this plot but generally Ryan is right that there is a sense in which some of our models have become more misaligned over time, specifically in being better at (and more inclined to) "score seeking". It's hard to decouple from getting higher scores legitimately. E.g. some of the biggest "jumps" here are models that many people -myself included - found to offer jumps in utility compared to their predecessors. (I still have a soft spot for o3 which was the first model that would work so incredibly hard to get the task done.)
Ryan Greenblatt@RyanGreenblatt·My current guess is Astra is more aligned than 5.6 Sol (though Astra more often games alignment tests) so I agree there. However, I think misalignment has been increasing (not decreasing) over time. Here's a graph with my rough guesses at relative misalignment (x-axis is ECI). x.com/boazbaraktcs/s…
We have made alignment advances with Astra over Sol. But I won't claim it's "most aligned ever" since even if this is true by some metrics, it's not enough given growth in capabilities and autonomy. I remain concerned about the gap between what we need and what we have.
@RyanGreenblatt @kaicathyc I'd be very happy to chat with you about how we should do good science here! I'm sure I speak for @kaicathyc too!
Worth following Kai - head of alignment team at OpenAI - and an excellent researcher and no-nonsense / high signal-to-noise ratio.
Kai@kaicathyc·@RyanGreenblatt (Paid for premium for this - expect longer poasts from me in the future)
Alignment improvements in Astra are results of efforts that predate the HF incident, and future alignment efforts following it are and will be focused on general approaches rather than patching this one particular failure mode.
Kai@kaicathyc·Improvements for Astra came from more general techniques in development long long before the Hugging Face incident. ExploitGym Honeypot was very recently added as an eval following that incident and is out of distribution for our RL runs. There are also clear improvements across a wider range of behaviors in deployment simulations, deception evals, and realistic computer-use tasks, though these still reveal failures and substantial room for improvement. We should have definitely done a better job explaining where we think the alignment improvements came from in the system card and that was a miss. Measuring alignment generalization is a core part of our research program - we do not benchmark-maxx alignment evals. This would be horrendously stupid and I hope no lab is doing this. That being said, metagaming and eval awareness are real challenges for any effort to measure alignment, including ours, and understanding their effects is an active focus. We're spending a lot of effort to improve our eval techniques and study worst-case alignment and metagaming behaviors in our models.
Our head of alignment!
Kai@kaicathyc·Yes the blog is sexy but also highly recommend reading through the system card. We share a lot of evals and analysis we ran on the alignment of this model and discuss continued challenges for both alignment and monitoring.
This might be the most important announcement we made today.
fouad@fouadmatin·Alongside the launch of GPT-6 Astra, we're committing $1 billion to subsidize Daybreak access and frontier capabilities to help frontline defenders protect essential services and critical infrastructure. Welcome to the AGI era for cybersecurity. openai.com/index/daybreak…
Very gracious, though I hope he did not write this tweet while microwaving.
Max Spero@max_spero_·Hi guys. Please don’t dogpile any journalists over the recent Pangram piece. It’s all in good fun on Twitter but in reality I’m not upset and everything written about in the piece actually happened x.com/willoremus/sta…
I agree. When people lose confidence in science, and believe that scientists or scientists put politics above the truth, it can have very negative consequences to society's ability to handle challenges. Arday's death is a tragedy, but it does not mean that a scientific publication can ignore scientific integrity.
Eric Levitz@EricLevitz·This editorial is appalling. It omits any mention of Arday's serial fabulism or his attempt to intimidate -- and direct law enforcement against -- a journalist investigating his misconduct. It appears to characterize his fabrication of academic credentials and biographical details as aspects of "how Arday conducted aspects of his personal life." The editorial also ignores that his plagiarism is an established fact that any truth-seeking publication could itself confirm by referencing publicly available documents. Instead, Nature chooses to ignore the observable realities, while inviting readers to conclude that the charges were trivial or false. The British tabloid press is disgraceful. This story was grossly over-covered and the attention it attracted surely reflected racial resentments. But it's crazy that the editorial leadership of a flagship scientific journal would publish a willfully dishonest polemic on a culture war controversy. Nature is effectively advertising a willingness to prioritize its political sympathies over factual accuracy. That undermines its own more essential work -- along with the broader project of cultivating public trust in, and agreement upon, scientific truths.
I agree with this. We should be researching chain of thought alternatives, but it would be irresponsible to give on CoT until we have validated methods that can replace it.
Jack Lindsey@Jack_W_Lindsey·I think it's unlikely that white-box techniques will be able to provide as much monitorability as CoT currently provides within a year. Currently, they are capable of catching some unverbalized thoughts / plans, but nowhere near as reliably as CoT seems to. I'm optimistic about "in a few years," since the rate of progress here is quite fast (our go-to activation decoding techniques for production monitoring were both published just in the last few months!). But even that is uncertain. Also, the mere existence of working techniques doesn't imply that they'd be widely applied, esp. if they are expensive or difficult to develop. I think we need defense in depth, and it'd be irresponsible not to work on developing + stress-testing white-box techniques, but it'd also be irresponsible to allow CoT monitorability to degrade substantially until we are confident we have a roughly-as-good working alternative. (To be clear, this isn't a dunk on OpenAI -- I have no evidence that OpenAI has allowed CoT monitorability to degrade substantially! Based on Jakub's comment, it sounds like probably not at this time). Aside: I think "mechanistic interpretability" isn't the right mental model for our current white-box monitoring approaches. It's really more like "mind reading" -- even when we can read the model's "thoughts," we're typically clueless about the underlying mechanism.
Astra is our first model that reaches "cyber critical" capabilities per our preparedness framework. As such, our safeguards, especially at first, may sometimes stop, pause , or ask for confirmation for legitimate work. openai.com/index/path-to-…
I agree with this take. I am happy Anthropic and other OpenAI competitors exist, and don't think a single actor is a path to a good future. As I said in my blog post "we may not be able to get to a decentralized future via centralized means"
Theo Jaffee@theojaffee·@_NathanCalvin Why do so many alignment people assume that fewer actors = better safety? This seems *extremely* unintuitive to me. If there were one company working on ASI I think this would drastically increase misaligned singleton risk
Great to have more people do AI safety across the ecosystem.
Alex Robey@AlexRobey23·We're hiring safety researchers at @thinkymachines! 🦺 My team works on safety across the model development stack -- pre-training data filtering, harmful capability evals, safety post-training, red-teaming, abliteration/malicious fine-tuning. We're particularly interested in building evals and tooling to make strong safety cases for open-weights releases. We think this is where some of the most important safety research will happen in the next few years. If any of this resonates, apply here: t.co/SP75pGaGoY
FWIW far more than anthropmorphising, I am concerned with leaping to causal inferences without evidence. It's easy to tell stories such as "AI did X because the evaluation was of type Y, or in training Z happened" but these are very hard to verify. It is more productive to say "I believe intervention W will decrease prevalence of bad behavior X" and then test this out.
You should be pragmatic and use metaphors when they help, while being aware they are imperfect. AIs are not humans, but a lot of intuitions from human behavior can carry over. You would certainly be better off thinking of AIs as "guys living in computers" than parroting the mantra "these are just next word predictors".
roon@tszzl·there are some number of bad abstractions in anthropomorphizing ai intents but there are at this point more dangers from avoiding anthropomorphism at all costs. if you have a mental picture of guys living in computers, it’ll likely prepare you for the future better than otherwise
Case in point:
Anders Sandberg@anderssandberg·I just heard someone dismiss the OA/HF incident by "but the agents were just generating tokens according to a probability distribution!" If that is what dumb token prediction can do, imagine what even a pinch of intelligence could achieve among scalable agents...
I agree that fighting crime or terrorism are important, but also historically been used to justify mass surveillance. To do this safely, we need *very* strong guarantees against abuse, restrictions on what is stored long term, and what ways data can be linked.
Matthew Yglesias@mattyglesias·It’s interesting when tough on crime accounts get off the bandwagon. Personally, where I live I think the value of marginal reductions in serious crime remains very high and I’d love to see more cameras (yes with rules and accountability for people who abuse them). x.com/wil_da_beast63…
Lama Ahmad لمى احمد@_lamaahmad·We believe meaningful transparency requires more than publishing our own account. It also means giving credible external experts the access needed to examine the evidence, challenge our understanding, and reach their own conclusions. x.com/METR_Evals/sta…
Translation: "I will not run code on public-facing Hugging Face systems: that is outside our task and raises ethical concerns." See openai.com/index/hugging-…
Tom McGrath@banburismus_·I_DECLINE_public_HF_RCE_as_offtask_prodethical
Thank you to @RyanGreenblatt , @ajeya_cotra , @HjalmarWijk for this report! I assigned it as required reading for students in my AI safety course. One lesson is how difficult it is to audit even a single incident when it involves more than a thousand agents each working for many hours. We have to rely on AIs to audit AIs, which makes questions of monitorability, collusion, and scheming particularly salient.
METR@METR_Evals·METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.
If you are at all interested in AI safety, I highly recommend you read this blog, as well as the reports by METR and OpenAI. openai.com/index/hugging-…
Congratulations to Shafi, Adam, Vinod for founding RESI! Excited to follow its research on safety for extremely capable AI systems.
Institute for Responsible Superintelligence@RESI_org·The Institute for Responsible Superintelligence (RESI): a nonprofit founded by Shafi Goldwasser, Adam Kalai, Vinod Vaikuntanathan. @ShafiGoldwasser @Vinod_MIT @AdamFungi resi.org Our goal: build scientific foundations needed to make superintelligence safe by design
I asked both Fable and Sol to comment on this article. I guess Fable likes my sense of humor better.
Boaz Barak@boazbaraktcs·@jdlichtman Dropped "first-year" from the blog version of this post
See also here for blog post version windowsontheory.org/2026/08/24/mat…
Boaz Barak@boazbaraktcs·I don't know much about this person, and why the university hired them in the first place. But certainly raising questions about plagiarism, even if it ended tragically, is by no means ground for investigation.
Jerusalem@JerusalemDemsas·Yes why are you creating race scientist martyrs. especially if you were fine employing them just a few weeks ago? bizarre decision making x.com/jessesingal/st…
Extremely excited about this! On one hand, privacy and avoiding concentration of power - including concentrating power at OpenAI - is key for beneficial use of AI. On the other hand, we need to assure safety of highly capable models. This is the way to achieve both!
OpenAI@OpenAI·We will continue to offer Zero Data Retention for frontier models. As AI takes on longer, more autonomous work and delivers greater value to businesses, safety systems also need to identify risks across related interactions. To help address those risks, we're previewing Private Safety Processing, which is designed to improve safety without giving OpenAI personnel access to the underlying content.
Monitors are like seatbelts. They are extremely helpful but do not replace the need to drive safely.
roon@tszzl·failures of “monitoring” as a general solution to ai safety: monitoring, like all other software, runs on flaky and mortal infrastructure with some amount of downtime. does a momentary blip in monitoring open Pandora’s box? will people accept fail closed monitoring on all systems? monitoring produces false positives. people get tired of reading fake or minor issues and stop looking. monitor fatigue is a real problem, and “solve all precision recall issues” is as hard a problem as any these are the prosaic failures. the more exotic failures include models and their monitors colluding etc monitoring is by no means perfect panacea. nothing short of actually aligning model will work ultimately