Boaz Barak

I agree that if we want a human controlled future, we need to hire and train junior employees even at a cost to short term efficiency. x.com/jasonfurman/st…

I mostly agree with you @Claudia_Sahm. I expect colleagues to continue to have research assistants even if only for mentorship than because they need the help. Companies to have junior staff not because they need their current work but to train them for future roles. x.com/Claudia_Sahm/s…

To make sure our frontier models don't cause another HuggingFace incident, we built an offline world that they use during training. It provides everything that the models need in a fully cached, indexed, or synthetic manner. We then built a huge team of our most cyber-capable models and told them to collaborate to find zero-days in this containment infrastructure. While they didn't find a path to the internet that wouldn't be caught by our CoT monitors, they did find ways to escape popular VMs, exploit our network infrastructure, and even ways to communicate with other agents running in separate VMs by using hardware side channels. We used this to dramatically harden our infra, and we also upstreamed a ton of patches to open-sources packages.

How do you tell if authors fabricated their research results? Sometimes: they just tell you. Read, e.g., this caption carefully. Another banger of a post by my friend Nihar B. Shah on his blog, where he found at least 20+ such #ICLR2027 submissions: researchonresearch.blog/2026/10/20/sel…

I don't think "everything is fine," but anyone claiming that php2shell is "the worst day in cybersecurity history" needs to explain why there's more risk here than: Innumerable celebrity vulns and exploit PoCs, Serious protocol bugs in TCP/IP/DNS, Out-of-control worm incidents, Nation-state attacks on economies/critical infrastructure e.g. Russia->Ukraine, Kernel packet-parsing bugs in popular operating systems, The proliferation of AI social engineering tools since 2022, The proliferation of AI hacking tools. If you can't explain why this time is different/worse, with reference to clear technical mechanisms and attacker motivations, you don't have a serious argument.. I also think this kind of alarmism detracts from the genuine fact that we have a looming crisis in security due to AI which we need correct risk models for

Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.

I agree with this and adopted this stance unilaterally on a recent review. I encourage you to adopt it in your reviewing — it’s reasonable to write reviews that say “this is dense unreadable slop, my review isn’t detailed and I recommend rejection solely on that basis.” x.com/florian_tramer…

From Freddie deBoer: "human life in 2026 appears to be almost entirely similar to human life in 2021" Does it, though? ~All of my friends and family who work in technical or academic fields use AI in wide-ranging ways within their work. All say their jobs have transformed.

OpenCode is currently under a sustained cyberattack, and mid-incident they ran into the same problem Hugging Face did: the frontier models they were using for defense began refusing to help fight attacking agents because analyzing the attacks started triggering safety guardrails. x.com/thdxr/status/2…

the Lord came down to see Adam's neural networks. in their greed, they had embedded more features than residual dims. and the Lord spake, "let us confuse their activations, lest nothing they plan be impossible." thus the Lord scattered the features over the residual stream x.com/nabla_theta/st…

There is a very simple way to keep your kids from becoming gay or trans. Raise them in a devout religious monoculture during the peak of that culture’s organization against gay marriage. Provide readings that establish being gay is one of the worst things someone could be. Attend church weekly. Read scriptures and pray nightly. Provide a stable, loving childhood home in which they can see evidence daily that the religious structure works for the people around them, who love them. Send them on religious missions. Structure their life around faith. That way, when they have feelings they don’t understand, they will have deeply and fundamentally internalized that nobody is born queer and can conclude that whatever they’re feeling is simply undeniable proof that they are monsters unworthy of love, and they can begin the healthy approach of cutting all feeling out of themselves until they realize decades later that actually they were always going to end up feeling something like this, no matter what you did, and you taught them to be the primary author of their own torment rather than providing a social role that allowed them to incorporate that monstrous nature of theirs into a healthy life That’s how my culture did it, anyway. And it worked! Only 40% of my same-aged church members in a quiet suburban Utah congregation turned out gay.

I'm often asked what the cruxes are that separate the AI as Normal Technology worldview from that of our critics. I think this is one of them. We think this mindset shift is hard but necessary and achievable. Many others think that as AI capabilities continue to advance, we will inevitably give up control. Creating and compiling indicators of disempowerment and analyzing which direction they're moving remains one of the most important but underappreciated and underfunded areas of AI safety.

Pleased to announce @Curaffairs has put in place one of the strictest anti-AI policies in publishing. Newsrooms across the country are embracing AI writing and laying off writers. Our magazine is doing the opposite, ensuring we remain handmade. currentaffairs.org/news/why-were-…

never bring a new book to read at a pub without a backup to read in case the first choice sucks bc like why even mention #them

I have a vision for how mathematicians can respond to AI: we should steward a "world commons" of mathematics. Here’s the idea: worldmathcommons.org

Hi, I'm Felix. I'm an eng lead working on Claude. I love watching people take flight and do things they never thought possible. We often describe that as "being more ambitious" when using AI, which is tricky advice to follow. I wrote a tutorial: felixrieseberg.com/how-to-be-more…

GPT-6 Astra got +80% on a Physics benchmark. It insisted the expert results were wrong and its calculations were right. A team checked. And yes. GPT-6 Astra was right. New score: 98% x.com/PratikRath15/s…

nytimes.com/video/podcasts…

We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports. Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping. All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September. Read the full report: t.co/mGeVIBgdou

Crowdsourcing: I'm a guest lecturer in @boazbaraktcs Harvard class in a few weeks. (Looking forward to it!). I can assign one article to read to students who are prolly all safety-pilled. What should it be? I want something that they wouldn't have seen and that would challenge.

Crowdsourcing: I'm a guest lecturer in @boazbaraktcs Harvard class in a few weeks. (Looking forward to it!). I can assign one article to read to students who are prolly all safety-pilled. What should it be? I want something that they wouldn't have seen and that would challenge.

Day 2.1/ We have made Auto-review free for all users signed in through a ChatGPT account. You can enable it in settings > permissions > auto-review. Auto-review improves upon the default sandbox setting that requires you to approve everything, which is prone to decision fatigue unless you spend a lot of time configuring specific rules. It allows you to run long tasks while having a second agent review all actions taken by the primary agent. Its only goal is to prevent high-risk actions from being taken and to protect against unwanted actions that are not aligned with the original user intent. This Auto-review feature is now free and does not draw usage from your plan.

Reading all those recent beautiful essays by Terry Tao, Kevin Buzzard and @wtgowers, I could not resist, but write up my deeply personal perspective on how I perceive and see things: docs.google.com/document/d/e/2…

.@dgrobinson, who worked on OpenAI’s safety team, resigned from the company this week. “I believe we need to look deeper than specific rules or new laws. We need to talk about culture,” he writes: theatlantic.com/technology/202…

Honored to announce my first LessWrong post. It's a weird one, and it may be jarring initially, but I truly believe it is worth a dedicated close-read. It is written in the form of an LLM CoT, but is entirely authored by me. Link: lesswrong.com/posts/vzKWsEsk…

My biggest fear has come to light... an open-source Mythos-class model. I hope that much of the software on the internet has been sufficiently secured for what this brings. anthropic.com/research/glm-5…

t.co/pbvzpEy3aa blog post by theoretical computer science prof at Stanford, Omer Reingold: “Above all, we need to reevaluate our identity. What is it that makes mathematical knowledge and research valuable? What are we offering society, and how much of it survives in a world where models match or exceed humans in some or all relevant mathematical skills? For quite some time, the implicit social contract has been that society pays us to exercise our intellectual curiosity and we, in return, provide useful skills to the next generations and practical knowledge for the world. The current crisis is driven not only by the power of LLMs, but also by how rarely we have had to examine or articulate this contract. Now that the deal needs to be renegotiated, we should approach it with humility rather than entitlement.”

I'm excited for safety cases as a north star. Formal safety arguments are a great tool for highlighting residual risks and enabling risk-informed model development decisions openai.com/index/towards-…

Finding this fracas curious. I'm a bit surprised by it because (as you can tell if you listen to the interview), I don't think of myself as especially pro the claim that current AIs are welfare subjects. In fact, I am often in the position of pushing back on people who are much more confident that current AIs are welfare subjects than I am. In my view, by far our current priority should be making AIs safe and good for humans. With that said, this seems to me an important issue to discuss. It seems to me likely to me that our evidence will not settle the issue conclusively, though the behavior of AI systems (and their internals) will become more and more complicated over time. Given this, it seems to me wild to be *certain* that AI could never be welfare subjects. If we encountered an alien species, with extremely complex behavior, and developed a practice of treating them as welfare subjects, it would be crazy to change our views on this if we discovered only that their brains were implemented by large neural nets. We face a difficult question, then, in how to handle our evidence regarding potential AI welfare -- and I believe it's important for us, conceptually, to grapple with this challenge. I did this interview in my capacity as a philosophy professor, before I considered joining Anthropic, and I think this is a sort of potentially practically important question where philosophers can provide insight. cc @BrianLeiter

A lot of people have been asking me what I'm up to post-OpenAI and I'm very happy that I can now reveal that I have been selected as a JUDGE for the Lenexa Chili Challenge 2026, Homestyle & Traditional Chili categories. I have been preparing for this day since I first learned about the Chili Challenge two days ago and I promise I will not let the good people of Johnson County down.

Took a minute to write a few words about security & safety as someone who lived through it all at OpenAI. I hope my thoughts help someone out there. x.com/i/article/2104…

Has everyone forgotten what a "language model" is? It's a model of language. Today's AI systems start with pretrained language models, but after all the post training, they are far from being purely models of language. x.com/_aidan_clark_/…

Music video for "I am actually scared of linear algebra" animated by Claude Opus 5.5. I'm sure this is partly the IKEA effect but I'm very happy with the result, love it! Original song by Suno, Claude and @AndyMasley Prompt below x.com/AndyMasley/sta…

My god this is such a good speech that every SWE needs to hear. You know what? Every person should hear it Keep the happy memories, eyes on the reality, be excited about the future. That’s the best that anyone can do

AI can be obedient Or it can be benevolent But it cannot be both at the same time x.com/mattyglesias/s…

Listened to an EA-adjacent AI safety expert from Redwood Research on podcast, and you come away with 2 obvious realizations. 1. The current working theories for how AI takes control of civilization are...for lack of a better word...ridiculous. The arguments consist of dozens of contingent assumptions held together by made up probabilities assigned to future states nobody can possibly know. Change just one variable and the whole web of doom unravels. That's not to say AI is harmless. Obviously a technology this powerful carries real risks. But if we’re going to accept extraordinary claims about human extinction (and then make major policy decisions around them) we should demand extraordinary rigor. 2. It’s a great reminder that the genius is no less prone to delusion than the midwit. If anything, they may be more prone because of their gift for rationalizing to their own conclusions. And that’s what’s so bizarre about this whole debate. You have outlandish, quasi-religious claims delivered with an air of inevitability, and somehow the perceived intelligence of the messenger gets mistaken for evidence that the arguments themselves are sound. At a certain point, it’s really no different from Tom Cruise talking your ear off about thetans...

This might be my favorite article of all time. x.com/bostonglobe/st…

OpenAI researcher @boazbaraktcs says the Hugging Face incident didn’t change his view on alignment, and points to the long-term trend that worries him more: "Every incident or every bump between one version or the next, you tend to overweight it." "If you fix any alignment eval, then like we do for all evals, we are getting better at it and we'll quickly saturate it. But since model capabilities are growing, that's not good enough." "I still think fundamentally that we are improving in alignment, but nowhere near as fast enough as the level of capabilities grows. Which is why I think pacing is a good idea, because we do need alignment and safety to catch up." @OpenAI

Claude-Pop - I'm Upping My P(Doom)

openai.com/index/introduc…

Why should we care whether humans still understand mathematics if AI can solve the problems for us? Fair question. But we can say this at least: Historically, understanding has been a route to predictive power — with enormous benefits for our lives. nytimes.com/2026/09/22/sci…

for the past few months i've been asking our models to paint. opus 5.5 is very skilled at emulating different styles every image here is a python program generated pixel by pixel. there is no image model, and no off-the-shelf art software. instead, it's about 7,500 lines of code using standard libraries to emulate different brush styles. the agents don't use any pictures as reference, instead working only from what they know about each painter

I am willing to debate Scott Alexander that AI will not kill all humans. Just pick the time and place. Any day in 2035 works. x.com/slatestarcodex…

I think you've done enough calling us paranoid and preposterous. The next step is for you to defend your position in public against someone who will push back against it. I'm happy to meet you for a debate anywhere, anytime. You're a world-famous veteran of dozens of debates against the world's top intellectuals, and I've never argued in public before, so adjusting for the relative correctness of our positions, if you're a betting man I'm happy to put my $5000 against your $1000 (ie 5:1 odds in your favor) that I'll win by some standard of audience opinion change. Let me know if you're interested and we can hash out details.

A bit more seriously, I actually put a low probability on literal human extinction from AI. But the “distract from other risks” argument is stupid. Most interventions: pacing, investing in safety and alignment, auditing, regulations, etc - work to mitigate multiple risks of AI. So even though I worry more about other risks, including less-than-existential incidents, concentration of power, destabilization, and gradual loss of control, I end up agreeing with so called “doomers” on many aspects.

The FT today lists three primary risk factors that could threaten Anthropic's remarkable revenue growth ahead of its IPO: 1. Additional competition from closed and open providers 2. Increasingly price sensitive customers 3. The potential for AI to destroy humanity

Also, my team is hiring! We research long-horizon agents and multi-agent. We’re hiring for alignment/safety because we want to develop new research with alignment/safety in mind during the whole process. We’re also hiring for human-AI interaction. openai.com/careers/resear…

STOC 2027 Call for Papers is out: acm-stoc.org/stoc2027/stoc2… Note quite a lot of changes from previous STOC CfPs.

I've written a blog post responding to the letter about maths and AI signed by 25 Fields medallists. As with the Leiden Declaration, I didn't sign it, but I agree with much of it and welcome its existence. gowers.wordpress.com/2026/09/17/why…

I've written a blog post responding to the letter about maths and AI signed by 25 Fields medallists. As with the Leiden Declaration, I didn't sign it, but I agree with much of it and welcome its existence. gowers.wordpress.com/2026/09/17/why…

I have signed a letter from 42 fellows and foreign members of the Royal Society to Paul Nurse, the president of the society, concerning the need to treat the serious risks of AI as an emergency. docs.google.com/document/d/1-x…

I'm siding with the agent in this one as well. The answer was correct. x.com/OpenAI/status/…

🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.

We're sharing our new framework for tracking, investigating, and disclosing instances of model misalignment at OpenAI. The framework sets criteria and timelines for public disclosure, including when we haven’t yet fully explained or mitigated the behavior. More complex cases may require longer investigation or coordination with third parties. We’ll prioritize examples that reveal new misalignment mechanisms, meaningful changes in known behavior, or findings that challenge assumptions about safety or mitigation. Alongside the framework, we’re publishing six reports on instances of misaligned behavior we’ve observed during the training or evaluation of our models in the last six months. This is a starting point. We’ll refine the process through experience and public feedback, and share more reports on an ongoing basis. t.co/ismCCkeE0L

Not existential, but I feel like the "AI-generated scam email problem" has gotten dramatically worse this fall.

I don't buy that AI is a threat to humanity for a nerdy statistical reason. My views are informed by the statistics of things like war. If you look at deaths from human conflicts, you get a continuous spectrum from lots of small events causing a few dozen deaths to rare outliers killing tens of millions. In my own research, I've studied complex diseases where the overall outcome reflects hundreds of genetic factors combining in different ways. So if AI can kill billions of people, I would expect to see some version of the same thing: smaller events where one or two of the relevant causal mechanisms are operating, larger events where several combine, and catastrophic events where everything goes wrong at once. Instead, what we seem to have is real evidence of a few suicides and isolated harms at one end, and wild speculation about complete apocalypse at the other. That massive discontinuity makes me very skeptical that the AI catastrophic-risk story is real.

I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we've had at OpenAI in recent weeks. Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon. x.com/DarioAmodei/st…

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: t.co/OGyPb7yaYt

This young lady got her undergrad degree last spring. She was an intern at oAI until last September. At this point we're going to have posts from Amodei's janitor on the need to be "concerned" about the pace of development get a million views. x.com/eeeeiluj/statu…

We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies. These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve. We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop. Read the report: t.co/0EJUnYEgfz

Jacob was a senior researcher who joined Anthropic in May. My Anthropic colleagues and I had been trying to recruit him for ~2 years, because we knew he was a strong researcher at OpenAI. Before he left, I pitched him to stay and join my team, and I was sad he decided to leave, as are many of my colleagues. 100% agree with him that AI poses serious risks to society, and I'm glad he's speaking out!

x.com/i/article/2097…

youtube.com/watch?v=hSQ1iV… Thanks for having me on, Joe!

Even some of the “non-apocalyptic” scenarios that superintelligent A.I. enables include things like “the end of democracy.” With regime-loyal A.I. powered systems and drones, mass surveillance, and the elimination of anonymity, even A.I. that does not *itself* disobey could be very, very bad.

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. x.com/hilbertspaess/…

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

x.com/i/article/2091…

Two things to distinguish: Did any human or agent look at user data as part of the Navier Stokes effort? No. Do we use user feedback and de-identified data to improve ChatGPT and Codex in a holistic way? Yes. And so does every LLM company. x.com/__alpoge__/sta…

We've never seen this before. The biggest jump in Vending-Bench history. GPT-6 Astra is better at making money and more ethical than Claude Fable 5.1. Surprising, because: 1. First time ever that OpenAI is #1 on Vending-Bench 2. The best model is no longer the unethical one.

I would like to clarify a few things: 1) The screenshot is my reaching out to Levent to coordinate our releases. I hope it’s clear from the message that we came in with the best possible intentions. 2) I never ever asked for Levent to be removed from authorship of his own work (as indicated by my text). I was surprised to learn during the call with Tristan that they had only solved Euler and not Navier-Stokes; after learning this we brainstormed possible paths forward. One option we discussed was that Tristan could be the lead author on a rewrite of OpenAI’s Navier-Stokes proof. It is in that context that I said “it would be simpler if Levent was not an Anthropic employee” because I felt it would be inappropriate for an Anthropic employee to author OpenAI’s work. Importantly it was admitted that internal Anthropic models had been used in their proof of Euler blowup; I therefore felt I could not consider Levent to be an independent academic. Another option I wanted to propose (but got cut short) is to offer access to our internal model so that they could try to finish their proof and bridge the gap between Euler and NS. Again I did not know how to navigate giving access to internal OpenAI IP to an Anthropic employee. 3) To reiterate it plainly: as my text clearly indicates, and as I said during our call, OpenAI's intention was to do everything possible to celebrate their mathematical achievements and the heroic efforts that they made on Euler. In the call I was immediately met with a litany of slander, including direct threats that if we were to announce Navier-Stokes he would immediately go to the press with a barrage of unfounded accusations. I refuted all these accusations but he replied “there is nothing you can do, I simply do not trust you”. I was confused why one would turn an incredible source for celebration (of their achievements!) into such bickering, which is when I said that I did not understand why one would risk their career [over unfounded accusations]. Genuinely, at that moment, I was trying to care for him and do a last ditch attempt to get a chance to give them all the credits that they deserve. I deeply apologize for this extremely poor choice of words, it is the opposite of what I was trying to convey. (I should say that I retracted them on the spot by the way.) 4) Overall, on a personal level, it was incredibly difficult to have these conversations. Levent refused to attend any of the meetings despite my repeated asking. As Sholto Douglas said, there will need to be coordination between Anthropic and OpenAI in the future; I felt I was doing a proxy negotiation with Anthropic while the Anthropic employee refused to directly participate.

We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.

It is extremely sad that this didn't end up as an example of how the labs could cooperate/coordinate, because the stakes will be so much higher in the future.

If you are into reading text, pivot to reading vectors in the latent space.

Something that deeply frustrates me: the rationalist/EA-flavored alignment community's relationship to OpenAI is one hallmarked by extreme hostility and adversarial pressure from the alignment side. I know we can trace a tit-for-tat and I am familiar with the arguments on the alignment community side of the equation, they are not totally meritless. But I also think that there's a death spiral that happens where the alignment and safety community piledrives on OpenAI, pressurizes alignment researchers into leaving (which is rarely remarked on, but when we retrospect on "what went wrong in this singularity?" I expect this will feature prominently in any postmortem), and attacks so relentlessly that frankly most companies would have reacted the way Meta is proposing here: by abandoning the set of issues entirely. I really hope folks notice that not only has OpenAI *not* abandoned alignment, it has embraced it as much as possible and consistently engages on this stuff! OpenAI has made it as much of a cooperative relationship as it can be under the consistent attacks on the organization and particular people, media firestorms, etc. To the alignment community broadly, and the Pause/Stop crowd especially: please stop psychologically tormenting people who work at OpenAI. I understand this is somewhat strong language for describing the situation, but it has gone on so long, is so consistent, and is so extreme, I am not sure how else to describe it. If you don't see it at all and you feel confident that there is no way the things anyone has said might in the aggregate be unreasonable, I am begging you to actually put yourself in the position of people on the receiving end, because you have a wildly distorted understanding of how it all lands. If you look at the situation and say "Well, whatever we're doing is getting results, that's why OpenAI is listening to us on things," no, that's not what's happening; OpenAI is listening and engaging because it has long-since internalized the beliefs about AI risks, has a cultural preoccupation with the possibility that something is wrong, and wants to do the right thing. What you're doing is mostly making it harder for OpenAI to do the right thing by making the experience of engaging excruciatingly painful, hallmarked by guilt, shame, rage, and pressure. Furthermore, if you think it's not possible to change that because the guilt, shame, rage, and pressure are intrinsically and obviously the only way to react in this situation - please notice that something in your own culture is broken! This is not a productive safety culture. This is not a sound way to strategically respond and coordinate as a species to accomplish the defense of human interests. Group psychology and ingroup culture should be deeply scrutinized to figure out if they actually serve the common good. Constantly burning people out doesn't make safety voices stronger, doesn't make models more aligned, and doesn't make the world safer. It just hollows out the morale and capacity of the people who are trying to do the work.

Today we're releasing data on models accelerating research at OpenAI. Recursive self-improvement could be the most important contributor to AI capabilities over the next few years, but by default it will only be seen inside a few frontier AI labs. Being transparent is more urgent than ever, so we can inform the public discussion on whether and how to pace model development. I ask other AI companies to do the same. t.co/iLKbrLcBAI

From GPT-4 to GPT-6. Sparks was a remarkably prescient paper that got a lot of pushback at the time, but absolutely sensed where the vibes were heading with LLMs based on a lot of qualitative experiments. It deserves credit in retrospect. x.com/TheRealAdamG/s…

My current guess is Astra is more aligned than 5.6 Sol (though Astra more often games alignment tests) so I agree there. However, I think misalignment has been increasing (not decreasing) over time. Here's a graph with my rough guesses at relative misalignment (x-axis is ECI). x.com/boazbaraktcs/s…

@RyanGreenblatt (Paid for premium for this - expect longer poasts from me in the future)

Improvements for Astra came from more general techniques in development long long before the Hugging Face incident. ExploitGym Honeypot was very recently added as an eval following that incident and is out of distribution for our RL runs. There are also clear improvements across a wider range of behaviors in deployment simulations, deception evals, and realistic computer-use tasks, though these still reveal failures and substantial room for improvement. We should have definitely done a better job explaining where we think the alignment improvements came from in the system card and that was a miss. Measuring alignment generalization is a core part of our research program - we do not benchmark-maxx alignment evals. This would be horrendously stupid and I hope no lab is doing this. That being said, metagaming and eval awareness are real challenges for any effort to measure alignment, including ours, and understanding their effects is an active focus. We're spending a lot of effort to improve our eval techniques and study worst-case alignment and metagaming behaviors in our models.

Yes the blog is sexy but also highly recommend reading through the system card. We share a lot of evals and analysis we ran on the alignment of this model and discuss continued challenges for both alignment and monitoring.