Sayash Kapoor

An update on our evaluations of AI agents conducting open-ended AI research. Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol. We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers. Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget? In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management. But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers. We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!

Read the research note: t.co/yzDzM6tZiy It was great collaborating with and getting feedback from @tilmanbayer, @matilda_orona, @DavidDAfrica, Viet Nguyen, @sethlazar, @HarryCoppock, @random_walker, @sayashk, @steverab, Nicholas Carlini, and @nityndg. Read @pkirgis's thread about the results:

New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems? In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.

@flxcn Paper: sage.cs.princeton.edu/trends/emergin… Post: normaltech.ai/p/new-paper-em… It was great collaborating with @flxcn, @steverab, @PKirgis, @nityndg, @RishiBommasani, @DubMagda, @CUdudec, @AnkaReuel, Elham Tabassi, and @random_walker on the paper!

Do we live in a world where AIs can be wildly superhuman at every single task? Or a world where different tasks have different inherent limits, and AI’s superhuman performance at cybersecurity and math won’t translate to many consequential tasks? This is a key crux that distinguishes the normal technology worldview from the superintelligence worldview, and it has nothing to do with the pace of AI progress — it's about our different assumptions about the world. Here's an intuition pump for why I think it's the second world: different tasks have different fundamental limits. *Category 1: Some tasks are like making a building taller*: there is no ceiling to how well you can do. More compute and better algorithms can lead to wildly superhuman performance. The entire history of computing is the history of building tools that allow us to be “superhuman” at computational tasks. For example, compilers are superhuman assembly code writers, and fuzzing tools are superhuman vulnerability detectors. AI builds on a long line of such improvements: solving math problems and finding exploits in code are two recent examples. *Category 2: Some tasks are like making a ball rounder*: there is only so much room for improvement, and the best AI systems can’t wildly exceed humans. Take persuasion. Some people believe AI can persuade anyone of anything, even convincing military commanders to launch nuclear weapons. In fact, “superpersuasion” is one of the pathways through which many catastrophic risk scenarios unfold. But the AI as Normal Technology thesis is that there are many consequential tasks that are limited by external factors, so such wildly superhuman performance is not possible. This is especially true against humans who use simple computational tools. Tasks like persuasion, or predicting the outcomes of social events (such as elections), fall into this category. Other tasks, like writing, can’t be ranked on a single scale; different people find different writing styles compelling. AI writing also quickly wears thin: even if a user is impressed by AI writing at first glance, after reading thousands of words of the same type of prose, it becomes mundane and irritating. All of these are different from Category 1 tasks like playing chess, where even expert humans can’t do much better than deferring to an AI system at each turn. *Category 3: Some tasks require experiments in the real world*, and the bottleneck isn’t the computational task; it’s the data collection. One example is Moderna’s Covid vaccine: it took two days to design the vaccine, but almost a year to test and authorize, even during a record-breaking vaccine-development effort! This is why a “superintelligence” won’t cure cancer from inside a datacenter: the bottleneck to many of AI’s most sought-after real-world impacts (such as developing cures to diseases) is in the real world, and requires experiments that we won’t (and shouldn’t) hand over to AI. People concerned about AI risks often assume that we live in a world where almost everything fits into Category 1, and we might soon see AIs with wildly superhuman performance at all tasks. I agree it would be scary to live in that world. One reason @random_walker and I are more optimistic about our ability to remain in control is that we don’t think we live in the Category 1 world.

I'm at the Curve this weekend! I'll be talking about RSI on Sunday at 2pm. Also excited to talk about: - What does a big-tent AI safety movement look like? - How can we make progress on policy proposals where there is common ground across the community? - How can we improve our RSI evaluations?

I had way too much fun using Claude to create a reply video. - Extremely fun to be able to convert scattered thoughts and ideas into a song + video - Creating this took <1 hour of my time (in occasionally steering the model in between other things) - But it took ~10 hours of wall clock time for the agent to generate the video + iteratively refine it based on feedback - In terms of AI usage: it required only a small number of Suno credits and 10% weekly usage of a Claude Max subscription (I was using Opus 5.5)

Vídeo

To be clear, while making and watching these videos is fun the first few times, I don't think they will remain compelling for very long. As these videos become easier to create and more pervasive, their novelty will wear out pretty quickly. I predict we'll look back at these the same way we look back at early ChatGPT demos (cute but kind of annoying). Of course, AI will be extremely useful for people who make videos, but since they'll all be competing for the same eyeballs, it's what they do with AI that will differentiate them. And there are already tells in these videos that will make us collectively groan as we see more of these, e.g.: t.co/nGyL7CHXDm

I have been using the new ultrafast mode for Astra 6 for the last couple of days. My main takeaways: 1. Token speed qualitatively changed how I interact with agents. The output speed is 300 tokens/second, 8x the regular speed. Rather than breaking flow and switching tasks, the model moves (just) fast enough that I can keep working on the same task, especially for work that requires frequent inputs from me. I am excited about ultrafast on Sol-class models, which are twice as fast as Astra. 2. Despite being 8x faster at generating tokens, the overall speedup is only 2-4x, since the time consumed for tool calling or running code remains the same. External factors (and not token generation speed) might soon become the main bottleneck for agentic work. (Notably @random_walker predicted both of these changes in: ) 3. One area where speed really helps is computer use. The previous generation of models wasn't reliable enough at computer use to be useful. Astra was reliable enough to be useful, but still slow. Astra with ultrafast resolves both concerns. Unlike tool calls for coding tasks, UIs react in milliseconds, so token speed is the main bottleneck. 4. At 300 tokens/sec, I am already struggling to keep up with the agent. But there is a lot more room for improvement. Cerebras serves GPT-OSS-120B at 3,000 tokens/sec and Sol at 750 tokens/sec. So in a year's time, it is plausible we'll see another 10x increase in speed in models with Astra-level capabilities. At that point, human oversight will become the bottleneck for what people can use computer-use agents for. 5. I can imagine blazing fast computer-use agents becoming extremely common for knowledge work. But I expect their use will be jagged (in the same way as LLM and agent adoption has been): given the oversight challenge, adoption will begin in verifiable tasks where the cost of errors is low (e.g., filling out a form using an agent before reviewing it yourself), and slowly diffuse to hard-to-verify tasks. 6. Given the challenges with human oversight, the key to this diffusion will be developing automated oversight tools. OpenAI's approval/verifier agents have already been useful to me on this front: The verifier understood my intent better than the computer-use agent, and stopped it from filling out a form incorrectly. Still, I expect there is a lot of headroom in improving verifiers for computer use, to make it as reliable in practice as auto mode in Claude Code/Codex is for coding tasks. 7. Ultrafast mode is undeniably resource intensive: I ran through my weekly limit in ~2 hours while filling out a bunch of forms and doing simple coding tasks. But given the pace of efficiency improvements (and the fact that Sol is 5x cheaper than Astra and already decent at computer use), I expect it to rapidly fall. Once fast, cheap models are common, users may choose "sub-frontier" models like Sol and make real tradeoffs between speed and quality.

OpenAI has talked a big game about AI for cyberdefense. But when @HacktronAI broke into their internal repository and reported it, they received a bug bounty of just $6,500 because one of the vectors for the attack was "out of scope". This is atrocious. If we actually want a flood of defenders auditing these systems, companies need to take bounties more seriously. Signing letters isn't enough.

What does it mean to pace the frontier? Over the last month, @random_walker and I have analyzed the loss-of-control incidents at AI companies to understand what technical and policy interventions can improve safety and what companies should do to pace the frontier. The result is a new 13,000 word essay — our most substantial writing on AI safety since AI as Normal Technology. A summary of our arguments: 1) The polarization between the cybersecurity and AI safety communities is counterproductive. The safety community largely sees these incidents as a crisis for alignment, and worries that these incidents will become more damaging as agents become more capable. Cybersecurity practitioners largely see companies failing to take basic security precautions. We offer a middle ground between these communities as a way forward for improving AI safety. 2) We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions. 3) We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other specific risks, such as biorisk and risks from military AI. 4) We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community. 5) Organization governance should be a key tool for pacing the frontier. Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. 6) How should we reason about AI's impact on cybersecurity? It's plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders. 7) How our views have evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity. 8) At the same time, many distinctive claims of AI as Normal Technology have held up. In particular, we think recent incidents support our continuity hypothesis — the behavior of "rogue" agents became apparent and widely publicized while they are still incompetent at causing serious harm or hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework. 9) In short, we’ve tried to synthesize the AI safety and cybersecurity communities' views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.

@random_walker Link to the essay: normaltech.ai/p/the-ai-as-no… Really grateful to @joshua_saxe @snewmanpv @curl_justin @steverab @RodMoshtagi for feedback that dramatically improved the essay.

I’ll be joining the @MTSlive stream in an hour or so to talk about our work on evaluating open-ended AI research (cruxevals.com) and to share some thoughts on the OpenAI HuggingFace incident.

I was curious, so tasked an agent with updating the figure with the latest data. Here’s the update until July; seems like April was the peak so far, though the following months have seen a much higher baseline than before. I’d expect Epoch’s graph to keep going straight up for a while, driven by new companies adopting AI defenses rather than an inexhaustible supply of bugs in companies that have already adopted AI. Still, it’s interesting to see that the post-Mythos rate of big fixes is much higher than the pre-Mythos rate. I’m also curious to understand the reasons for the persistence of bugs. Are the latest bugs are being newly introduced? Or could they just not be found with previous models or levels of compute spending? How much more inference budget could hypothetically have detected these much earlier?

@BenDLaufer Congratulations! So excited about this :)

One of the best parts of working on CRUX has been the diversity of priors/opinions in the coauthor group. I'm glad we had Seth's feedback (and pushback) on our results and interpretations. The paper is stronger as a result. On that note, we are open to adversarial collaboration: if you think we are undereliciting performance and you could improve upon the scaffold, we're happy to get feedback, test new scaffolds, and figure out ways to collaborate.

ICYMI: We plan to continue these evals and are looking to collaborate with AI researchers who have unpublished, open-ended research. If you would like to test frontier agents on your research questions, we'd love to hear from you: forms.gle/LLaSo1AZ3NvAZc… x.com/sayashk/status…

Can AI agents conduct open-ended AI research? Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. Researchers pick hypotheses, decide what evidence is appropriate, and recognize a failing approach. We gave agents research questions from two unpublished papers, six days, and thousands of dollars of API credits and compute. The authors of the original papers then reviewed the AI-generated papers. They unambiguously rejected agents' outputs. t.co/6HTHywjwzZ We call these "shadow evaluations", since the agents are shadowing the original research effort by the authors. Agents were fluent at most *engineering* tasks They conducted serious literature reviews, debugged GPU environments, ran hundreds of experiments, and turned in camera-ready LaTeX without human help. We also found no evidence of reward hacking. If anything, we found the opposite: the agents started with marketable claims and walked them back to negative results as the evidence came in. Neither agent output was close to the bar of a top conference paper Both papers suffered from similar failures: poor judgment about the bar for an AI paper submitted to a top conference, the lack of creative problem solving and ineffective backtracking, poor awareness of resources, and instruction drift. 1) Lack of judgment about the bar for a top conference. The agents had a poor model of the bar for an AI paper submitted to a top conference. We allowed agents to self review their papers. Despite the poor paper quality, their reviews predominantly labeled the papers "weak rejects". 2) Lack of creative problem-solving to address feedback. When they received negative reviews, the agents typically narrowed their hypothesis and claims, rather than working out creative ways to address these concerns. 3) Ineffective backtracking. The agents dropped their most ambitious hypotheses within the first fifteen hours of carrying out the experiment and never changed course afterwards. 4) Poor resource awareness. Both runs ended with over half the API budget unspent. One agent declared itself done seven hours before the deadline, right after its own self-reviewer returned another reject. 5) Instruction drift. They did not follow explicit instructions on minimum exploration time, incorporating feedback for reviews, and on paper length (the outputs exceeded the page limits in both cases). This research design has many limitations Limitations include the small sample size, non-blind reviews, and the reviewers knowing that the work was AI-generated. We also couldn't test Anthropic's strongest model, because Fable 5 is deliberately limited on frontier AI research tasks, so ended up using OpenClaw with Opus 4.8 (extra-high) for our main experiments and Codex with Sol 5.6 (ultra) for a robustness check. But we think the research design is still helpful in assessing AI agents' ability to conduct research, and it is complementary to evaluations on verifiable tasks, as well as blinded reviews of AI outputs. Our results show early evidence that even though agents are proficient on verifiable research tasks, they do not make genuine progress on open-ended ones. It is worth understanding if this is a fundamental limit, or if better models, scaffolds, and more compute could help close it. As the evidence for the gap between open-ended and verifiable tasks firms up, it is also worth understanding how much progress in AI depends on open-ended research rather than hill-climbing on well-specified objectives. In follow-up studies, we are expanding the set of non-public papers we evaluate. If you are an AI researcher with unpublished papers, we would love to collaborate with you on our next shadow evaluation. Expression of interest: t.co/7YcYAIbYka Conducting shadow evaluations involves a lot of researcher degrees of freedom. In many places, our coauthors disagreed with our interpretation of the findings, and we have surfaced those disagreements in the paper. (This is one reason why having a group of coauthors with different priors is important for open-ended research.) We also release the agent logs, one of the AI-generated papers (the other original paper is still not public), and all the code and data, so that others can conduct their own analyses of our results: t.co/K0zvuD1Xpk Finally, we plan to conduct shadow evaluations regularly, and are hiring a senior researcher to help lead these efforts. Apply here: t.co/TPyOFyNxJE I'm grateful for the core team leading this effort: @PKirgis, Andrew Schwartz, @steverab, and @random_walker, and to our collaborators who reviewed AI papers, analyzed agents logs, and gave feedback on the paper: @DavidDAfrica, @KozzyVoudouris, Viet Nguyen, Toby Pilditch, @DubMagda, @HarryCoppock, @CUdudec, @nityndg, Matilda Orona, @tilmanbayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, @hlntnr, @ghadfield, @sethlazar, @snewmanpv, @shostekofsky, @RishiBommasani

How can academics impact AI policy? I had the pleasure of talking to @r_zwetsloot about my PhD experience in AI policy. While the interview is about policy impact, most of the points apply to AI research broadly.

@r_zwetsloot Link to full interview: horizonlaunchpad.substack.com/p/shaping-ai-p…

The open release of Kimi K3 once again shows that our interventions for resilience should be focused on a world where advanced AI is *abundant* rather than scarce. x.com/sayashk/status…

I'm defending my PhD next Tuesday! The talk is titled "The Missing Science of AI Evaluation" and is based on my faculty job talk. I'll talk about my work on AI-based science, agent evaluation, and open model risks. The talk will be livestreamed and is open to all.

More details about the talk: citp.princeton.edu/events/2026/sa…

Can AI agents turn an architect's rough sketch into a floor plan? Not yet. But GPT-5.6 Sol comes surprisingly close. (Note: This eval was a collaboration with my mom @ViniKapoor7, who is a prolific but old-school architect. To this day, she drafts floor plans freehand instead of using software like AutoCAD, and junior architects convert them into computer-readable plans, stored as CAD files.) She shared three of her recent hand drawings, and we asked frontier agents to convert them into CAD files, which she then graded. Models from one generation ago fail badly. GPT-5.5 Pro scored 1/5 or 2/5 on every drawing; her comment on many outputs was that the model "read the drawing totally wrong." GPT-5.6 Sol (ultra) was uniformly the best model, with an average score of about 80%. It read her sketches correctly, recognized furniture, got room dimensions right, and even inferred things like where she had placed furniture (without specific instructions telling it her drawing conventions). Her verdict was that if a junior architect produced this in a work trial, she would hire them. Of course, this doesn't mean AI agents can replace junior architects, just like AI doing well at a bar exam doesn't mean lawyers will be obsolete. Junior architects quickly learn on the job, improve their skills over time, and can take accountability for their outputs. And GPT-5.6 Sol remains far from perfect: there were small errors in each of its outputs. The worst failure AI mode was silently changing the floor plan, and Fable 5 regularly hallucinated and committed this error (though note that this is with a small sample size: we only evaluate it on 3 runs). The results also suggest that models (including Fable 5) continue to fail at some visual reasoning tasks. For example, they often misread dimensions, drew double walls, got staircases wrong, and struggled with offsets and projections from base lines. This is surprising, since AI labs have been collecting data on architecture for months. For instance, Mercor has been hiring architects for data labeling, with CAD expertise as a key criterion, since at least September 2025. On the other hand, GPT-5.6 Sol was impressive not only in its outputs, but also in its calibration. It annotated its output with the assumptions it made, and my mom noted that with clearer instructions or a few examples of similar drafting, it could likely solve even the tasks it currently failed at. I was surprised that there aren’t existing AI benchmarks for carrying out this task, even though there are many benchmarks for working with CAD files (such as 3D reconstruction, editing existing plans, answering questions about plans, and developing text-to-floor-plan pipelines). This is probably because making hand-drawn floor plans is a disappearing skill. Most architects now start working directly with CAD files, and most architecture schools no longer teach hand drawing in as much detail. But this also makes it a very interesting evaluation task. It is out of distribution for most AI models, and requires skills like reading messy handwriting, visual and spatial reasoning, working with CAD software, and judgment about ambiguous or incomplete parts of a sketch. I'll continue to run these small-scale evaluations with new model releases to see how the landscape evolves. If you're an architect who still uses hand drawings and would like to participate, please reach out.

We are hiring a senior researcher for our AI evaluation projects. You will lead open-ended evaluations of frontier AI on challenging tasks. We value demonstrated expertise in leading and executing on projects more than a PhD. We are reviewing applications on a rolling basis. 🧵

CRUX is our project to conduct open-ended, long-horizon evaluations of frontier AI systems on challenging real-world tasks. It pushes frontier AI systems farther than benchmarks can by focusing on a small set of realistic challenges, on which we analyze AI systems' performance deeply. x.com/sayashk/status…

One of the best pieces of advice I got during my PhD was that there are no repeatable paths in academia. This might sound daunting because of the lack of certainty. But once you internalize it, it is actually incredibly freeing! You don't need to participate in career games that don't interest you. You can choose to work on projects that are aligned with *your* values rather than your institution's. And because you're working on things you deeply believe in, there's much more likelihood that you'll do impactful things. Like so many other things I learned, this was advice by @random_walker. I would even extend it beyond academia: there are no repeatable paths when transformative technologies like AI are being deployed; even career paths otherwise considered "safe" will undergo massive structural changes. Might as well lean into it.

Of course, @jasminewsun has written a much more evocative piece about this: jasmi.news/p/2026-advice

Overwhelmed by the support I have received since I announced my faculty appointment yesterday. I'm so grateful for the mentorship and support of @random_walker, who has been the best collaborator and mentor I could have asked for. (We aren't done yet; we have a book to write and a new venture to begin, and I look forward to continuing our collaboration.) Someone told me that getting a faculty appointment requires a minor miracle, given the number of talented applicants for each position, and I'm sure there was a heavy dose of luck in my appointment. That said, I'm very grateful for the support of so many people who wrote letters, strategized, and gave me advice and feedback during the applications and beyond. @percyliang, @PeterHndrsn, @alondra, Matt Salganik, @b_m_stewart, @JessicaHullman, @manoelribeiro, @RishiBommasani, @steverab, Jonathan Mayer, @karthik_r_n, Varun Rao, @ang3linawang, Mona Wang, @SciOrestis, @dsallentess, Serena Booth, Sarah Scheffler, @sethlazar, @ghadfield, and so many others supported me throughout the process. I'm also glad I have had such a strong group of collaborators over the years, without whom none of my work would be possible. Finally, I have received well over 100 expressions of interest in working with me for a PhD. I'm grateful for all of this interest, and look forward to engaging with your materials.

Thrilled to share that I am joining UC Berkeley as an Assistant Professor in the School of Information! I start in Fall 2027, and I am recruiting PhD students this cycle. List me in your application if you're interested in frontier AI evaluation, AI policy, and AI's impacts on institutions such as science, law, and medicine. I'm especially keen to work with students interested not just in high-quality research, but also in communicating it with a broad audience such as by public writing and policy impact. Fill out the form in the next tweet to indicate your interest. As for this coming year, I'm moving to Berkeley this fall to start something new with @RishiBommasani and @random_walker. We'll have much more to share soon.

Form to indicate your interest: t.co/3hvnvhEiW5 A few examples of my past research: - Long-horizon, frontier AI evaluations: t.co/5Lljx7hCwv - AI policy: t.co/LiNsQwV4KT - AI for science: t.co/xkvmafsHOM Examples of public writing on AI's impacts on institutions: - Legal services: t.co/yIdHhbnU6c - Software engineering: t.co/viSFv7rwyH - Science: t.co/to7GY770Ij

Update on our long-horizon AI R&D evals: In April, we launched CRUX, a project to regularly run open-world evaluations. These long, messy, real-world tests of what AI agents can actually do. Our second evaluation is underway, and we ask: AI agents automate AI research? There is a lot of interest in studying AI research automation. But most of the systems built so far follow one of three patterns. 1) keep a human in the loop to guide the agent and course-correct along the way. 2) focus on narrow problems where ground truth is clear and progress is easy to verify, as in AutoResearch. 3) use scaffolds engineered for one specific type of research question, so strong results may say more about the scaffold than about the agent's general research ability. These efforts are helpful, but a lot of AI research is much broader. Success is not immediately clear or verifiable. Researchers need to test and reject promising hypotheses, backtrack, consider new or unconventional approaches, and do a lot more to make progress on answering research questions. In CRUX #2, we are trying to test whether agents can answer novel, open-ended AI research questions. - One major risk in such a task is contamination. We want the agent to have access to the internet and all the tools it needs to solve the task, so we can't use research questions from publicly available papers. At the same time, we want high quality papers to serve as the source of challenging research questions. - To address this, we partnered with AI researchers from UKAISI, UToronto, Princeton, and other institutions who have written high-quality papers that aren’t yet public, so there’s no risk of contamination. - The authors pose open-ended research questions without giving away answers. The agent must produce a NeurIPS-quality paper and a reproducible codebase, which the authors of the papers then review. - We built a general-purpose scaffold on OpenClaw and Opus 4.8. (We would have loved to use Fable 5, but given the filters on AI R&D capabilities, we don't want to confound results.) - Agents get generous resource budgets set in consultation with the original authors, such as access to VMs, GPUs, and any other compute needed to answer the question. They also have $3,000 in API credits per paper. We evaluate them on week-long time horizons to make progress on answering the research question, far more than typical agent evals. - The agent needs to manage its own budget. It can track its spend and stay within its limits, and it can modify its scaffold and reasoning effort as it sees fit. - In addition to the final artifacts, such as the paper's code, we are also evaluating the agent's trajectories in depth. When we announced CRUX, we planned to conduct an open-world eval every month. Given the scope and ambition of this project, we have spent a lot more time making sure we are confident in our setup and results. That said, the early results we have are exciting, and we look forward to sharing them soon.

Nuclear weapons are an anti-analogy for advanced AI x.com/johnnysaks130/…

Link to essay: normaltech.ai/p/agi-is-not-a…

Really strong showing for @nityndg's first first-author paper. - A small uplift study that finds researchers using Codex reproduce papers more than 2x faster. - Updates to CORE-Bench to add OOD tasks, fix errors, and evaluate reliability and efficiency. - Stronger scaffolds, including Claude Code, Codex, and OpenCode.

My favorite part of our recent essay on AI's impact on software engineering is this figure. It succinctly describes the *irreducible complexity* of software predicts how software engineering will evolve. The most impactful AI-based software will adopt this mental model to empower SWEs for decision-making and delivery, while taking on a bigger chunk of execution. @lateinteraction's work (such as on @DSPyOSS) is a perfect early example.

Link to the essay: normaltech.ai/p/why-ai-hasnt…

Many people have written about advanced AI being too restricted, but my fear is we haven’t planned enough for a world where advanced AI will continue to proliferate. This is despite the availability of models like GLM-5.2, despite the "Deepseek moment", and despite consistent evidence that open models lag closed models by a matter of months. What happens in a world where, even with restrictions on US frontier models and compute, advanced AI continues to be broadly available? Non-proliferation is likely to fail; at best, it provides a few months of buffer A common response is to dispute the premise by claiming that the narrow open vs. closed gap is only a result of distillation by Chinese model developers, and if the US ecosystem shut itself off, Chinese competitors won't be able to keep up. Maybe. But the evidence we have so far points the other way: 1) Existing defenses against distillation don't work. OpenAI, Anthropic, and Google already censor their models' reasoning chains to prevent distillation, and Anthropic has prevented Chinese customers from accessing their models. This hasn't led to an increasing gap between closed and open models. 2) Chinese companies have made genuine improvements to their model training stack that are discounted too heavily by US observers. Notably, preventing distillation might systematically push Chinese AI labs to innovate, rather than follow the lead of US labs. 3) AI has also seen the equivalent of the "four-minute mile": once a developer shows what's possible, others can prioritize and quickly improve their models on those capabilities, even when they previously weren't able to. We've seen this with reasoning models (Deepseek R1), coding agents (GLM-5.2), and will likely continue to see it with future advances. 4) There are many improvements that don't require large training runs. Updating scaffolds, finetuning models, using more inference compute can all narrow the gap between the capabilities elicited from US and Chinese models, even if the gap between the base models increases. In other words, even if we ban open models, improving the agent scaffolds could lead to continued improvements in capabilities, such as by the creation of specialized scaffolds for cyberattacks (this is known as the "scaffold overhang"). So we are likely to continue seeing new misuse risks emerge for years. 5) Most importantly, there is no secret sauce for training advanced AI models. The main training techniques for developing and serving frontier models are well understood. Labs' roadmaps can be gleaned based on the roles they (and their third-party providers) are hiring for. The data being collected through these efforts doesn't require new innovations either. If non-proliferation can at most buy us a few months, we should use that time to improve resilience AI policy can't take for granted that US restrictions on models and compute will lead to a worldwide slowdown in the availability of advanced AI. What changes if we plan for a world with widespread access to advanced AI? For one, we need to prioritize resilience. If licensing, export controls, and other restrictions can at most give the government a few months of lead time, that time is best spent giving defenders broad and deep access to AI systems to improve our ability to withstand attacks. This becomes especially important since many of the misuse risks that the US govt. is concerned about, such as cyberattacks, are a result of AI's *absolute* capabilities, rather than *relative* the gap between open and closed models. For example, even if US model developers can maintain a lead compared to foreign developers, this doesn't help unless the lead is used to find and fix cybersecurity vulnerabilities in the meantime, as @joshua_saxe and others have argued. To their credit, OpenAI and Anthropic have both emphasized the importance of improving resilience. There is also widespread consensus within AI policy that resilience is important. But resilience is often still described as a way to improve society's ability to withstand the availability of *US* frontier models, controlled in its pace of diffusion by US companies and the government. This is also reflected in Anthropic's recent letter to US Senators, where they argue that if only we address distillation attacks and enforce export controls better, advanced AI capabilities would not be available broadly. Because of all the reasons mentioned above, I think this is unlikely. This makes investing in resilience that much more important.

There is a lot of justified anger at Anthropic for sandbagging Fable 5 for AI development tasks. But an unanticipated side effect is that third-party evaluators can no longer credibly use the model for evaluations. Case in point: we are in the middle of running *really hard* AI

Anthropic’s move might sound reasonable if you consider their actions as a company chasing superintelligence. But consider that their customers are spending billions of dollars on their services! That is precisely what has led to their recent surge in ARR, popularity, and fund x.com/auyonomous/sta…

Did Google’s AI agents really build an operating system for $916? With @steverab, @RishiBommasani, Andrew Schwartz, @random_walker At Google's developer conference earlier this week, the company launched its latest model, Gemini 3.5 Flash, alongside a new agent app,

@steverab @RishiBommasani @random_walker The full post with links: normaltech.ai/p/did-googles-… The paper on open-world evaluations: cruxevals.com

New essay: Do AI Risks Require Extraordinary Government Intervention? In a recent essay, @DKThomp engages with AI as Normal Technology. He agrees with our thesis about AI's slow labor market impacts, relying on the fact that GDP growth has so far been average, unemployment is

@DKThomp We are grateful to Katy Glenn-Bass for feedback on a draft of the essay and Shira Minsk and for editorial support. Read the essay with links: knightcolumbia.org/blog/do-ai-ris… Link to @DKThomp's essay: derekthompson.org/p/the-fundamen…

many such cases x.com/dwarkesh_sp/st…

From AI as Normal Technology: normaltech.ai/p/ai-as-normal…