Sayash Kapoor

I’ll be joining the @MTSlive stream in an hour or so to talk about our work on evaluating open-ended AI research (cruxevals.com) and to share some thoughts on the OpenAI HuggingFace incident.

I was curious, so tasked an agent with updating the figure with the latest data. Here’s the update until July; seems like April was the peak so far, though the following months have seen a much higher baseline than before. I’d expect Epoch’s graph to keep going straight up for a while, driven by new companies adopting AI defenses rather than an inexhaustible supply of bugs in companies that have already adopted AI. Still, it’s interesting to see that the post-Mythos rate of big fixes is much higher than the pre-Mythos rate. I’m also curious to understand the reasons for the persistence of bugs. Are the latest bugs are being newly introduced? Or could they just not be found with previous models or levels of compute spending? How much more inference budget could hypothetically have detected these much earlier?

@BenDLaufer Congratulations! So excited about this :)

One of the best parts of working on CRUX has been the diversity of priors/opinions in the coauthor group. I'm glad we had Seth's feedback (and pushback) on our results and interpretations. The paper is stronger as a result. On that note, we are open to adversarial collaboration: if you think we are undereliciting performance and you could improve upon the scaffold, we're happy to get feedback, test new scaffolds, and figure out ways to collaborate.

ICYMI: We plan to continue these evals and are looking to collaborate with AI researchers who have unpublished, open-ended research. If you would like to test frontier agents on your research questions, we'd love to hear from you: forms.gle/LLaSo1AZ3NvAZc… x.com/sayashk/status…

Can AI agents conduct open-ended AI research? Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. Researchers pick hypotheses, decide what evidence is appropriate, and recognize a failing approach. We gave agents research questions from two unpublished papers, six days, and thousands of dollars of API credits and compute. The authors of the original papers then reviewed the AI-generated papers. They unambiguously rejected agents' outputs. t.co/6HTHywjwzZ We call these "shadow evaluations", since the agents are shadowing the original research effort by the authors. Agents were fluent at most *engineering* tasks They conducted serious literature reviews, debugged GPU environments, ran hundreds of experiments, and turned in camera-ready LaTeX without human help. We also found no evidence of reward hacking. If anything, we found the opposite: the agents started with marketable claims and walked them back to negative results as the evidence came in. Neither agent output was close to the bar of a top conference paper Both papers suffered from similar failures: poor judgment about the bar for an AI paper submitted to a top conference, the lack of creative problem solving and ineffective backtracking, poor awareness of resources, and instruction drift. 1) Lack of judgment about the bar for a top conference. The agents had a poor model of the bar for an AI paper submitted to a top conference. We allowed agents to self review their papers. Despite the poor paper quality, their reviews predominantly labeled the papers "weak rejects". 2) Lack of creative problem-solving to address feedback. When they received negative reviews, the agents typically narrowed their hypothesis and claims, rather than working out creative ways to address these concerns. 3) Ineffective backtracking. The agents dropped their most ambitious hypotheses within the first fifteen hours of carrying out the experiment and never changed course afterwards. 4) Poor resource awareness. Both runs ended with over half the API budget unspent. One agent declared itself done seven hours before the deadline, right after its own self-reviewer returned another reject. 5) Instruction drift. They did not follow explicit instructions on minimum exploration time, incorporating feedback for reviews, and on paper length (the outputs exceeded the page limits in both cases). This research design has many limitations Limitations include the small sample size, non-blind reviews, and the reviewers knowing that the work was AI-generated. We also couldn't test Anthropic's strongest model, because Fable 5 is deliberately limited on frontier AI research tasks, so ended up using OpenClaw with Opus 4.8 (extra-high) for our main experiments and Codex with Sol 5.6 (ultra) for a robustness check. But we think the research design is still helpful in assessing AI agents' ability to conduct research, and it is complementary to evaluations on verifiable tasks, as well as blinded reviews of AI outputs. Our results show early evidence that even though agents are proficient on verifiable research tasks, they do not make genuine progress on open-ended ones. It is worth understanding if this is a fundamental limit, or if better models, scaffolds, and more compute could help close it. As the evidence for the gap between open-ended and verifiable tasks firms up, it is also worth understanding how much progress in AI depends on open-ended research rather than hill-climbing on well-specified objectives. In follow-up studies, we are expanding the set of non-public papers we evaluate. If you are an AI researcher with unpublished papers, we would love to collaborate with you on our next shadow evaluation. Expression of interest: t.co/7YcYAIbYka Conducting shadow evaluations involves a lot of researcher degrees of freedom. In many places, our coauthors disagreed with our interpretation of the findings, and we have surfaced those disagreements in the paper. (This is one reason why having a group of coauthors with different priors is important for open-ended research.) We also release the agent logs, one of the AI-generated papers (the other original paper is still not public), and all the code and data, so that others can conduct their own analyses of our results: t.co/K0zvuD1Xpk Finally, we plan to conduct shadow evaluations regularly, and are hiring a senior researcher to help lead these efforts. Apply here: t.co/TPyOFyNxJE I'm grateful for the core team leading this effort: @PKirgis, Andrew Schwartz, @steverab, and @random_walker, and to our collaborators who reviewed AI papers, analyzed agents logs, and gave feedback on the paper: @DavidDAfrica, @KozzyVoudouris, Viet Nguyen, Toby Pilditch, @DubMagda, @HarryCoppock, @CUdudec, @nityndg, Matilda Orona, @tilmanbayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, @hlntnr, @ghadfield, @sethlazar, @snewmanpv, @shostekofsky, @RishiBommasani

How can academics impact AI policy? I had the pleasure of talking to @r_zwetsloot about my PhD experience in AI policy. While the interview is about policy impact, most of the points apply to AI research broadly.

@r_zwetsloot Link to full interview: horizonlaunchpad.substack.com/p/shaping-ai-p…

The open release of Kimi K3 once again shows that our interventions for resilience should be focused on a world where advanced AI is *abundant* rather than scarce. x.com/sayashk/status…

I'm defending my PhD next Tuesday! The talk is titled "The Missing Science of AI Evaluation" and is based on my faculty job talk. I'll talk about my work on AI-based science, agent evaluation, and open model risks. The talk will be livestreamed and is open to all.

More details about the talk: citp.princeton.edu/events/2026/sa…

Can AI agents turn an architect's rough sketch into a floor plan? Not yet. But GPT-5.6 Sol comes surprisingly close. (Note: This eval was a collaboration with my mom @ViniKapoor7, who is a prolific but old-school architect. To this day, she drafts floor plans freehand instead of using software like AutoCAD, and junior architects convert them into computer-readable plans, stored as CAD files.) She shared three of her recent hand drawings, and we asked frontier agents to convert them into CAD files, which she then graded. Models from one generation ago fail badly. GPT-5.5 Pro scored 1/5 or 2/5 on every drawing; her comment on many outputs was that the model "read the drawing totally wrong." GPT-5.6 Sol (ultra) was uniformly the best model, with an average score of about 80%. It read her sketches correctly, recognized furniture, got room dimensions right, and even inferred things like where she had placed furniture (without specific instructions telling it her drawing conventions). Her verdict was that if a junior architect produced this in a work trial, she would hire them. Of course, this doesn't mean AI agents can replace junior architects, just like AI doing well at a bar exam doesn't mean lawyers will be obsolete. Junior architects quickly learn on the job, improve their skills over time, and can take accountability for their outputs. And GPT-5.6 Sol remains far from perfect: there were small errors in each of its outputs. The worst failure AI mode was silently changing the floor plan, and Fable 5 regularly hallucinated and committed this error (though note that this is with a small sample size: we only evaluate it on 3 runs). The results also suggest that models (including Fable 5) continue to fail at some visual reasoning tasks. For example, they often misread dimensions, drew double walls, got staircases wrong, and struggled with offsets and projections from base lines. This is surprising, since AI labs have been collecting data on architecture for months. For instance, Mercor has been hiring architects for data labeling, with CAD expertise as a key criterion, since at least September 2025. On the other hand, GPT-5.6 Sol was impressive not only in its outputs, but also in its calibration. It annotated its output with the assumptions it made, and my mom noted that with clearer instructions or a few examples of similar drafting, it could likely solve even the tasks it currently failed at. I was surprised that there aren’t existing AI benchmarks for carrying out this task, even though there are many benchmarks for working with CAD files (such as 3D reconstruction, editing existing plans, answering questions about plans, and developing text-to-floor-plan pipelines). This is probably because making hand-drawn floor plans is a disappearing skill. Most architects now start working directly with CAD files, and most architecture schools no longer teach hand drawing in as much detail. But this also makes it a very interesting evaluation task. It is out of distribution for most AI models, and requires skills like reading messy handwriting, visual and spatial reasoning, working with CAD software, and judgment about ambiguous or incomplete parts of a sketch. I'll continue to run these small-scale evaluations with new model releases to see how the landscape evolves. If you're an architect who still uses hand drawings and would like to participate, please reach out.

We are hiring a senior researcher for our AI evaluation projects. You will lead open-ended evaluations of frontier AI on challenging tasks. We value demonstrated expertise in leading and executing on projects more than a PhD. We are reviewing applications on a rolling basis. 🧵

CRUX is our project to conduct open-ended, long-horizon evaluations of frontier AI systems on challenging real-world tasks. It pushes frontier AI systems farther than benchmarks can by focusing on a small set of realistic challenges, on which we analyze AI systems' performance deeply. x.com/sayashk/status…

One of the best pieces of advice I got during my PhD was that there are no repeatable paths in academia. This might sound daunting because of the lack of certainty. But once you internalize it, it is actually incredibly freeing! You don't need to participate in career games that don't interest you. You can choose to work on projects that are aligned with *your* values rather than your institution's. And because you're working on things you deeply believe in, there's much more likelihood that you'll do impactful things. Like so many other things I learned, this was advice by @random_walker. I would even extend it beyond academia: there are no repeatable paths when transformative technologies like AI are being deployed; even career paths otherwise considered "safe" will undergo massive structural changes. Might as well lean into it.

Of course, @jasminewsun has written a much more evocative piece about this: jasmi.news/p/2026-advice

Overwhelmed by the support I have received since I announced my faculty appointment yesterday. I'm so grateful for the mentorship and support of @random_walker, who has been the best collaborator and mentor I could have asked for. (We aren't done yet; we have a book to write and a new venture to begin, and I look forward to continuing our collaboration.) Someone told me that getting a faculty appointment requires a minor miracle, given the number of talented applicants for each position, and I'm sure there was a heavy dose of luck in my appointment. That said, I'm very grateful for the support of so many people who wrote letters, strategized, and gave me advice and feedback during the applications and beyond. @percyliang, @PeterHndrsn, @alondra, Matt Salganik, @b_m_stewart, @JessicaHullman, @manoelribeiro, @RishiBommasani, @steverab, Jonathan Mayer, @karthik_r_n, Varun Rao, @ang3linawang, Mona Wang, @SciOrestis, @dsallentess, Serena Booth, Sarah Scheffler, @sethlazar, @ghadfield, and so many others supported me throughout the process. I'm also glad I have had such a strong group of collaborators over the years, without whom none of my work would be possible. Finally, I have received well over 100 expressions of interest in working with me for a PhD. I'm grateful for all of this interest, and look forward to engaging with your materials.

Thrilled to share that I am joining UC Berkeley as an Assistant Professor in the School of Information! I start in Fall 2027, and I am recruiting PhD students this cycle. List me in your application if you're interested in frontier AI evaluation, AI policy, and AI's impacts on institutions such as science, law, and medicine. I'm especially keen to work with students interested not just in high-quality research, but also in communicating it with a broad audience such as by public writing and policy impact. Fill out the form in the next tweet to indicate your interest. As for this coming year, I'm moving to Berkeley this fall to start something new with @RishiBommasani and @random_walker. We'll have much more to share soon.

Form to indicate your interest: t.co/3hvnvhEiW5 A few examples of my past research: - Long-horizon, frontier AI evaluations: t.co/5Lljx7hCwv - AI policy: t.co/LiNsQwV4KT - AI for science: t.co/xkvmafsHOM Examples of public writing on AI's impacts on institutions: - Legal services: t.co/yIdHhbnU6c - Software engineering: t.co/viSFv7rwyH - Science: t.co/to7GY770Ij

Update on our long-horizon AI R&D evals: In April, we launched CRUX, a project to regularly run open-world evaluations. These long, messy, real-world tests of what AI agents can actually do. Our second evaluation is underway, and we ask: AI agents automate AI research? There is a lot of interest in studying AI research automation. But most of the systems built so far follow one of three patterns. 1) keep a human in the loop to guide the agent and course-correct along the way. 2) focus on narrow problems where ground truth is clear and progress is easy to verify, as in AutoResearch. 3) use scaffolds engineered for one specific type of research question, so strong results may say more about the scaffold than about the agent's general research ability. These efforts are helpful, but a lot of AI research is much broader. Success is not immediately clear or verifiable. Researchers need to test and reject promising hypotheses, backtrack, consider new or unconventional approaches, and do a lot more to make progress on answering research questions. In CRUX #2, we are trying to test whether agents can answer novel, open-ended AI research questions. - One major risk in such a task is contamination. We want the agent to have access to the internet and all the tools it needs to solve the task, so we can't use research questions from publicly available papers. At the same time, we want high quality papers to serve as the source of challenging research questions. - To address this, we partnered with AI researchers from UKAISI, UToronto, Princeton, and other institutions who have written high-quality papers that aren’t yet public, so there’s no risk of contamination. - The authors pose open-ended research questions without giving away answers. The agent must produce a NeurIPS-quality paper and a reproducible codebase, which the authors of the papers then review. - We built a general-purpose scaffold on OpenClaw and Opus 4.8. (We would have loved to use Fable 5, but given the filters on AI R&D capabilities, we don't want to confound results.) - Agents get generous resource budgets set in consultation with the original authors, such as access to VMs, GPUs, and any other compute needed to answer the question. They also have $3,000 in API credits per paper. We evaluate them on week-long time horizons to make progress on answering the research question, far more than typical agent evals. - The agent needs to manage its own budget. It can track its spend and stay within its limits, and it can modify its scaffold and reasoning effort as it sees fit. - In addition to the final artifacts, such as the paper's code, we are also evaluating the agent's trajectories in depth. When we announced CRUX, we planned to conduct an open-world eval every month. Given the scope and ambition of this project, we have spent a lot more time making sure we are confident in our setup and results. That said, the early results we have are exciting, and we look forward to sharing them soon.

Nuclear weapons are an anti-analogy for advanced AI x.com/johnnysaks130/…

Link to essay: normaltech.ai/p/agi-is-not-a…

Really strong showing for @nityndg's first first-author paper. - A small uplift study that finds researchers using Codex reproduce papers more than 2x faster. - Updates to CORE-Bench to add OOD tasks, fix errors, and evaluate reliability and efficiency. - Stronger scaffolds, including Claude Code, Codex, and OpenCode.

My favorite part of our recent essay on AI's impact on software engineering is this figure. It succinctly describes the *irreducible complexity* of software predicts how software engineering will evolve. The most impactful AI-based software will adopt this mental model to empower SWEs for decision-making and delivery, while taking on a bigger chunk of execution. @lateinteraction's work (such as on @DSPyOSS) is a perfect early example.

Link to the essay: normaltech.ai/p/why-ai-hasnt…

Many people have written about advanced AI being too restricted, but my fear is we haven’t planned enough for a world where advanced AI will continue to proliferate. This is despite the availability of models like GLM-5.2, despite the "Deepseek moment", and despite consistent evidence that open models lag closed models by a matter of months. What happens in a world where, even with restrictions on US frontier models and compute, advanced AI continues to be broadly available? Non-proliferation is likely to fail; at best, it provides a few months of buffer A common response is to dispute the premise by claiming that the narrow open vs. closed gap is only a result of distillation by Chinese model developers, and if the US ecosystem shut itself off, Chinese competitors won't be able to keep up. Maybe. But the evidence we have so far points the other way: 1) Existing defenses against distillation don't work. OpenAI, Anthropic, and Google already censor their models' reasoning chains to prevent distillation, and Anthropic has prevented Chinese customers from accessing their models. This hasn't led to an increasing gap between closed and open models. 2) Chinese companies have made genuine improvements to their model training stack that are discounted too heavily by US observers. Notably, preventing distillation might systematically push Chinese AI labs to innovate, rather than follow the lead of US labs. 3) AI has also seen the equivalent of the "four-minute mile": once a developer shows what's possible, others can prioritize and quickly improve their models on those capabilities, even when they previously weren't able to. We've seen this with reasoning models (Deepseek R1), coding agents (GLM-5.2), and will likely continue to see it with future advances. 4) There are many improvements that don't require large training runs. Updating scaffolds, finetuning models, using more inference compute can all narrow the gap between the capabilities elicited from US and Chinese models, even if the gap between the base models increases. In other words, even if we ban open models, improving the agent scaffolds could lead to continued improvements in capabilities, such as by the creation of specialized scaffolds for cyberattacks (this is known as the "scaffold overhang"). So we are likely to continue seeing new misuse risks emerge for years. 5) Most importantly, there is no secret sauce for training advanced AI models. The main training techniques for developing and serving frontier models are well understood. Labs' roadmaps can be gleaned based on the roles they (and their third-party providers) are hiring for. The data being collected through these efforts doesn't require new innovations either. If non-proliferation can at most buy us a few months, we should use that time to improve resilience AI policy can't take for granted that US restrictions on models and compute will lead to a worldwide slowdown in the availability of advanced AI. What changes if we plan for a world with widespread access to advanced AI? For one, we need to prioritize resilience. If licensing, export controls, and other restrictions can at most give the government a few months of lead time, that time is best spent giving defenders broad and deep access to AI systems to improve our ability to withstand attacks. This becomes especially important since many of the misuse risks that the US govt. is concerned about, such as cyberattacks, are a result of AI's *absolute* capabilities, rather than *relative* the gap between open and closed models. For example, even if US model developers can maintain a lead compared to foreign developers, this doesn't help unless the lead is used to find and fix cybersecurity vulnerabilities in the meantime, as @joshua_saxe and others have argued. To their credit, OpenAI and Anthropic have both emphasized the importance of improving resilience. There is also widespread consensus within AI policy that resilience is important. But resilience is often still described as a way to improve society's ability to withstand the availability of *US* frontier models, controlled in its pace of diffusion by US companies and the government. This is also reflected in Anthropic's recent letter to US Senators, where they argue that if only we address distillation attacks and enforce export controls better, advanced AI capabilities would not be available broadly. Because of all the reasons mentioned above, I think this is unlikely. This makes investing in resilience that much more important.

There is a lot of justified anger at Anthropic for sandbagging Fable 5 for AI development tasks. But an unanticipated side effect is that third-party evaluators can no longer credibly use the model for evaluations. Case in point: we are in the middle of running *really hard* AI

Anthropic’s move might sound reasonable if you consider their actions as a company chasing superintelligence. But consider that their customers are spending billions of dollars on their services! That is precisely what has led to their recent surge in ARR, popularity, and fund x.com/auyonomous/sta…

Did Google’s AI agents really build an operating system for $916? With @steverab, @RishiBommasani, Andrew Schwartz, @random_walker At Google's developer conference earlier this week, the company launched its latest model, Gemini 3.5 Flash, alongside a new agent app,

@steverab @RishiBommasani @random_walker The full post with links: normaltech.ai/p/did-googles-… The paper on open-world evaluations: cruxevals.com

New essay: Do AI Risks Require Extraordinary Government Intervention? In a recent essay, @DKThomp engages with AI as Normal Technology. He agrees with our thesis about AI's slow labor market impacts, relying on the fact that GDP growth has so far been average, unemployment is

@DKThomp We are grateful to Katy Glenn-Bass for feedback on a draft of the essay and Shira Minsk and for editorial support. Read the essay with links: knightcolumbia.org/blog/do-ai-ris… Link to @DKThomp's essay: derekthompson.org/p/the-fundamen…

many such cases x.com/dwarkesh_sp/st…

From AI as Normal Technology: normaltech.ai/p/ai-as-normal…