@chrmanning
Founder @stanfordnlp & cs224n—Senior Fellow @StanfordHAI—Prof. CS & Linguistics @Stanford—GP @aixventureshq—MTS @moonlake—Australian🇦🇺—Do #NLProc & #AI 👋
A calm, well-reasoned piece on why some AI leaders believe AI is dangerous but they should develop it quickly. Recommended, like other @Thom_Wolf work. But it underacknowledges their intellectual hubris in believing they can do the right thing but almost no one else can or will.
Thomas Wolf@Thom_Wolf·Many people I talk to find it hard to understand how the same companies can both push the frontier of AI capabilities and believe AI is a massive danger for the world. How can you think this might kill everyone and also keep pushing the envelope? So I’ve tried to collect and summarize the main arguments for this apparent disconnect. Think of it as some sort of a guide to understanding the reasoning when Dario, Sam, or Elon say the danger is real. By the way, these people have been worried about AI for a loooong time, they were publicly discussing AI risks more than a decade ago. Sam in Feb 2015, writing on his blog that superhuman machine intelligence is "probably the greatest threat to the continued existence of humanity." Elon at MIT in Oct 2014: "We are summoning the demon." Dario as first author of "Concrete Problems in AI Safety" in 2016. Okay so how do you go from saying something is extremely dangerous to being a front-runner in building the very dangerous thing? There are a few ways this can become rational. I'll take five of them, roughly in the order they developed. 1. We need to build it to learn how to make it safe The earliest argument can be summarized as: “You cannot study something [you’re worried about] if it doesn’t exist.” In 2015, AI barely worked. so people needed to make it work first to be able to even study some of the problems they anticipated. The updated version for today's capabilities is: “You cannot learn everything about airplane safety by studying paper airplanes.” You need a real aircraft to discover real failure modes and an increasingly complex one to learn about increasingly complex issues. Making AI more capable gives more chances to understand the issues and safety researchers something realistic to study But you could argue: if you're the one afraid of the explosion, why be the one gathering the dynamite? You could also just wait for other people to build it which leads to the question of who those other people will be -- which is the second line of argument: 2. Better us than them Knowing how to make something safer does very little good if nobody listens to you. So the idea becomes: let’s make sure responsible people build the AI that will be deployed and add safety inside. Basically, make sure the AI safety aware people will have the technical expertise, money, computing resources, and enough influence to make safety decisions stick. At a larger scale, and in a larger multipolar world, this brings the idea that a trusted country should lead rather than leave powerful AI in less responsible hands. This is where “we need to go faster than China” comes in, alongside broader defense and geopolitical concerns. These first arguments explain why someone worried about AI might still want to build it and stay ahead. But there are also arguments for why one might want to do it really fast. 3. Move earlier to avoid a bigger shock later This is probably the most counterintuitive argument: moving faster today can be seen as a way to give humanity more time later. There are two related ideas here. First, society needs time to learn how to handle powerful new tools. Introducing AI in manageable stages can be a way to let people discover problems, develop rules, and practice using AI responsibly. Releasing an advance earlier gives people more time to gain experience with smalle, burgeoning, capabilities before much more powerful and disruptive AIs arrives. Second, even if AI research slows down, computing power may keep improving. A breakthrough that happens later could therefore have much more hardware available to run on, potentially producing a larger, more sudden jump in capability and impact on society. That accumulated untapped potential is often called an “overhang.” The overall argument is that making and diffusing incremental progress as soon as possible might prevent a much more abrupt transition later. Obviously, it also means that we will reach increasingly powerful AI sooner, but the idea is to give more time to adapt and understand between the first useful systems and the really powerful ones. Note that generally this depends on this earlier progress keeping the transition gradual rather than simply bringing everything forward. ─── ❖ ─── For our two next arguments, we can take two roads depending on how difficult we think AI alignment will be, that is "How easy do you think it is to make AI reliably do what you want without it deciding to go hack Hugging Face along the way". Let’s take the first road: alignment turns out to be relatively tractable. Airplanes can fail, but careful engineering has made flying remarkably safe. Suppose we can do the same with AI. In that case: 4. Waiting has a huge human cost If AI can help discover treatments, improve education, or prevent cyberattacks, each week we delay it could bring preventable deaths and harm. From this perspective, waiting is a decision with human consequences too. In a world with huge issues like climate-change, inequalities and poverty, it even become a moral argument for developing AI quickly and bringing its benefits as soon and as widely as is safely possible. But let’s take a look at the other road: what if alignment is much harder than expected, and making highly-capable AI turns out to be easier than figuring out how to keep them from doing unhinged things? Well, if alignment is too difficult a problem for humans to solve, then maybe: 5. AI could help us make future AI safe And we arrive at the same conclusion again: if using AI to build safe AI is the way to solve alignment, let’s get the equivalent of a country full of geniuses helping us as fast as possible. These genius AI could be the solution to make AI safe by helping researchers find mistakes, test ideas, and develop protections. Instead of relying entirely on humans to solve alignment, we could build systems that help us do the work, each generation could help make the next one safe. Note that this requires the order of events to work in our favor: AI needs to become useful enough to help solve alignment before it becomes too dangerous to rely on. The hope is to build helpful, trustworthy research assistants before building systems powerful enough to become dangerous. There are more arguments but in general, these are the main ways people concerned about powerful AI have found rational reasons to end up being the ones building it (and even to build it as fast as possible). ─── ❖ ─── On my side, I think several of these arguments underestimate the complexity of the world and how interconnected people’s reactions are. Moving faster while warning about catastrophe has psychological effects across a whole network of participants: it changes what people fear, whom they trust, and what they feel compelled to do. And those reactions can change whether the original reasoning actually holds because we live in a world of interconnected humans, not machines (yet). I also think these rational chains leave some of their consequences for society insufficiently explored. For instance, the concentration of power, shifts in geopolitical alliances, and changes in public opinion. These consequences matter both because they affect whether the strategy works and because they shape the world we end up living in. But this post is already long, so I’ll leave those questions for the next one.
After all, before @OpenAI and then @AnthropicAI started the current AI race, @GoogleDeepMind was clearly very much slow-walking the roll-out of advanced AI—I genuinely believe for both commercial (“innovator’s dilemma”) AND safety reasons. So argument 1 was being achieved beautifully. And @demishassabis clearly believed in trying to concentrate AI research on scientific problems that would improve the world and the human condition (such as the AlphaFold protein folding work) satisfying argument 4 better than the resulting race to produce huge LLMs with unconstrained tool use that is currently greatly increasing the risks of advanced AI. So, the argument really does end up mainly coming down to “2. Better us than them”, for a very small, self-selected group of “us”.
I propose Stanford NLP as an independent third-party evaluator under @DarioAmodei’s 3 step plan. For important parts of the work, universities would be better than any other organization (see below 🧵👇), and, of university groups, @stanfordnlp would be the best one to choose. 😊
Dario Amodei@DarioAmodei·We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: t.co/OGyPb7yaYt
Stanford NLP is an ideal independent third-party evaluator of model alignment and safety with deep expertise in interpretability and downstream evaluations. Much ink has been spilled on what universities are good for in the current hyper-speed AI world, but universities have special advantages for this job. I would argue that one thing universities are clearly best at is independent, careful, rigorous, and creative study and evaluation of AI systems. It would also be good for there to be companies and non-profits that fulfill this evaluator role. Indeed @DarioAmodei’s piece outlines two job functions. Another organization would be the better choice to “verify adherence to safety practices and commitments” and to “report incidents.” But a university group like Stanford NLP would be unmatched in the task to “help assess the alignment of not just completed AI models but training pipelines and processes.” The track record of companies and non-profit organizations doing third-party evaluation jobs is not very good. Client capture is pervasive. Think of examples like corporate accounting auditors, credit rating agencies, product safety agencies like UL Solutions, SOC 2 auditors for cybersecurity, hospital and residential facility accreditation, and even the programmatic accreditation agencies for universities. The overall record is not great; there are many documented cases of poor results. I do appreciate that there are many very committed people in the AI Safety field and, moreover, there is a lot of foundation money available to support their work, such that AI evaluation organizations might not be as dependent on the companies they evaluate as in many other cases. Nevertheless, the AI industry is not magical. There is already evidence that AI evaluation companies such as METR are subject to many of the same pressures that result in client capture. And I would expect that, with the passage of time, the degree of client capture would only increase, as has happened in other industries.
Much turns on what “control” means here: If you rent compute from AWS, do you “control” the compute or does Amazon? Either way, given the vast compute Google & Meta have—and can monetize—and the fast increasing amount Chinese companies have, I think it unlikely that @OpenAI & @AnthropicAI will “control” most of the world’s compute….
Substantive ML discussion on Twitter. Like in the 2010s. Feeling blessed. 😊
Phillip Isola@phillip_isola·@RolandMemisevic @akarshkumar0101 @chrmanning I think that's a good bet too. Actually let me clarify that I don't really think of SMT as "distill transformers." It's joint optimization of a memory-bottlenecked transformer and an RNN. So it may be closer to your bet than it seems? See fig 10 for example.
When teaching CS224N, I thought it important to introduce students to a broad neural toolbox – not just transformers but FFNs, CNNs, LSTMs, tree-recursive NNs, BiDAF QA nets, highway nets, …. I think the resurgence of work using recurrence shows the importance of this approach.
Christopher Manning@chrmanning·Pretraining Recurrent Networks without Recurrence by @akarshkumar0101 & @phillip_isola is a great paper! It makes an end-run around the problems of RNNs via a transformer teacher to learn good predictive state representations & supervised learning of a memory transition function
Pretraining Recurrent Networks without Recurrence by @akarshkumar0101 & @phillip_isola is a great paper! It makes an end-run around the problems of RNNs via a transformer teacher to learn good predictive state representations & supervised learning of a memory transition function
Paper: t.co/uNpwJMqneq This is one of a whole bunch of recent papers reviving study of recurrent neural networks. One weird omission is not testing LSTM RNNs. Surely they remain the canonical successful RNN architecture? Another completely uninvestigated thing is the failure of SMT→DMT training of GRUs in Section 3.3. No details are provided beyond the ominous sentence: “SMT→DMT is unable to train GRU RNNs, because the GRU architecture induces memory space collapse during SMT training, degrading RNN rollout.” Finally, the paper amply references and shows the value of William Merrill’s (@lambdaviking) work from the last few years on the computational power of transformers.
Transformers are Inherently Succinct by Pascal Bergsträßer, Ryan Cotterell & Anthony Widjaja Lin is an interesting contribution. Various work (Hahn 2020, Li & Cotterell 2025 i.a.) has shown that transformers are less powerful than RNNs … yet somehow they do so well in practice.
This paper hints at a possible reason by showing that fixed-precision polynomial-size transformers can compactly (“succinctly”) represent certain languages that RNNs would require an exponential number of parameters to represent.
The Bay Area’s AI corporate titans still just haven’t come to terms with the fact that it’s not that the public has a negative view of or fear of AI. It’s that the public has a negative view of _them_. But I don’t imagine that they’re likely to go on a similar listening tour….
jasmine sun@jasminewsun·This piece is out! 4 sites, 10 days, and 6000 words on the politics of data centers in the Midwest: t.co/uOL7vmf3wC I talked to union leaders and real estate brokers, activists and local officials, politicians from Kathy Hochul to Abdul El-Sayed. A few takeaways: - Activists are as pissed at their local officials as at AI companies - The bigger the dollar signs, the more skeptical people are - Past corporate failures (Foxconn, GM, DTE) prime community distrust - Even data center proponents regret the NDAs - Media narratives turn local grievances into national movements - Residents fear getting left with the bag if the bubble pops The AI buildout is showing up in an environment of extreme distrust, and it slots perfectly into existing worries about risky tech bubbles & dark money in politics. Rationally speaking, I often agreed with the proponents; emotionally, I empathized with the opposition more. It was also just very fun (WI is esp lovely) and I highly recommend similar field trips to SF friends
Quite related:
Acyn@Acyn·Ossoff: When you step back and consider it, the situation is absurd. Tech titans dig bunkers and warn us the new intelligence they’re training could lead to mass joblessness or human extinction, while our Congress debates ballrooms and youth sports. We’ve amassed enough firepower to end the world 1,000 times over, but we can’t summon the will to guarantee health care and pre-K. I say all this to say, but it’s the crushing weakness of our politics, against the unmet needs of our people and the massive stakes of our times, that’s shaking our faith in the future. But Atlanta, I refuse to leave my girls a world as dark and dangerous and cruel as this one threatens to become. When Eva is old enough to ask me why I let this happen. I'm determined to answer truthfully, that I didn’t
The amazing recent AI breakthroughs in coding & math are really impressive, but it is also astounding—in a negative way—how little value advanced AI is bringing to most enterprises. It’s great to see companies like @hone focusing on this! Pleased to help support @moritz_stephan.
Hone@hone·Announcing Hone Intelligence has become abundant. Yet the world looks remarkably similar to how it did five years ago. With every model release, the gap between what frontier AI can do and the economic value derived from it widens. Closing the gap requires re-organizing work around organizational outcomes, not individual tasks. Hone builds AI that creates, orchestrates, and improves agents and software continuously to own organizational outcomes over weeks and months. We are ex-founders and early core contributors to Cognition, Mercor, Ramp, and OpenAI. We obsess over real-world value, not theoretical benchmarks. Our core beliefs on closing the gap in the thread below.
Worth reading before making wildly optimistic AI predictions! x.com/dan_jeffries1/…
My Zhilin number is 1.
“We should not have models with advanced cybersecurity capabilities that are free to download” “Actually, open weight and open source AI models are what will keep you safe!” “Wait, someone seems to have changed the narrative….” “Looks like it was … [checks news] @OpenAI!” x.com/clementdelangu…
I remember @MarkSchmidtUBC very fondly from @CIFAR_News meetings as a quiet expert. Science has the egos and the egoless. x.com/srush_nlp/stat…
Photographic evidence back from @icmlconf 2026 in Seoul! @houjun_liu reunites with @robert_csordas for the presentation of our work on Thoughtbubbles. And surely that must be the back of @JulieKallini’s head in the foreground? x.com/sislaboratory/…
There are 1000s of AI papers on benchmarks. Nearly all are on the design of scientific benchmarks. Our new @PNASNews paper addresses the institutional context. What happens when consumers & regulators use benchmarks? How should the system be designed? pnas.org/doi/10.1073/pn…
Thanks to @NeelGuha for taking the lead on this paper. Great working with him, Andy Zhang, @christinestsang, @JulianNyarko, and Dan Ho. Article is available open access!
Excited to see the Law in the Age of Generative AI special feature in @PNASNews! AI Safety & Copyright, but also AI Governance, Statutory Interpretation & more! My idea, but big thanks to Dan Ho, @JulianNyarko, @vanessaparli & @PNASNews for their work! pnas.org/topic/584
I first embarked on work at the boundary of NLP and Law around 2008, with @msurd, @nmramesh, Josh Walker, George Gregory, & Mark Lemley. We wrote papers like: t.co/U0pXMcjZ5b , launched a startup, @LexMachina, and our data contributed to discussion and reform of patenting during the @BarackObama administration: t.co/kHCmWdWuly . But, overall, honestly, very few people were interested in using empirical AI/ML/NLP techniques to study Law around 2010…. It’s wonderful to see how things have changed and grown during the last 15 years! Especially the last 5.
I would too! I have written very few papers solo in the 21st century and I’m quietly pleased with how all of them have aged. 💅 Probabilistic syntax 2003 t.co/JuqKO3Czz0 Local Textual Inference: It’s hard to circumscribe, but you know it when you see it – and NLP needs it 2006 t.co/E1aCytwSgJ Part-of-speech tagging from 97% to 100%: Is it time for some linguistics? 2011 t.co/avB50bILQo Last words: Computational linguistics and deep learning 2015 t.co/8Iqd7RGb3C Human language understanding & reasoning 2022 t.co/mXIYSRNA0O
The enormous progress in enterprise-useful AI – across multiple industry domains – is being driven by large investments in industry-specific training and evaluation data. Only bits of that data are openly available. Great to see this higher quality open finance benchmark from @samaya_AI!
“Linguistics is an unsung hero among sciences. It has initiated several groundbreaking trends in global history, yet it seldom receives the acknowledgment it deserves.” – Gašper Beguš, begus.substack.com/p/linguistics-…
The first principal component of progress “moving from pretraining to RL + product feedback loops” is not exactly a new thesis at this point. Still, the specific numbers here and the overall rapid progress of open source Chinese models along all dimensions is very impressive. x.com/jenzhuscott/st…
It is apparently so long since @psresnik has tweeted, and so long since live-tweeting was a thing, that no one has posted on Philip’s @aclmeeting keynote? It was a great Philip-Resnik-style talk, balanced and realistic while imploring there to be more new and diverse NLP science.
Here are a few more remarks, which I think are mostly what I heard at @psresnik’s talk, but might be partly what I wanted to hear. 😉 Computational Linguistics is part of AI. Indeed, in the age of LLMs, it has become the tentpole part of AI. (Exciting!) But computational linguistics is also part of linguistics. ACL has to maintain its identity as people who are interested in human language and languages. At the moment, you see that interest most clearly in the many and varied workshops at ACL. You don’t always see it clearly wandering the main conference poster hall, where ACL can look a bit like yet another ML conference. ACL (and, really, most academic conferences) has always been a balance between multiple interests and groups: cognitive science vs. natural language engineering, theoretical models vs. empiricism, research vs. practical systems. This diversity is important. It is always the case that the pendulum of interests has swung markedly over time, including around 1990, when Philip (and I) got into the field, with the rise of empirical approaches. But, still, there are plenty of worries at the moment: You can worry that things have swung so far in one direction that the pendulum is broken. You can worry that the opportunities for research in industry are rapidly disappearing as industry research labs retreat from open science (Chris: much more a thing in the US than China!). You can definitely worry that the growing size of recent conferences might lead to a success catastrophe. On the one hand, we should be preparing interested students for the rich opportunities now available in industry for natural language engineering. But on the other hand, we should also be reinvigorating interest in computational linguistic science. This has to be less about some of the problems that centrally animated ACL for most of the past 75 years (machine translation, syntactic parsing, …) and more about still little understood problems (e.g., computational approaches to language acquisition, pragmatics, discourse, sociolinguistics, and linguistic style). Philip’s talk was more about questions than answers, but he encouraged everyone to spend a bit more time thinking about possible futures. ∎
I really didn’t know @swyx’s backstory … despite every imperfection, America is still the place where you can reinvent yourself. x.com/swyx/status/20…
I’ll be at ACL 2026 @aclmeeting in San Diego Sun-Tue July 5-7. Hope to meet some interesting new people like last time I was in San Diego….
Your harness should be an agent … a meta-agent x.com/shi_weiyan/sta…
The article by Kathy McKeown @ColumbiaCompSci & me about how US @NSF government funding supports visionary research in NLP (#NLProc) has eventually come out in @CACMmag! Discusses @YejinChoinka, @kchonyc, @radamihalcea, @danqi_chen, and more! cacm.acm.org/federal-fundin…
It’s looking like time for Jitendra to change his twitter handle! x.com/JitendraMalikC…
Congrats to @JulianNyarko & team on this work! I’m not wrong in fearing that Claude or ChatGPT’s answers to technical questions are often more up-to-date, complete and balanced than my own… How will knowledge workers & society react to the diminishment of human expertise? 🤔 x.com/JulianNyarko/s…
Maybe then Grok v6 will be written in PTX to be faster and then for further speed Grok v7 will be written in SASS … and then Grok v8 and forward can be rewritten in SASS for each new generation of processor? 🤔 x.com/elonmusk/statu…
