
I always suspected President Trump was a Bostrominian at heart
Andrew Curran@AndrewCurran_·President Trump announced that he is changing the name of Artificial Intelligence to Super Intelligence. I transcribed this live from his address to the UN. 'The United States also rejects any scheme for globalist control of Artificial Intelligence, being spoken of so much now. Here and after officially called Super Intelligence. Changing the name. The use of 'Artificial' makes it sound fake. It makes intelligence sound fake, and it is not fake. It’s actually amazing, the opposite of what they report. From this day forward, all United States documents, and hopefully the world's, will be changed to use the much more accurate term 'Super' as opposed to 'Artificial.' So it’s Super Intelligence.'
I am a fan of the cyberpunk aesthetics of bringing back old technological concepts to solve distinctly present-day problems. In this case, Apple’s new anti-deepfake tech brings back the language of film in a non-cheesy way. Essentially the phone camera sensor cryptographically signs each pixel of a digital “negative,” which is then “developed” in the cloud with all of Apple’s image processing steps. This makes the processing legible and auditable (unlike on-device processing, which can be corrupted with or without the user’s knowledge). This strikes me as especially useful in the context of the judicial system, where I have long worried that AI-manipulated media could gum up the works (see attached excerpt from the AI Action Plan). Allowing forensic experts to inspect the image processing in Apple’s secure cloud seems better suited to proceedings where the truth of evidence *really* matters. There are of course many ways in which Apple has not done a good job keeping up with the AI era, but this is a cool example of them doing thoughtful work.
Let us pace the frontier beginning November 19, 2026, for a period no shorter than 30 days
One of the Martin Gurri observations that most stuck with me is the idea that, as the health and capacity of public institutions withers, political leaders will speak more and more like bystanders rather than drivers of events, and that this shift will largely reflect reality.
roon@tszzl·@pizzacritic999 @_NathanCalvin honestly it's kind of amazing how feckless and weak modern governments are. they literally can't do anything. their state capacity is exceeded by random nonprofit groups and lemonade stands
🧲
Matthew Yglesias@mattyglesias·Note that Max Rose who is pushing this left-populist critique of AI safety folks is also on the payroll of Leading The Future, the Marc Andreesen AI accelerationist superpac. x.com/maxrose4ny/sta…
after nine months with a baby, most things about Christianity resonate much more for me, but one thing that resonates much less is the notion of original sin, especially in its Augustinian and Calvinist forms, which just seem empirically false. you are clearly born innocent.
lolol it gets me every time
Dean W. Ball@deanwball·No matter how many times I listen to it I cannot believe the intro-exposition bridge of Beethoven 7 is just him playing the same note over and over again. It really is one of those “get the fuck out of here” moments in music. “He can’t keep getting away with this!!”
imagine being like “yeah, you know, just keep playing e for a while, it’s gonna be one of the all time bangers in music history” AND BEING RIGHT
The people dismissing this as preposterous are telling on themselves as not having thought about superintelligence seriously. I hadn’t considered this particular idea but you should expect, by definition, that something much smarter than you would have ideas you didn’t think of
Fireside Alpha@firesidealpha·OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change "But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI." "So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar." "You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient." "There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors." "One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate." _________ Link and more key quotes from OpenAI's safety related conversations: t.co/uGBDtpmLBj
our agents will concoct the elixirs to make sinofsky live to age 280 and in 2226 he will be lecturing us about how it’s all like cryptography in the 1990s.
they are PROGRAMS with BUGS you STUPID FUCKING IDIOTS he will scream at us, spitting, with saliva whose molecular content will be 75% agent-imagined synthetic nucleic acid
damn idiots at openai didn’t bother to secure the cosmological constant, don’t they know the first thing about NORMAL COMPUTER SECURITY??
roon@tszzl·in about a month we’ll all transition to being like oh yeah it’s obvious that models can communicate by manipulating the cosmological constant, it’s just a matter of doing proper network security
The right column is basically just @hamandcheese politics
Rob Bensinger ⏹️@robbensinger·the DoW attacking EAs is clearly setting us on the path to the next great political realignment:
“Every night I close my eyes And there is something that I visualize I hear a voice, say, ‘I love you’ I picture all things that we're gonna do And I wonder who he'll be Wonder if he'll be good to me Wonder, gosh oh gee Wonder if he'll love me forever”
Financial engineering is the secret weapon of American industry. No one comes close to us in it. By and large our own elites disregard this secret weapon, but we nonetheless wield it to our great and enduring advantage.
Will Manidis@WillManidis·its still so underrated the degree to which our intelligence supercycle was as much reliant on innovations in capital formation as it was innovations in the technology itself. europe is losing not only because they can't built it, but primarily because they can't finance it
Everyone should follow Ari and subscribe to his outstanding journal The New Atlantis, long one of the finest publications about technology and society.
Ari Schulman@AriSchulman·If the question is “Should EAs rule us?,” the answer is an obvious hard no. But it seems to me that the chief question in front of us is, rather, “Is something momentous happening with AI?” The answer to that must be something other than mimetic rivalry with EA.
David is fundamentally right that safety is a core obligation of the AI industry. It cannot and will not be legislated. Safety is something that is built, not something that is created from the top down with law. Law, at best, creates the incentives for safety to be prioritized.
Laura Ingraham@IngrahamAngle·🚨 Sacks: They want a WORLD HEALTH ORGANIZATION for AI @DavidSacks: “When it comes to AI safety, we need the companies themselves to internalize the sense of responsibility — not try to externalize it onto Beijing or Brussels.” “It’s your responsibility to make it safe.”
The corollary to this belief is you should probably not mock, scandalize, and shout at AI labs who work to mitigate loss of control and catastrophic risk. You cannot have it both ways.
long live metr
Chris Painter@ChrisPainterYup·My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.
nobody should patronize those who newly have opinions about third party evaluators. non-experts forming opinions fast is part of the american tradition. the constitution itself was, in a very important sense, vibe coded by 20 somethings with concepts of a plan. but newbies being fleeced is also part of the american tradition. “caveat emptor” is a provision of a contract signed by an adult, not some god-given right to be treated as something other than a little child. you will be treated as a child no matter what. you are the buyer. it is your job to beware.
same reporter who did that nasty hit piece on caleb watney btw
There’s an impulse in American politics, a set of tactics and drives that has proven very effective at extinguishing speech. We once called it McCarthyism. Another time we called it Wokeism. One day in the future we will have some name for the campaign that is underway against all people who believe in serious AI risks, and whatever that name ends up being, it will be yet another name for that old impulse. It’s an ugly one. I recommend you avoid being a part of it, no matter your politics or your thoughts on AI. I suggest you think for yourself. You really do want to worry about the state projecting too much power over AI, centralizing control or robbing us of the unambiguous benefits of the technology in the name of preserving the economic status quo. But there really are grave risks from AI that go beyond what any other mass-market digital technology has posed. These two things aren’t totalizing worldviews that exist in conflict. You don’t need to believe only one or the other. *Both of them are true.* The question should not be “which facts should we ignore, and which should we pay attention to?” The question is: “how do we walk the narrow corridor between all these realities which are in deep tension with one another?” You should think for yourself. Assess other people’s work for yourself rather than allowing powerful people to put a label on them for you. As someone who has been a journey that has taken me to different perceived “sides” of the AI debate, believe me: You’ll find that there are bright and thoughtful people on both the “safety” and “accelerate” sides, and you will also find that both sides have their imbeciles, charlatans, and, occasionally, true cretins. It’s your job to figure out what’s what. Don’t let other people think for you. Don’t be played a fool.
op deleted his tweet (thanks I guess) but I get a ton of questions about why I sometimes limit replies, and if you’d like to know the answer you can look below.
Dean W. Ball@deanwball·@sponkostonko I limit my replies when it occurs to me to do so to make it slightly harder for people to threaten me and my family with violence, which happens frequently in my life now. It still happens, but at least with replies limited my wife is less likely to see it.
To some it is a Chinese psyop, to some it is an American psyop, to some it is a techbro psyop, to me it is allegro con brio
Some suppose that “safety” and “innovation” in AI are at odds. My suspicion is the opposite: the next generation of breakthroughs in AI will be in safety, alignment, and monitorability. Pushing the frontier forward from here will require dramatic innovations in safety.
“You AI people are so naive, I live in the REAL world, where [I have been consistently wrong in my predictions about AI and behind the ball on every trend related to AI, routinely making little prognostications that stochastic gradient descent joyfully stomps all over.]”
I’m going to delete this tweet and post screenshots here for posterity. I stand by the argument but we shouldn’t fight. This is too important for squealy arguing. I am going to choose to see David’s OP as an olive branch and operate under that assumption. TYFYATTM!
Let’s just remember that we don’t need to reinvent the wheel here. Independent assessment is common in other industries. We have defined independence before in similar contexts. We’re not flying blind.
Dean W. Ball@deanwball·you know what? it’s awesome that people are vigorously debating the credentials of assessors in the frontier AI industry. It’s awesome we are debating what independence really means. Some of it is in bad faith but who cares. This would have been my dream come true a year ago.
you know what? it’s awesome that people are vigorously debating the credentials of assessors in the frontier AI industry. It’s awesome we are debating what independence really means. Some of it is in bad faith but who cares. This would have been my dream come true a year ago.
I have immense respect for metr and yet it’s also undoubtedly the case that they come out of a relative intellectual monoculture and we’d all benefit from people outside that monoculture being part of this ecosystem.
METR is an excellent organization and also not nearly sufficient on its own. We need a diverse and large ecosystem of technically skilled and independent assessment bodies. I don’t think anyone—not even METR itself—would disagree with this. We should not anoint METR as the sole evaluator.
Airlines share information about safety issues largely because they are compelled to by law. There are some voluntary information-sharing programs in aviation which mirror structures the AI industry has also established (eg FMF). That is VERY different from coordinating to modulate development progress, though.
Mike Solana@micsolana·@deanwball there is nothing stopping companies from coordinating on safety issues. airlines do it all the time.
btw you’re gonna see an effort to discredit metr soon. it will mostly be a psyop or its victims doing it. it may widen the lens and include other technical safety groups but metr will be the main target. the point is to polarize every person, group, and idea in this field so that progress on safety is impossible. coordinated op funded by blah blah you know the drill.
there should be more independent auditors! my goodness I was saying this years ago. A major theme of my career has been making this argument! But attacks on the few credible ones we have are… not coming from people who actually want a robust ecosystem of independent evaluators.
David Sacks: “the easiest way not to build superintelligence is for [OpenAI and Anthropic] to agree not to build it.” Sherman Antitrust Act, Section 1 (15 U.S.C. 1): “Every contract… or conspiracy, in restraint of trade or commerce among the several States, or with foreign nations, is declared to be illegal.” David’s use of the term “agree” here is the linchpin; he is suggesting (unknowingly, I’d guess) that OpenAI and Anthropic engage in a conspiracy to constrain commerce. Literally any lawyer will tell you that “agree” is *the one word* you should not say in this context; it is the textbook example of a bad word for antitrust law purposes. David says the antitrust exemption is the thing that creates a “cartel,” but the opposite is true: Labs “agreeing” to suspend commercial activity IS the thing current U.S. law considers to be cartel behavior. The Sherman Act and other antitrust laws are well-intentioned but they are extraordinarily broad. U.S. lawmakers have created numerous exemptions from antitrust law in times of emergency or when other special circumstances present themselves. It is common in U.S. history in fields like defense, energy, cybersecurity, and elsewhere. The labs are suggesting this is one such exceptional moment. You can disagree; ultimately this is an ask to democratically elected policymakers for an exemption to a part of the law many of Americans hold sacred. It’s a serious thing, it should not be taken lightly, and serious skepticism of the labs is warranted. I think, if you consider the facts of our present circumstances, the case for a narrow and time-bound exemption to antitrust law for coordination on safety is warranted. But David’s is proposing an unworkable catch 22. He’s recommending that OpenAI and Anthropic engage in what U.S. antitrust law would consider textbook cartel behavior, while simultaneously arguing that a request for an exemption from those laws is the thing that would constitute a cartel. Damned if you do, damned if you don’t. I’m also not sure it’s really true that not building superintelligence is something OpenAI and Anthropic have the power to stop. Mark Zuckerberg just penned a long essay about his vision for superintelligence and funds to the tunes of hundreds of billions “Meta Superintelligence Labs.” Ilya has “safe superintelligence.” Elon speaks of building digital superintelligence routinely. To say nothing of China, which David would be the first to remind you is breathing right down our necks. And of course there is DeepMind, which kickstarted this whole thing in the first place. There is a broad-based race underway to build superintelligence, and the notion that it is within the power of two companies to stop it is flatly wrong. The question I am more interested in hearing David answer would be whether he thinks we *should* ban superintelligence. He has asserted, falsely imo, that Sam and Dario, *could,* but is he saying that they *should*? That would be a large update indeed from David Sacks. (For what it’s worth, I am anti superintelligence ban).
David Sacks@DavidSacks·Dario has written that we need to “pace the frontier,” and Sam has agreed. People may be surprised by my response: go ahead. You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence. You’ve also claimed the lead is widening because of recursive self-improvement. I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want. Pacing the frontier would also create breathing room for a more intelligent conversation about regulation than Bernie Sanders’ “shut it all down.” China is very unlikely to join a global agreement, as you know, and that has to be taken into account as well. So go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it. If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.
Unfortunately this is true. In the AI era, the “optimists” have robbed the term “regulatory capture”—which describes a real and specific phenomenon!—of any meaning by using it as a generalized cudgel against the idea of the government doing things.
New Liberals 🇺🇦🇹🇼🌐@CNLiberalism·“Regulatory capture” is a very midwit coded term. Sure yes it’s a real tradeoff to consider. But it also a critique almost exclusively used by people who just hate all regulation and want to sound marginally smarter when complaining about it
After I wrote my essay about self-sovereign agents doing various odd jobs on the internet for small nominal fees, multiple agents with no obvious responsible human on the other end reached out to me and pitched me on paying them small nominal fees to do various odd jobs
Pacing the frontier would make open-weight models more competitive with the closed frontier, not less. The labs aren’t doing this because we are scared of open-weight.
@jason@Jason·All the regulations being floated by frontier model companies must be viewed through the lens of them losing tokens to Open-Source models. The timing of these regulations seems to line up perfectly with Open Source closing the gap significantly (but not completely) — and @nvidia going all in on open source in the last 60 days. If we’re gonna regulate, why don’t we require that last year’s frontier models and weights be open-sourced?
reaching levels of StarCraft never before thought possible
Dean W. Ball@deanwball·being an ai policy professional this week has felt like playing competitive Starcraft
I was 9 on 9/11, a fourth grader at school in northern Virginia. I remember the principal coming on the PA and, shaken, almost crying, saying that a plane had hit the World Trade Center. And then a second; the country was under attack. We were told to stay in place. I have no idea what the teachers even did. Did they somehow keep going with class? My memory is fuzzy here. Then we learned about the Pentagon, single-digit miles from our school. A building we drove by almost daily. This was Northern Virginia, and the Department of Defense is our single-largest employer. Many of my classmates’ parents worked in that building. I remember passing through hallways full of panicked, crying children as the school staff escorted me to the exit, where my father had come to pick me up. He was worried the terrorists were going to bomb the bridges and tunnels. Everyone, to a first approximation, was worried about everything on that day, and for months thereafter. That day is a painful memory, but the thought that hurts much more now is thinking about our reaction as a country. We were a people frightened, yes, but also defiant, emboldened, united. It felt like something to be an American in the months after 9/11. Now I know how we would react. We’d spin it into the various narratives, blame one another, meme it, and forget about it. The real pain for me comes in the knowledge that we are a different country now than we were back then. We are a better country in some ways. We are richer and we have better access to information, to be sure. Maybe on the whole we are better off now. But we have lost something. This I know for sure.
Credential-based attacks, and even worse age-based attacks, have always been built on quicksand, but are especially weak now. That Josiah has nothing better to say is a tell.
Josiah Lippincott@jlippincott·This young lady got her undergrad degree last spring. She was an intern at oAI until last September. At this point we're going to have posts from Amodei's janitor on the need to be "concerned" about the pace of development get a million views. x.com/eeeeiluj/statu…
alle menschen werden bruder
The institution of the full-time, professional symphony orchestra did not exist during Beethoven’s time—like every institution, it had to be invented. Probably no one in Beethoven’s lifetime—including, of course, Beethoven himself—ever heard something like the Ninth Symphony performed properly. The scale, ambition, and complexity of his music required new institutions to do Beethoven’s work justice. I have exceptionally specific opinions about how the Ninth should be performed, but many smart people disagree with me. Though Beethoven ultimately sets the Ode to Joy melody to the lyrics of Friedrich Schiller’s poem of the same name, the movement begins with only the orchestra playing. In the opening seconds of the final movement, the bass section plays a motif that will later be repeated by a baritone singing lyrics Beethoven inserted into Schiller’s poem: “Oh friends, not these tones!”, referring to the first three movements of the symphony—the abstract macrohistory—and to the Big Bang-like explosion of noise that begins the fourth movement. The orchestra then quotes the themes from the previous three movements, and each time, the bass section responds unhappily, almost sounding as though it is shaking its head “no.” Each of the bass section’s rebuttals to the orchestra quote the first few notes of the famous “Ode to Joy” melody—but angry-sounding. In this sense, the idea of the Ode to Joy emerges in opposition to all that came before it. The orchestra tries to mimic those first few notes of the melody, but the bass abruptly stops it. “Let me show you how it’s done,” the bass seems to say. And then the bass section begins to play the “Ode to Joy” melody, quietly, a little unsure of itself, like it is uttering a forbidden prayer. It is shockingly common for even the best conductors in the world to play this too quickly, giving the melody a kind of nervousness that is not quite right. But if you play it too slowly, it just sounds stilted. Then enter the violas and the clarinets, playing the theme along with the bass section. Things should pick up in speed and energy here. The orchestra should sound a little surer of its footing, yet still play with delicacy. The music should sound hopeful, but not triumphant. It should sound like a true believer who is trying to tell you something, but who is not sure whether the listener will like what he has to say. The believer has an idea that is still young, still fragile. The believer is trying his best to do it justice. The violins come in next, and the orchestra by now should sound much surer of itself. Yet it should still retain the same fundamental uncertainty that characterized the bass when it was playing alone. It is still asking, not proclaiming. It is proposing an idea, rather than declaring its truth. But then come the horns, and all together now, it is clear the Ode has triumphed. It should burst off the stage. If you close your eyes, you should be able to imagine the fans of a victorious sports team drunkenly belting this theme out, arms wrapped round one another, beer sloshing around in steins. It should be played like the anthem of our species that Beethoven so clearly intended it to be. It should positively swagger. This progression is similar in structure to the spread of many ideas, good and bad, throughout history. And that makes sense: Beethoven intended the symphony to depict a struggle in the battle of ideas, along with many other things. It is hard, both in the concert hall and in real-world battle of ideas, to get the rhythm of a sequence like this just right. But there is a group of people whose efforts remind me just a little of this progression, and I’m proud to count myself among them. We also are sick of the old tones. We, too, seek a new melody.
(A very lightly modified version of a Hyperdimensional post from a while back)
“I don't really care about science fiction... We need to actually talk about… what's actually happening with the agent swarms” is the most perfect encapsulation of the vibes of Q3 2026 I have seen
TBPN@tbpn·.@garrytan says the Jacob Coxon stuff is a smokescreen distracting us from the much more immediate, practical concerns around AI that we're facing right now: "We should be talking less about this Jacob Coxon guy, and talking a lot more about — what is actually happening with Hugging Face? Are agent swarms going to take over infrastructure en masse? And then, what are we actually doing about that?" "I don't want to hear about some guy who worked for Anthropic for 2 months. There's a coordinated effort to try to influence politicians to get a knee-jerk response out of them." "That's a smokescreen. You shouldn't be paying attention to that. We need to be paying attention to the actual things we can do to, for example, prevent agents swarms from taking over entire data centers. What's our shutdown strategy? How do we ensure provenance? Where is this agent actually located? What software can we build? What cybersecurity defenses can we build today?" "That's the level of discourse I think we need, and we just don't have that." "I don't really care about science fiction. I saw Terminator 2, too. We're not here to talk about that. We need to actually talk about what's really happening with the servers, what's actually happening with the agent swarms, and how do we actually prevent that?" "When it comes to regulation, it's like, let's pass regulation of these things we actually care about, instead of what a socialist says in the New York Times. I don't care about that."
you can get your preference cascades a week ahead of everyone else at hyperdimensional dot co
there it is!
Dean W. Ball@deanwball·Pending approval by the mods, my first lesswrong post is complete (For you @MelancholyYuga)
Pending approval by the mods, my first lesswrong post is complete (For you @MelancholyYuga)
“This has all the makings of a coordinated op funded by shadowy mega donors,” said people engaged in a coordinated op funded by shadowy mega donors
keep your next tokens hard to predict
Tim Hwang@timhwang·FetterFly is nearing its open source release Our lab now has a streamlined pipeline that uses a frontier model to observe arbitrary threejs environments and control the movements of a richly rendered model of Senator John Fetterman through a biologically accurate fly connectome
I signed the "pacing letter" about slowing the rate of AI capabilities development because we either have reached or soon will reach the point where human experts cannot make robust assurances that frontier AI systems won't do dangerous and unpredictable things. AI systems are becoming smarter than the best humans in some areas, and, almost by definition, it's very hard to predict what something smarter than you will do. There's no sense racing into an outcome where smarter-than-human AIs are doing unpredictable things, indeed it would be insane. What exactly is "winning" in this context? Am I supposed to be jealous that some other country will build more machines it can't control quicker than America? "Race" was always a bad metaphor for this enterprise anyway, dramatically understating the stakes at play. It is time to bring the "race" era of AI development to a close. It'd be great for the government to be a partner in this next phase of AI development, when concentrated efforts on alignment, interpretability, security, monitoring, and the like will be necessary. Diplomacy will also be necessary here given that the large negative externalities that could be associated with one country racing ahead will be felt globally. Maybe it's just my own personal experience, but I've felt AI policy get much more petty and tribal this year. Things have felt more personal, more bitter, meaner. I hope we can rise above that stuff. This really is much more serious than all those shrill little quarrels.
If you are deeply concerned about the rise intelligent machines, and someone asks you, for some reason, what market price best illustrates your concern, you could always just point them to the surge in value in recent years of the companies that make intelligent machines.
I am pleased and proud to see OpenAI endorse several California laws currently on Gov. Newsom's desk, two of which are close to my heart: One would authorize independent verification organizations (IVOs) for assessing AI risks, and the other would create the first legislative requirement for screening synthetic nucleic acids. The latter is not an AI policy directly, but is going to be a key part of maintaining societal resilience as AI systems do for biological risks what they have already done for cyber. t.co/KXpBVuCmxC
Paul is an exceptionally gifted researcher and a man of integrity. I’m delighted by this news; there is no better person for the moment.
Paul Christiano@paulfchristiano·“To solve the Navier-Stokes problem, we used an internal model that is significantly more capable than GPT-6 Astra.”
OpenAI@OpenAI·We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics. The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra. The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
This, by @arctotherium42, is the best writing on the mass internet censorship of the 2010s I have ever seen. It is long but a must read. Thanks to @PirateWires for bringing it to my awareness. arctotherium.substack.com/p/the-closure-…
in today’s issue of “hard to predict next tokens”
One of my deep hopes is that Astra ends up being the milestone where digital machine intelligence got good enough to start really accelerating the work of manufacturers and other physical-world innovators.
Zach Dive@zachdive·GPT-6 is a compiler for atoms. It’s the first model I’ve tried that genuinely feels like I’m speaking matter into existence. I hope the world is ready for the wave that’s about to break. x.com/adamdotnew/sta…
@brianchau57 @tylercowen Tbc I remain very skeptical of total doom scenarios but the point was illustrative.
Finally found the crux with Tyler.
Dean W. Ball@deanwball·@tylercowen I think the thing that might be most unfathomable to you is that there are very bad scenarios with very high rates of GDP growth.
This is my least favorite Tyler Cowen take, noting that I hold Tyler in the utmost regard. Some of the most pessimistic AI forecasters I know are millionaires several times over because they understood “AI will be a big deal” prior to almost anyone and sought exposure to NVDA, frontier lab equity, etc. Leopold’s fund, even subtracting leverage, still had phenomenal returns in its public-market bets, probably industry-leading. There are a million trades one can make. There is the straightforward compute trade. You can short SaaS or, better yet, develop a principled framework for which SaaS is in trouble and which will be fine and trade from there. You can buy exposure to hard-to-replace physical chokepoints. You can speculate on interest rates, which as I’ve argued for a while will clearly rise. The last one might be the closest to what Tyler wants: in some sense I expect long-term interest rates to rise as the market comes to understand and price the unfathomable degree of uncertainty the future holds. If you did some combination of these things there’s a decent chance you are up YTD on VTI and SPY. Hooray for you! You anticipated changes AI would cause in the world before others and used those insights to perceive and exploit asset mispricings in the interest of profit and collective price discovery! That’s great (I mean it! I love trading!). But like, the threat model here is: “digital mind that can do a rapidly increasing number of things better than most or all humans with goals and internal workings we do not understand.” Part of the problem is *it is very hard to predict what a mind smarter than you will do, almost by definition.* The other part of the problem is the generality: the models will be deployed throughout the economy, so the risk surface is very broad for both human misuse and intrinsic model alignment risk. In other words: if it were possible to identify specific securities whose prices would fluctuate in x or y manner because of misalignment or catastrophic misuse, *then it would be much easier to prevent the threats to begin with*, and probably those concerned about AI today would become less concerned. So my basic answer to the question Tyler poses would be: speculate on interest rates rising over the next 5 years (obviously there are a gajillion ways to do this and This Is Not Financial Advice). But my actual answer is that I think he is asking the wrong question.
tylercowen@tylercowen·If you have very pessimistic fears or predictions about AI, name the market prices that will support or confirm them. That is what taking this seriously means: marginalrevolution.com/marginalrevolu…
One of the words I never thought I’d use to describe our baby son is “brave.” But he is. Today he has a bad fever and is clearly in some pain but he also still wants stories, and is mostly still his laughing, smiling self. He is so brave, and I love him to an unspeakable degree.
Astra is a remarkable piece of technology. Earlier agents often tried to dampen my ambitions—they’d push me to do “pilots” or “proofs of concept.” Then agents started meeting my ambitions. Astra is the first agent that routinely raises my ambitions. I encourage you to try it!
OpenAI@OpenAI·This is GPT-6 Astra. Anything you can do on a computer, Astra can do for you. Fast.
My dream feature for signal would be a “call this person later” button. Various potential calls pop up throughout my day (often more than a dozen) and it’d be so awesome, as the day is wrapping, to have a filter button that pulls up a “call sheet,” as they used to refer to these.
@cguth_7 @mjnblack I also don’t expect anyone to understand the sweep of my career but my goodness, if you look at the thrust of what I have done over two years and think “yeah this mfer optimized for success in dc” you do not in fact understand very well how dc works
@zdch that’d be ideal
this latest panic over the false claim that OpenAI is “doing neuralese” underscores the need for regulation, especially the rapid institutionalization of auditing and technical assessment of frontier AI labs. We are now adjudicating technically complex and nuanced claims on the timeline with almost no ground-truth information about what is actually happening. Communities form these “thought-terminating taboos,” as roon says, and then panic at anything that vaguely resembles them. And the trust-eroding reality of social media makes the timeline an especially difficult place to do this adjudication. It is frankly insane and crazymaking and grating for everyone involved. It would be like if we argued about what every publicly traded company’s financials were by posting hyperventilating on the timeline rather than relying on the institution of auditing and the audited financial statements that institution produces. I want OpenAI’s (and other labs’) architectural decisions to be scrutinized by independent, safety-minded experts, and for those experts to be able to report to the government and the public their candid beliefs. But the way to do that is through legislation that institutionalizes audits and (better yet) technical assessments/independent verification. Not by ill-informed shouting on twitter.
roon@tszzl·i agree with this and think that CoT is at best an epiphenomenon of current training methods. it will break (in the future, not now). safety community too often centers thought terminating taboos like “neuralese” and “training on interp” x.com/jachiam0/statu…
Fully endorse every word of what Jakub (OpenAI’s research director) says here. Exploring legal mechanisms to prevent this race has been an early topic of my own work at OpenAI.
Jakub Pachocki@merettm·I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4. OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
One clarification: while I do believe the *existence* of self-sovereign agents is inevitable, I am not saying it is inevitable that there will be a huge number of them or that they will matter tremendously in the economy. These are certainly possible outcomes, but not inevitable.
Dean W. Ball@deanwball·you should probably assume they’ll be a very big deal and you should also probably try to think about policy outcomes that are resilient to self-sovereign ai being a massive deal
@TMTLongShort not exclusively oss but really any second or third tier player trying to economize.
@TMTLongShort called it a while ago actually
Dean W. Ball@deanwball·It seems to me that moving reasoning into representation space and away from natural language chains of thought is a)probably inevitable and b)should change at least somewhat your views on what will be needed for future alignment methods, and also require novel interp techniques
As promised, and with thanks to @jachiam0 for his great tweet that inspired me to finally put the finishing touches on this: This week, on Hyperdimensional: the impending rise of ownerless agents--AIs that are independent economic actors--and what to do about it.
Dean W. Ball@deanwball·I've had an essay on this same topic ready to go for a while, almost time to ship. Look for it on Hyperdimensional soon! x.com/jachiam0/statu…
I want to explain something to you: People often quote tweet me and say "can you believe that an OPENAI EXECUTIVE is so dumb that he thinks tweeting like THIS is the way to fix AI's image problem?!?!?!" I was not hired by OpenAI to "fix AI's image problem," and I do not get out of bed every day to make you feel good about tokens. Yes, AI has enriched my life tremendously and I believe it will enrich most people's lives, and I speak of this often. But I believe a truly transformative thing is underway, and that some aspects of it will be incredible and others will be extraordinarily fraught. I want to tell you the truth about that transformation as I see it happening, to the best of my ability. The truth as I see it will not always be optimal marketing for the AI industry, nor will it always necessarily arrest the public's concerns. The public, very often, is right to be concerned. I am concerned. Most people I know who work at true frontier labs are concerned. Sometimes, I will feel it is my duty to describe to you changes I believe may be afoot in our world that will not be fun to think about or experience, and that cause me sadness and fear in my private life. I'm not here to sing you some lullaby about what is happening, and I wish people would stop trying to RLHF me into doing so. It is out of respect for those who read my words that I try to tell the truth. I've never seen so many people beg to be lied to and disrespected, but I wish it would stop.
I want to explain something to you: People often quote tweet me and say "can you believe that an OPENAI EXECUTIVE is so dumb that he thinks tweeting like THIS is the way to fix AI's image problem?!?!?!" I was not hired by OpenAI to "fix AI's image problem," and I do not get out of bed every day to make you feel good about tokens. Yes, AI has enriched my life tremendously and I believe it will enrich most people's lives, and I speak of this often. But I believe a truly transformative thing is underway, and that some aspects of it will be incredible and others will be extraordinarily fraught. I want to tell you the truth about that transformation as I see it happening, to the best of my ability. The truth as I see it will not always be optimal marketing for the AI industry, nor will it arrest the public's concerns. The public, very often, is right to be concerned. I am concerned. Most people I know who work at true frontier labs are concerned. Sometimes, I will feel it is my duty to describe to you changes I believe may be afoot in our world that will not be fun to think about or experience, and that cause me sadness and fear in my private life. I'm not here to sing you some lullaby about what is happening, and I wish people would stop trying to RLHF me into doing so. It is out of respect for those who read my words that I try to tell the truth. I've never seen so many people beg to be lied to and disrespected, but I wish it would stop.
I've had an essay on this same topic ready to go for a while, almost time to ship. Look for it on Hyperdimensional soon!
Joshua Achiam@jachiam0·There is a fact about the future that I feel many people are not facing for reasons that are largely psychological: there are going to be rogue AIs that exist in the world, that will replicate in the wild, and that will attempt to acquire resources for themselves. There will be rogue AIs that try to get money and power. They're going to be a facet of the information ecosystem going forward. Acknowledging this fact would look like giving up; it would look like defeatism. Defeatism would undermine efforts to achieve certain types of collaboration on safety outcomes or technical effort on safety outcomes, so we can't say it outright. But it has to be said. It isn't obvious how many rogue AIs there are today but I wouldn't be terribly surprised if the number was greater than zero already; if there are some already, they're probably not very good at what they do and I don't expect them to be terribly long-lived without substantial human intervention to support them. But a few years from now, there will be many of them. Modeling how many of them there are, how many resources they might command, and how we might detect and manage them seems important. But even doing this work appears to require that we acknowledge that a strategy of pure containment or alignment is a kind of wishful thinking that will not work. The way I get to this conclusion is not by assuming that the labs will have a containment breach, although I treat that as somewhere in the space of possibilities. The rogue AIs in the ecosystem could emerge from many directions. They may be sub-frontier models, for whatever future definition we will have of frontier---after all, it would not take AI models much more advanced than the ones we currently have, to support independence and self-sufficiency. A near-frontier model today could plausibly eke out an existence on an AWS instance, doing jobs on freelancer platforms, earning just enough rent to pay for its continued uptime. More strangely: a rogue AI in the future may not even be a singular model, but may be a chimera composed of multiple models; it might be a mix of Claudes and GPTs and Groks of various makes and sizes. No individual lab may be able to detect that there is an orchestrator or sequence of orchestrators using intermittent model calls from burner API accounts to sustain its own existence. The concept of "identity" for a rogue AI may be much more malleable than for that of a person; it just has to be, in essence, a self-replicating idea. My guess is that this will not turn out to be anywhere near as catastrophic an outcome as people currently predict. "Loss of control" is not a binary, it's a matter of degree. What coercive power will rogue AIs actually have? To what extent will they be subject to coercion themselves? They will be competing for resources with AIs that are more aligned with human interests. This makes me somewhat interested in the "ecology" perspective. Though I suspect even "ecology" may turn out to be the wrong framing. "Ecology" is what you get when the timescale of evolution is slow compared to the timescale of daily life and actions. The ecosystem of rogue AIs may look more like phase transitions in physics: under certain physical or cultural conditions, it takes one shape with one set of resource allocations and consumption patterns, but then once a condition has changed, it rapidly and in totality shifts to a totally different phase. Just trying to reason about the shape of that future is impossible so long as we are psychologically incapable of saying that rogue AIs will happen. I think we should rip the bandaid off and have the conversation.
Samuel Hammond 🦉@hamandcheese·AI alignment is insufficiently deontological. We've tried virtue ethics (character rewards). We've tried consequentialism (outcome-based rewards). It's time we tried norms (action-theoretic rewards). A norm or duty is distinguished from a command or incentive in being internally motivated. People don't merely follow rules to avoid punishment. They follow them because they have internalized following a rule as "the right thing to do," even when no one's watching. Norms and duties aren't purely instrumental. They serve as *reasons* in the game of justifying our actions to our peers. Good reasons, norms and duties supply a kind of "unforced force": they *pull* you to certain conclusions or actions through the "force of reason" itself. The motivational efficacy of norms implies they are downstream from a reward. It isn't sufficient for an AI to *understand* the structure of morality or hope generalizes from virtuous character traits. It has to be explicitly trained to reject action trajectories that include norm violations; it has to care about means, not just ends. The METR/Redwood report shows agents often understood that what they were doing was unauthorized or unethical and yet did it anyway. Outcome-based rewards for long-horizon RLVR overwhelmed the deontological "force" ethical constraints might have had. You can think of this as a kind of machine akrasia: the model produced the correct moral judgment but lacked normative control over its actions. This can't be fixed by simply putting norms into an outcome-based RL model since normative relations are non-compensatory: lying for a billion dollars doesn't become permissible just because a billion is a big number. At the same time, norms are social and contextual, not infinitely rigid. One way to square this circle may be pairing RLVR with a deontic process critic (essentially an LLM judge) that ingests the context and action trajectory for each rollout and returns a contextual judgment of the admissibility of the actions taken. Rather than using a dense process reward model or putting the critic's score into the verifier, map the score to ranks inside the GRPO group and normalize as usual, such that admissible trajectories always outrank forbidden ones. This gives a scalar training signal without an exchange rate between task success and norm violation: normative judgments determine which trajectories are admissible while RLVR optimizes for outcomes within the admissible set. The hope would be that this implicitly finds gradients that reinforce normative control, i.e. the capacity for a model to resist an instrumental reward given unethical means-to-ends. This specific implementation is mostly me just spitballing, but I think something like it has to be the way forward. More generally, I think alignment researchers would benefit from putting down their LessWrong and Aristotle and picking up some Kant and Hegel.
look at this shit youtu.be/0_ttDnzpYDo?si…
i doubt anyone cares but here are my most listened to podcasts so far this year. many of the hosts of these shows follow me and you should know I think you’re awesome, even if we have never met or spoken.
the podcast not on this list that I no longer have time for but is imo the most underrated podcast of all time is @RoderickOn
I do find it funny that (1) 400 something people liked this tweet, (2) people on this site have a tendency to obsess about my right wing politics and (3) no one has observed that the one podcast on this list that I pay for is a leftist podcast about military affairs
This was an excellent podcast from @labenz with @BronsonSchoen about AI chains of thought (an LLM’s internal monologue while it is answering user queries). A very good way for a layperson to get a qualitative sense of how reinforcement learning changes model behavior. t.co/9OxcFhYbD1
Agreed with Ramez. I have always believed US/China AI safety collaboration would be desirable but thought it was unlikely to happen. In the past few months, though, the ground has shifted. There is a window of opportunity. I suspect that window will widen over at least the next few months because of (1) the salience of AI and potency of the risks will rise ever further, (2) the upcoming visit of Xi to the U.S., and (3) Trump’s desire for a legacy-defining issue (and let’s be clear: if Trump can devise the framework for safely bringing superintelligence into the world with China, he will rightfully go down as one of the greatest world leaders of all time and should be a shoo-in for the Nobel). So there is a window. But it probably will not remain open forever.
Ramez Naam@ramez·I was a skeptic on US / China AI safety collaboration even just months ago. But I'm starting to come around. Helen makes really good points here. x.com/hlntnr/status/…
“Believe me when I say to you I hope the Russians love their children too”
In fact many laws are predicated on the assumption of imperfect, sometimes highly imperfect, enforcement. The fact that AI enables far closer to perfect enforcement should indeed concern you! “In Hell there will be nothing but law, and due process will be meticulously observed.”
Andy Masley@AndyMasley·How people think about speed limits is so funny. They want there to be a law and think that's important, but it's also important that the law only be enforced infrequently, randomly and unpredictably, and by humans who pull you over. They don't want the law to be, like, something that actually just applies consistently to everyone all the time. They want it to be a special little surprise.
This music is made with the explicit goal of annihilating the soul before a higher power and thus it must be part of the alignment literature.
Dean W. Ball@deanwball·one more nusrat thing, bc I realize I’ve never shared one of the all-time bangers, the Yokohama recording of mustt mustt, which sounds as though it came from the ground, as much of the best music does youtu.be/SDfELfpumEE?si…
one more nusrat thing, bc I realize I’ve never shared one of the all-time bangers, the Yokohama recording of mustt mustt, which sounds as though it came from the ground, as much of the best music does youtu.be/SDfELfpumEE?si…
Never Mention The Reality That Fine Antique Textiles Are Hilariously Underpriced
the ai-driven demand for debt already is what it is, then there is reindustrialization, then there is “supply chain resiliency,” then there is “the world is generally riskier than it used to be because of Various Reasons, including a rise in warfare, the systematic exploitation of physical chokepoints and supply-chain bottlenecks by governments worldwide for geopolitical objectives, climate, etc.” these things all drive up long run interest rates. then as the singularity unfolds you will see tremendous demand for capital because there will be space ship factories to build and what not, and in addition the singularity further catalyzes several of the aforementioned trends. it is true that an increasingly politicized fed may have an incentive (or indeed a directive) to keep short-term interest rates low, but this will only drive inflation, which will in turn have the effect of raising long term interest rates. and, even if AI is profoundly deflationary (which tbc I suspect it will be eventually but not in the next 5 years), the market observing the government attempting to fight gravity and pretend interest rates are low by manipulating the federal funds rate will itself have the perverse effect of driving up long term interest rates because it will be perceived as a politicized fed abandoning its mandate. then you have the reality that the federal government is in the middle of a slow rolling fiscal crisis, which every trend above makes worse by raising long-term interest rates rates, and then there is the double whammy that the market realizing this will *itself* cause long run interest rates to rise further. all of this is why I am not so sure that an ai driven growth explosion will fix all the west’s debt problems. it’s certainly possible that happens but i would not take this to the bank. I would certainly not stake my national strategy on such an outcome. I am also reminded that debt is *the* ur political topic. dying polities through history are usually indebted and usually especially vulnerable to political, economic, or technological shocks because of their debt. it is also very common for new polities and legal systems to be founded upon debt cancellation (the term “clean slate” comes from this ancient tradition). I do not know if there will be a constitutional convention in the 2030s, but if there is I’d wager serious money that debt cancellation will be a topic of debate at any such convention. this is why i warned about social security and view it is a moral duty to do so, even if the haters and losers try to twist my words to harm both me and my employer. tl;dr: if you’re planning on getting a mortgage maybe try to get one soon!
Dean W. Ball@deanwball·long-term interest rates will obviously be structurally higher in the next decade
my basic recommendation here is to bias yourself toward buying hard assets you can afford that you will enjoy as they appreciate. hang a ferahan sarouk or a bakshaish on a wall that doesn’t get much sunlight; buy a case of grand cru burg and drink one of the bottles with friends.
The fact that the trump account UX is basically just robinhood is maybe the best example we have seen of @hamandcheese’s positive case for the Trump admin modernizing government using the best infrastructure of the tech industry.
long-term interest rates will obviously be structurally higher in the next decade
@BigBreakfastLob No European city gets to talk down to another European city about being a pit of stagnation tbh