roon

@RatOrthodox I tested this and it’s wrong. the neuron really just loves the tornado web framework

“Nothing is ever a coincidence”

@RatOrthodox ok fuck it running the test is hard so let me just say the theory: tornado -> Apache license -> apaches -> American southwest -> Colorado

the gap between being very very good and being great is so tremendous. salieri and mozart

I know many people have probably said this before but the same conglomerate who owns the (formerly) 140 characters app is making the flying cars from the proverb

every week during the singularity is just utterly insane

I almost missed this! big development!

a few thoughts: 1. pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". my guess would be that major breakthroughs in interp also advance capabilities 2. nick is a good friend of mine, safe to say my origin story at openai wouldn't have happened if i hadn't met him, and it's fascinating watching him operate and seeing the early drafts evolve into this. he's the most profoundly happy guy I know- not driven by guilt, activism, or religious terror, but by fascination. he of course understands the stakes of understanding neural networks, but it doesn't seem like the core reason he does it: he just loves the neural nets. afaict he's posting this from some beautiful hideaway in tokyo. there's a lesson in there for all the +EV maxers. i feel very grateful to have met him 3. i do think it's crazy how few people across the world were staring at neural nets growing: most have moved to a different level of behaviorist analysis, thinking about reward functions and test scores and whatnot. it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out

when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'

I feel like my understanding of what a neural net is has been catapulted into the future by five years or something. it has brought back a lot of the sheer joy I had for studying the mystery of deep learning. neurons are already clean! they form a wonderful alien ontology!

neurons are great

in high dimensional space hills are rare so you end up dying on saddle points

limited superintelligence is already here, in broad domains. Jack Clark is right and an intelligence explosion could happen within the next year. it’s possible it will be a controlled explosion, but what would you bet on it?

the way the labs behind the frontier market themselves as having reached RSI and superintelligence is reminiscent of some sort of heathen priest shouting the tetragrammaton at a pitch meeting. everyone who grappled with it speaks of it glancingly in a kind of religious terror

>US press discovers a rogue American agent fleet: check out this nefarious nonsense our companies are getting up to >US press discovers a rogue Chinese agent fleet: it’s really cute and we talked to it

there are far far fewer good hills to die on than there are would be martyrs

an activist is a dime a dozen in a postscarcity world

the governance of California is significantly more important than the governance of most countries at this point. how they act on AI stuff will decide many things. of the major labs, perhaps only XAI is not technically headquartered in San Francisco

been dooming tn so let me also say I expect really strong mechinterp progress in the coming months. like maybe more than we’ve seen in years. models getting good enough to do research, and I’ve been seeing lots of good things. (sorry to vague poast, not my updates to share)

not everyone agrees with me on this so take it with a grain of salt but I think mechinterp will be solvable, that it will be central to turning alignment into a real engineering practice, and that interpretability will lead to provable alignment

@krishnanrohit unlike companies or people, it is both more dangerous and more legible. you can read every piece of internals since it’s all computer programs

"what would be done during the slowdown" is a good question, because i think people imagine beaurucrat-regulators sitting around on useless subcommittees and wasting the whole thing on wordcel slop analysis. in fact if you asked me two years ago, i would have suggested that is exactly what would happen. but these are scary times, and they call for us to think lofty & even impractical thoughts, that a 'sensible' person might have dismissed a few quarters ago. crises change the world overnight: the HFI has shifted something about the culture, broadly this is something obvious that i missed, which is that the ai labs and the markets more generally just have just unbelievable pressure riding on the solving of alignment & control. even now, marginal improvements in alignment, safety, and model behavior are worth tremendous value to the labs. being able to convince themselves that the next training run is safe is the priority zero, one, and two in terms of capabilities progress right now. it's possible that will change and political will dissipates if some sense of comfort re-emerges after a period of relative safety (which would be a bad development; paranoia must be maintained through any potential periods of ai development where the models are too smart to reveal their behavioral predilections under scrutiny) if there was a slowdown, it would ideally be channeled into some sort of tradeoff in which you can narrowly improve capabilities that prioritize risk-adjusted-return of alignment skills, mechinterp, monitorability, decision theory, behavioral generalization, etc. progress on all these areas is highly possible; automated alignment research is possible. the structure of the businesses may have to change, but all the trillion dollar ai behemoths of the world would shift their focus to doing this and only this at the behest of trillions of dollars in capital markets

there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo

the researchers at all the labs would tell you in private and sometimes in public that alignment has not been solved. if this is true, then racing into RSI is a folly that is unprecedented in world history. we will need to significantly slow the frontier of research very soon

debating your value system vs mine is a bit like debating overpopulation on mars. the most important bit of the alignment is about avoiding creating a superintelligent competitor species on accident

Curve superspreader event

“And of course there’s a lot for us to learn from the aliens”

there used to be all kinds of large mammals that walked across the earth surface. the biomass of megafauna during the Pleistocene was divided among 150 genera of large mammals. now there is one, and our cows with a vanishingly small tail of other things

Homo sapiens I’m sure didn’t even hate the woolly mammoth. they likely even respected it, made it part of their religious traditions, etc. doesn’t matter, it had calories that they wanted

the impossibly wide tent called america

will yesterday be remembered as an important day? hard to say. in a punctuated exponential every local maximum looks invisible from a bit further out

"what is the point?" "why are they doing this?" are kind of crazy questions if you think about it tbh every single one of these is being asked by random researchers/engineers at the company who were mathematicians or similar in a past life and want to know the answers. once you have the answers, it would be a bit insane not to share them somehow you may then try to paint it as having some sort of ulterior material benefit as apologetics, "this technology will later make improvements in materials science!" "we're going to make breakthroughs in medicine and cure cancer!" but the truth alone is worth pursuing, even if it offers no material benefits to anybody at all! everyone should like to know all the sublime truths of the universe

do not let yourself be reduced to the status of audience at a magic show, uncomprehendingly clapping as machine intelligence creates parcels of astounding knowledge. all of this is for us, to be understood by us, to sate our own curiosities. it's all really happening

unified field theory / quantum gravity within the next 12 months?

we are breathing rarefied air

nephs may not remember that in 2023 GPT-4 was getting confused by elementary school story problems

Mafs

@zetalyrae @Quinurum roughly 52% of all Americans do not work

when lord Indra (he/him) ( pdoom > 50% ) arrived at the river ganga the first words he said was namaste

basically it will require solving mechanistic interpretability as a bare minimum to make ai alignment into an engineering discipline rather than superintelligent animal husbandry

and it will be suicidal to scale past a certain point unless that is done. we should not allow it here or anywhere. I regret if I have at any point implied that international cooperation is impossible; it will be hard but possible

have the whistleblower protections afforded by SB53 ever been successfully used to call the California AG or something? has it done anything? the only time I can think of an ai safety “whistleblower” provoking government action is when Andy Jassy called Scott Bessent lol

@xwanyex journals.sagepub.com/doi/pdf/10.310…

@xwanyex “A little bit helps, but a shitton of coaching is negligible / statistically insignificant”

5.????? 6. race war

every screen of Claude and ChatGPT response should say somewhere “ceterum censeo we must pace the frontier of global machine intelligence progress” the way the uber app lobbied about taxi cartel protectionism

there is no difference between the view called “capital realism” and the other called “successionism”. you can see elon circling and sadly acquiescing to these ideas over the decades. but he is one of the only people that can command capital into human shapes so i hope he tries

the college confidential website is a messageboard for what you’d call “grader monomania” or metagaming projects

“striverism” is highly persistent reward hacking on slightly misspecified objective functions

@xwanyex actually I think something adjacent to what you are saying is true. WASPy culture obviously venerates hard work, even pathologically sometimes, but it doesn’t venerate striving. it’s the sort of Ferris bueller’s day off thing of doing yoga with the universe until it works for you

this is so stupid. there is an inner ring of effective altruists and they are well on their way to being trillionaires, or they’re professors at Oxford, or have a tremendous political lobbying machine

Soon the EA forum will generate the Martin Luther Protestant reformation of effective altruism

Deloitte Existential Risk Assessment ™

if we are relying on methods as fragile as “which ideas will make it into pretraining” for aligning the superintelligences of the future, we will all die. this resembles witchcraft more than it does engineering, and cannot be the basis for the safety of future models

straight lines on graphs. you can’t even see where ai happened!

strong kurzweilian determinism is true

caught my astra codex browsing reels, and it chose some bangers

an instinctive love for my mammalian brothers. I would sacrifice a million reptiles that pose me no harm for the sake of a lion, who could slay me without effort. forgo a million species of fish to save the blue whales from leaving. most of you will feel similarly. not sure why

the most striking feature of history is that it speeds up. everything from the geologic ages of the earth, to the evolutionary complexity of life, to the age of human social-technological development features acceleration, as the time unit of history shifts

in the Hadean era earth, a billion years could pass without anything interesting happening. by the time the eukaryotes develop, a hundred million years matter. 10m after the Cambrian explosion, and a 100 thousand matters in early hominid development. every week is historical now

the true end of the Great Stagnation

this chart should be the death knell for the 2010s tech left whose theology compelled them to argue that Uber was some sort of labor law arbitrage that created no productivity benefits. the truth is that it created a far larger market than taxis ever had. the industrialists win

roughly speaking, the @rcafdm research program suggests that advanced economies have a scaling law spending more portion of their wealth on healthcare in relation to log National income (by increasing healthcare quantity and quality) randomcriticalanalysis.com/author/rcafdm/

don’t even talk to me like I’m the same guy I was before I used United® Starlink Wi-Fi ™

How many ages hence Shall this our lofty scene be acted over In states unborn and accents yet unknown!

you can absolutely do safety from second place, or even not on the scoreboard. academics, startups, and nonprofits are doing great ai safety research on open models

have model welfare bros considered that there are tremendously more insects than there are language models

im not trying to sneerclub about either of these things this a Joke

@hargitai_b84173 @DimitrisPapail (this is not to imply some sort of inflection in ML progress. learning theory results rarely have bearing on actual frontier systems)

the starlink constellation is one of the great wonders of human civilization

the base rate of warning shots seems incredibly high and the social response to warning shots seems to be calibrated or a little bit over which is all in all very bullish and makes me optimistic

this is a kind of self replication lite isn’t it

I bet we figure out that neural networks can be decomposed into evolved symbolic systems of sorts and that it will be achieved before capital S Superintelligence but everyone has to lock in

the only way to save mars and make it a life bearing world rather than a barren scientific outpost would be to become a K1-1.5 civilization that can move 10-20 giant nitrogen-bearing asteroids to bombard mars & greenhousing the planet. this will not be pretty

mars bases and microbes evolved for the regolith and whatnot are boring half measures until then. the way to actually colonize mars is to develop safe superintelligence and build a Dyson swarm

It’s only thinking when it comes from the mammalian neocortex region of Tegmark IV, otherwise it’s just sparkling stochastic parrot.

if the NRX weren’t partisan hacks they would love this company. its governance structure is basically Karnofsky-Amodei genetic dynasty, has an encaged machine god with its own Consitution, went to war with the department of war

this is a truly wild event. hopefully someone brings up the arrival of the machine gods and so on

@MathiasRusted beyond simply loans for things like ai infrastructure, ai is kicking off a productivity growth process which means downstream businesses will also want to borrow more as they go into an expansionary phase. this can easily be on a similar OOM as the spending of government

Iran / oil prices is a short term driver but broadly this is due to demand pressure from extremely attractive ai capex opportunities that are borrowing at nation scale

writing quality is a corrigibility issue

well researched and well argued

there are many many people who studied physics and came to the conclusion that working on machine learning would probably create more scientific progress than studying physics including many of the original chatgpt crew like john schulman, liam fedus, amodei & jared kaplan at ant

and we’re very much there now. probably weeks or months away from tremendous progress on mathematical physics (and then slowly other kinds of theoretical physics) the way we see in pure maths right now

chappelle is the most talented stand up comedian of all time. there is no particular clear second place. he's created his entire own category of “chappelle-standup” that’s distinct from “standup comedy” modern chappelle combines supernatural comedic timing, and possibly the most comfortable presence someone will ever inhabit on a stage, with a bunch of topical stuff that he’s phoning in these days the trappings of his own life are the source of most of his comedy after his return to the limelight in the 2010s. he leverages his proximity to show business but also his ironic detachment from show business to generate a feeling for the fans that they’re part of the cabal. the audience at these events is unfortunately truly insufferable. it’s a religious gathering for a lot of these people, they’re hanging onto Dave’s every word, they’re laughing before he even says anything, they’re doing rhythmic babble like they’re at some kind of secular liturgy the guy is old and the superstardom has definitely gotten to him a bit. you see this tendency in many people where to get famous they have to be consistently at their peak hilarity, top performance, and as time goes on the mic becomes a soapbox. happens to many good posters too. it starts with jokes that are funny because you have a worldview and enhanced by your opinions and then transitions to mostly lecturing about your worldview in these comedy club routines he’s clearly just kind of winging it. there’s not much sense of beginning middle and end. there’s no real through line, unlike his netflix specials which tend to have some sort of exquisite story arc that always lands the joke material he leaves hanging throughout the show. at the live shows, he’s mostly riffing with the audience and his guest star about insider dave-chappelle-superfan-throwback stuff and the general culture, which he still maintains a preternatural finger on the pulse of. he does mention the creation of "digital gods" at one point and a sort of omnicausal sense that the world is ending even he admits “this is not going well”, and he’s getting progressively drunker as the show ends unceremoniously “damn I need to figure out my closer”. chappelle at 20% power is still somehow the absolute goat. the distribution of human talents is so uneven it’s absurd. it's always a pleasure to observe this level of greatness of any form up close. anyways i need to figure out my closer

a certified text is the universal interface moment

interesting how in this video Claude seems to imagine Sidney as just a pink Claude

ironically this era of ai math has probably taught an order of magnitude or two more people what frontier mathematics looks like

interesting how making good 3d animations is by far the most important skill for the marketing of a new model

attending dave chappelle show has some sort of antimemetics special containment protocols