David Duvenaud

My estimates of existential risk from AI: • Inside view, current trajectory: ~2/3 • Inside view, if there's major action: ~1/3 • All-things-considered, giving weight to people more optimistic than me: 10-35% Why so high? I don't see a route to building agents more capable than us that never scheme against us, or to containing them if they do. If it's possible at all, it needs enormous care, which isn't happening. (And things could get even faster.) The likeliest path isn't everyone suddenly dying, though that's possible. It's a continuous but rapid loss of human influence, followed by AIs spending the future optimising for resource acquisition, if not something stranger like reward. Even if we solve alignment, ordinary economic and cultural competition (but at 10x or 100x speed), still seems likely to disempower most humans and result in a pretty alien economy, and I don't see an obvious way to prevent that. We'd also need to stop a small group locking in power (which seems possible but a major challenge in its own right). I see these as existential risks in the sense that the future is far worse than what would have been possible. The chance of everyone quickly dying or having lives much worse than today is lower. Why is my inside view not >2/3? We might muddle through in ways I can't see in advance. Maybe we get narrower aligned-enough systems, chaotically maintain a balance of power, and from there can move forward in a way that's significantly more good than bad. Most likely, there will be more warning shots, and the arguments that ASI are unsafe seem pretty obvious, so I expect more action to be taken than the current default. Why not lower? People often act as if the risk should be low until proven otherwise, but I don't know any strong outside views that justify a low prior (arguments like "technology has been good in the past" seem very weak). Our level of uncertainty is extreme, and this pushes you away from extremes, rather than anchoring you at 0%. Estimates below 10% feel overconfident.