The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Interesting and good breakdown.

I place much higher odds on the "death due to takeover" for a pretty specific reason. We seem to have an excellent takeover mechanism in place which kills all or most of us: nukes. We have a gun pointed at our collective heads, and it's deadlier to humans than AGIs.

Igniting a nuclear exchange and having just enough working robots, power sources, and industrial resources (factories, etc) to rebuild seems like a pretty viable route to fast takeover.

Igniting that exchange could be done via software intrusion or perhaps more easily by spoofing human communications to launch. This is basically the only use of deepfakes that really concerns me, but it concerns me a lot.

This becomes increasingly concerning to me in a multipolar scenario, which in turn seems all too likely at this point. Then every misaligned AGI is incentivized to take over as quickly as possible. A risky plan becomes more appealing if you have to worry that another AGI with different goals may launch their coup at any point.

This logic also applies if we solve alignment and have intent-aligned AGIs in a multipolar scenario with different human masters. I guess it also favors everyone who can, getting themselves or a backup to a safe location.

This is also conditional on progress in robotics vs. timelines; even with short timelines it seems like robotics will probably be far enough along for clumsy robots to build better robots. But here my knowledge of robotics, EMP effects, etc, fails. It does seem like a nontrivial chance that triggering a nuclear exchange is the easiest/fastest route to takeover.

One very well-informed individual told me nobody knows if any humans would survive a nuclear winter; but that's probably less important than whether the "winning" AGI/human wanted them to survive.

The number of likely deaths given takeover seems higher than your estimate if that logic about nukes as a route to takeover mostly goes through.

But the question of extinction still hinges largely on whether AGI has any interest in humanity surviving. I think you're assuming it will have a distribution of interests like humans do; I don't think that's a safe assumption at all, even given LLM-based AGI and noting that LLMs do really seem to have a distribution of motivations, including kindness toward humans. 

I think how they reason about their goals once they're capable of doing so is very hard to predict, so I'd give it more like a 50% chance they wind up being effectively maximizers for whatever their top motivation happens to be. There seems to be some instrumental pressure toward reflective stability, and that might favor one motivation winning outright vs. sharing power within that mind. I wrote a little about that in Section 5 and more in the remainder of LLM AGI may reason about its goals and discover misalignments by default, but it's pretty incomplete.

Like the rest of alignment, it's uncertain. LLMs have some kindness, but that doesn't mean that LLM-based AGI will retain it. If we could be confident they would, we'd be a lot better set to solve alignment.

Worrying that ASI will do bad stuff because we told it to without bothering to understand the consequences is pretty far down my list of things to worry about. I can understand "eliminates the world as we know it" without understanding the physics by which it does this. Summaries and simplifications are a thing. I'm gonna ask "so hey what consequences would this have that I'd care about" and the ASI, because it's super-smart, will answer in terms I can understand.

If it doesn't, I'll stick to asking it to do things I can understand. Like improving its ability to summarize and communicate.

You haven't addressed my point that a smart ASI will be good at summarizing and simplifying.

Maybe you're not concerned with practical dangers, just the possibility that humans won't always understand everying ASIs come up with. In which case, that's fine; I'm worried about everyone dying long before we get the opportunity to be limited by our understanding. Not being able to fully appreciate everything an ASI is coming up with might be a limitation, but it's far beyond the level of success we can imagine, so I'm putting it in the category of planning the victory party before working out a plan to win.

I'll just toss in my answer (I think it's a fairly common perspective on this):

An agent capable of running a one-year research project would probably (not certainly) be a general reasoner. One could construct an agent that could do a year worth of valuable research without giving it the ability to do general reasoning. But when you think about how a human does a year of valuable research, it is absolutely crucial that the employ general problem-solving skills often to debug that progress and make it actually worthwhile (let alone producing any breakthroughs).

If it can do general reasoning, then it can formulate the questions "why am I doing this?" and "what if I did become god-emperor?" (or other context-expanding thoughts). That creates the new problems I discuss in LLM AGI may reason about its goals which I think are background assumptions of the "alignment is hard" worldview. 

I think that's why the common intuition is that you'd need to solve the alignment problem to have an aligned one-year human-level atent.

Perhaps your "year of research" isn't meant to be particularly groundbreaking, just doing a bunch of studies on how LLMs produce outputs. Figuring out how those results contribute to actually solving the alignment problem might be left to humans. This research could be done theoretically by a subhuman or non-general-reasoning AI. It doesn't look to me like progress is going in this direction, but this is a matter of opinion and interpretation; my argument here isn't sufficient to establish general reasoners as the easier and therefor likelier path to year-long research agents, but I do think it's quite likely to happen that way in the current trajectory..

If we do make year-long research agents that can't really reason but can still do useful research, this might be helpful, but I don't think it's reasonable to assume this will get alignment solved. Having giant stacks of poorly-thought-out research could be really useful, or almost not at all. And note the compute bottleneck in running research, and the disincentives against spending a year running tons of agents doing research.

 

You really can't answer that question yourself? He's suffering on purpose and that makes people wonder why someone would do that. It won't make a difference in lab policy by itself. It draws attention to the arguments. It's a costly signal that he deeply believes ASI will kill us all. That could make some difference.

Stroke victims and comic geniuses sometimes say the same things but they're not the same.

Sounds like you're assuming progress goes on forever. I'd think it would slow down just based on physical limitations. And your supposed ASI is terrible at explanations.

I doubt anything of import is going to work that way. You don't seem to make an argument for it. I see how the exponential suggests it but most exponentials are really s-curves based on limiting factors.

Seems like the king could still understand with a decent explanation, particularly if he bothers to ask about the effects before using it.

Agreed; the alignment plan sketched here skips over why alignment for a merely-human agent should be a lot easier than for a superhuman one, or instruction-following should be easier than value alignment. I think both are probably true, but to a limited and uncertain degree. See my other comment here for more.

I agree with your claim as stated; 98% is overconfident.

I have in the past placed a good bit of hope on the basin of alignment idea, although my hopes were importantly different in that they start below the human level. The human level is exactly when you get large context shifts like "oh hey maybe I could escape and become god-emperor... I don't have human limitations. If I could, maybe I should? Not even thinking about it would be foolish..." That's when you get the context shift.

Working through the logic made me a good bit more pessimistic. I just wrote a post on why I made that shift: LLM AGI may reason about its goals and discover misalignments by default.

And that was on top of my previous recognition that my scheme of instruction-following, laid out in Instruction-following AGI is easier and more likely than value aligned AGI, has problems I hadn't grappled with (even though I'd gone into some depth): Problems with instruction-following as an alignment target.

Could this basin of instruction-following still work? Sure! Maybe!

Is it likely enough by default that we should be pressing full speed ahead while barely thinking about that approach? No, obviously not! Pretty much nobody will say "oh it's only a 50% chance of everyone dying? Well then by all means let's rush right ahead with no more resources for safety work!"

That's basically why I think MIRIs strategy is sound or at least well-thought out. The expert pushback to their 98% will be along the lines of "that's far overconfident! Why, it's only [90%-10%] likely!  That is not reassuring enough for most people who care whether they or their kids get to live. (and I expect really well-thought-out estimates will not be near the lower end of that range).

The point MIRI is making is that expert estimates go as high as 98% plus. That's their real opinion; they know the counterarguments. 

I do think EY is far overconfident, and this does create a real problem for anyone who adopts his estimate. They will want to work on a pause INSTEAD of working on alignment, which I think is a severe tactical error given our current state of uncertainty. But for practical purposes, I doubt enough people will go that high, so it won't create a problem of neglecting other possible solutions; instead it will create a few people who are pretty passionate about working for shutdown, and that's probably a good thing.

I find it's reasonably likely that the basin of instruction-following alignment you describe won't work by default (the race dynamics and motivated reasoningh play a large role), but that modest improvements in either our understanding and/or the water level of average concern and/or the race incentives themselves might be enough to make it work. So efforts in theose directions are probably highly useful.

I think this discussion about the situation we're actually in is a very useful side-effect of their publicity efforts on that book. Big projects don't often succeed on the first try without a lot of planning. And to me the planning around alignment looks concerningly lacking. But there's time to improve it, even in the uncomfortably possible case of short timelines!