The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Great analysis. I'm impressed by how thoroughly you've thought this through in the last week or so. I hadn't gotten as far. I concur with your projected timeline, including the difficulty of putting time units onto it. Of course, we'll probably both be wrong in important ways, but I think it's important to at least try to do semi-accurate prediction if we want to be useful.

I have only one substantive addition to your projected timeline, but I think it's important for the alignment implications.

LLM-bots are inherently easy to align. At least for surface-level alignment. You can tell them "make me a lot of money selling shoes, but also make the world a better place" and they will try to do both. Yes, there are still tons of ways this can go off the rails. It doesn't solve outer alignment or alignment stability, for a start. But GPT4's ability to balance several goals, including ethical ones, and to reason about ethics, is impressive.^[1] You can easily make agents that both try to make money, and thinks about not harming people.

In short, the fact that you can do this is going to seep into the public consciousness, and we may see regulations and will definitely see social pressure to do this.

I think the agent disasters you describe will occur, but they will happen to people that don't put safeguards into their bots, like "track how much of my money you're spending and stop if it hits $X and check with me". When agent disasters affect other people, the media will blow it sky high, and everyone will say "why the hell didn't you have your bot worry about wrecking things for others?". Those who do put additional ethical goals into their agents will crow about it. There will be pressure to conform and run safe bots. As bot disasters get more clever, people will take more seriously the big bot disaster.

Will all of that matter? I don't know. But predicting the social and economic backdrop for alignment work is worth trying.

Edit: I finished my own followup post on the topic, Capabilities and alignment of LLM cognitive architectures. It's a cognitive psychology/neuroscience perspective on why these things might work better, faster than you'd intuitively think. Improvements to the executive function (outer script code) and episodic memory (pinecone or other vector search over saved text files) will interact so that improvements in each make the rest of system work better and easier to improve.

1. ^{^}

   I did a little informal testing of asking for responses in hypothetical situations where ethical and financial goals collide, and it did a remarkably good job, including coming up with win/win solutions that would've taken me a while to come up with. It looked like the ethical/capitalist reasoning of a pretty intelligent person; but also a fairly ethical one.

Excessive counter-tension would slow movement, yes. But muscles are usually maintaining some small level of tension. There's not really a fully untensed state.

That said, I'm not actually sure that more counter-tension does speed up reaction times within normal ranges. There is such a thing as anticipatory muscle tension when an action is prepared, but that may be a by-product, not a functional way of speeding up reaction times. And I'm not sure it happens when no specific action is being prepared.

So I don't know. If I'd thought a little harder about this, I'd have framed it as a hypothesis. Sorry!

I think we're talking about two different things here. You're talking about how to have agents interact well with each other, and how to make their principles of interaction legible to humans. I'm talking about how to make sure those agents don't take over the world and kill us all, if/when they become smarter than we are in every important way.

Ah. Now I understand why you're going this direction.

I think a single human mind is modeled very poorly as a composite of multiple agents.

This notion is far more popular with computer scientists than with neuroscientists. We've known about it since Minsky and think about it; it just doesn't seem to mostly be the case.

Sure you can model it that way, but it's not doing much useful work.

I expect the same of our first AGIs as foundation model agents. They will have separate components, but those will not be well-modeled as agents. And they will have different capabilities and different tendencies, but neither of those are particularly agent-y either.

I guess the devil is in the details, and you might come up with a really useful analysis using the metaphor of subagents. But it seems like an inefficient direction.

Knowing how much time we've got is important to using it well. It's worth this sort of careful analysis.

I found most of this to be wasted effort based on too much of an outside view. The human brain gives neither an upper nor lower bound on the computation needed to achieve transformative AGI. Inside views that include gears-level models of how our first AGIs will function seem much more valuable; thus Daniel Kokatijlo's predictions seem far better informed than the others here.

Outside views like "things take longer than they could, often a lot longer" are valuable, but if we look at predicting when other engineering feats would first be accomplished, good predictions would've taken both expert engineers and usually, those who could predict when funding and enthusiasm for the project would become available.

In any case, more careful timeline predictions like this would improve our odds.

Sharing well-informed, carefully-reasoned scenarios of how things might go right or wrong helps figure out how to steer the future.

This post engages substantively and clearly with IMO the first or second most important thing we could be accomplishing on LW: making better estimates of how difficult alignment will be.

It analyzes how people who know a good deal about alignment theory could say something like "AI is easy to control" in good faith - and why that's wrong, in both senses.

What Belrose and Pope were mostly saying, without being explicit about it, is that current AI is easy to control, then extrapolating from there by basically assuming we won't make any changes to AI in the future that might dramatically change this situation.

This post addresses this and more subtle points, clarifying the discussion.

I think you are probably attending more often to sensory experiences, and thereby both creating and remembering more detailed representations of physical reality.

You are probably doing less abstract thought, since the number of seconds in a day hasn't changed.

Which do you want to spend more time on? And which sorts? It's a pretty personal question. I like to try to make my abstract thought productive (relative to my values), freeing up some time to enjoy sensory experiences.

I'm not sure there's a difference in representational density in doing sensory experience vs. abstract thought. Maybe there is. One factor in making it seem like you're having more sensory experience is how much you can remember after a set amount of time; another is whether each moment seems more intense by having strong emotional experience attached to it.

Or maybe you mean something different by more subjective experience.

It depends entirely on what you mean by consciousness. The term is used for several distinct things. If my mom had lost her sense of individuality but was still having a vivid experience of life, I'd keep valuing her. If she was no longer having a subjective experience (which would pretty much require being unconscious since her brain is producing an experience as part of how it works to do stuff), I would no longer value her but consider her already gone.