The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

There are a lot of merits to avoiding unnecessary premises when they might be wrong.

There are also a lot of merits for reasoning from premises when they allow more progress, and they're likely to be correct. That is, of course, what I'm trying to do here.

Which of these factors is larger has to be evaluated on the specific instances. There's lots more to be said about those in this case, but I don't have time to dig into it now, and it's worth a full post and discussion.

Yes, I think it does make sense ot think of this as a continuum, something I haven't emphasized to date. There's also at least one more dimension, that of how many (and which) humans you're trying to align to. There's a little more on this in Conflating value alignment and intent alignment is causing confusion.

IF is definitely an attempt to sidestep the difficulties of value alignment, at least partially and temporarily.

What we want from an instruction-following system is exactly what you say: one that does what we mean, not what we say. And getting that perfectly right would demand a perfect understanding of our values. BUT it's much more fault-tolerant than a value-aligned system. The Principal can specify what they mean as much as they want, and the AI can ask for clarification as much as it thinks it needs to- or in accord with the Principal's previous instructions to "check carefully about what I meant before doing anything I might hate" or similar.

If done correclty, value alignment would solve the corrigibility problem. But that seems far harder than using corrigibility in the form of instruciton-following to solve the value alignment problem.

I am definitely thinking of IF as it applies to systems with capability for unlimited autonomy. Intent alignment as a concept doesn't end at some level of capability - although I think we often assume it would.

How it would understand "the right thing" is the question. But yes, intent alignment as I'm thinking of it does scale smoothly into value alignment plus corrigibility if you can get it right enough.

(A small rant, sorry) In general, it seems you're massively overanchored on current AI technology, to an extent that it's stopping you from clearly reasoning about future technology.

You are right that I am addressing AGI with a lot of similarities to LLMs. This is done in the interest of reasoning clearly about future technologies. I think good reasoning is a mix of predicting the most likely forms of AGI, and reasoning more broadly. Perhaps I didn't make clear enough in the post that I'm primarily addressing LLM-based AGI. Much of my alignment work (informed by my systems neuroscience work) is on routes from LLMs to AGI. In this theory, LLMs/foundation models are expanded (by adding memory systems and training/scaffolding them for better metacognition) into loosely brainlike cognitive architectures. In those posts I elaborate reasons to think such scaffolded LLMs may soon be "real AGI" in the sense of reasoning and learning about any topic, including themselves and their cognition and goals.  (although that sort of AGI wouldn't be dramatically superhuman in any area, and initially subhuman in some capabilities).

If you have an alternate theory of the likely form of first takeover-capable AGI, I'd love to hear it! It's good to reason broadly where possible, and I think a lot of the concerns are general to any AGI at all or any network-based AGI. But constraining alignment work to address specific likely types of AGI lets us reason much more specifically, which is a lot more useful in the worlds where that type of AGI is what we really are faced with aligning. 

You're talking about AGI here. An agent capable autonomously doing research, play games with clever adversaries, detecting and patching it's own biases, etc. It should be obvious that you can't use current LLM flaws as a method of extrapolating the adversarial robustness of this program.

Yes, good point. I didn't elaborate here, but I do think there's a good chance that the more coherent, intelligent, and introspective nature of any real AGI might make jailbreaking a non-issue.  But jailbreaking might still be an issue, because the core thought generator in this scenario is an advanced LLM.

No. You're entirely ignoring inner alignment difficulties. The main difficulties. There are several degrees of freedom in goal specification that a pure goal target fails to nail down. These lead to an unpredictable reflective equilibrium.

Yes I am entirely ignoring inner alignment difficulties. I thought I'd made that clear by saying earlier

There are substantial technical challenges [including] problems with specifying goals well enough that they won’t be misgeneralized or misinterpreted (from the designer’s perspective). There are serious implementational problems for any alignment target. [...] Here I am leaving aside the more general challenges, and addressing only those that are specifically relevant to instruction-following (and in some cases corrigibility) as an alignment target.

I didn't use the term "inner alignment" because I don't find it intuitive or clarifying; there isn't a clear division between inner and outer, and they feel like jargon. So I use misgeneralization, which I feel encompasses inner misalignment as well as other (IMO more urgent) concerns. Maybe I should get on board and use inner and outer alignment just to speak the lingua franca of the realm.

Thanks, that's all relevant and useful!

Simplest first: I definitely envision a hierarchy of reporting and reviewing questionable requests. That seem like an obvious and cheap route to partly address the jailbreaking/misuse issues.

I've also envisioned smarter LLM agents "thinking through" the possible harms of their actions, and you're right that does need at least a pretty good grasp on human values. Their grasp on human values is pretty good and likely to get better, as you say. I haven't thought of this as value alignment, though, because I've assumed that developers will try to weigh following instructions over adhering to their moral strictures. But you're right that devs might try a mix of the two, instead of rules-based refusal training. And the value alignment part could become dominant whether by accident or on purpose.

I haven't worked through attempting to align powerful LLM agents to human values in as much detail as I've tried to think through IF alignment. It's seemed like losing corrigibility is a big enough downside that devs would try to keep IF dominant over any sort of values-based rules or training. But maybe not, and I'm not positive that value alignment couldn't work. Specifying values in language (a type of Goals selected from learned knowledge: an alternative to RL alignment) has some nontrivial technical challenges but seems much more viable than just training for behavior that looks ethical. (Here' it's worth mentioning Steve Byrnes' non-behaviorist RL in which some measure of representations is part of the reward model, like trying to reward the model only when it's thinking about doing good for a human). 

Language in the way we use it is designed to generalize well, so language might accurately convey goals in terms of human values for full value alignment. But you'd have to really watch for unintended but sensible interpretations - actually your sequence AI, Alignment, and Ethics (esp. 3-5) is the most thorough treatment I know of why something like "be good to everyone" or slightly more careful statements like "give all sentient beings as much empowerment as you can" will likely go very differently than the designer intended - even if the technical implementation goes perfectly!

I don't think this goes well by default, but I'm not sure an LLM architecture given goals in language very carefully, and designed very carefully to "want" to follow them, couldn't pull off value alignment.

I also think someone might be tempted to try it pretty quickly after developing AGI. This might happen if it seemed like proliferation of IF AGI was likely to lead to disastrous misuse, so a value-aligned recursively self-improving sovereign was our best shot. Or it might be tried based on worse or less altruistic logic.

Anyway, thanks for your thoughts on the topic.

I am sorry for your loss. Death is natural but it is so, so bad.

I'm assuming you're posting this here in part to foster a discussion of this tension between preventing deaths in the short term and taking more risks on killing everyone by getting alignment wrong if we rush toward AGI.

This tension is probably going to become more widespread as the concept of AGI becomes more prominent. Many people will want faster progress, in hopes of saving themselves or their loved ones. Longtermism is pretty dominant here on LW, but it is very much a minority view in society at large. Thus, this urge to rush will have to be countered by spreading an awareness of how rushing toward AGI is increasing the odds for older people, while risking the lives of their children and grandchildren. And all of the glorious generations to follow - while most people aren't longtermist, the idea of unimaginable flourishing does hold some weight in their minds, and it isn't that ahrd to imagine in broad form.

I face this dilemma myself. At 50 and in imperfect health, my likely end falls somewhere in the middle of my predicted range of hitting longevity takeoff. Any small speedup or slowdown might shift my odds substantially. I don't know what I'd do if I had real power over our rate of progress, but I don't. So I'll continue advocating that we slow down as much as we can, while also working as fast as we can to align our first AGI/ASIs. That speed will improve our odds of collective survival in the likely case that we can't slow down substantially. And it might even save a few more of the precious unique minds now alive.

I found this super useful, thank you so much! Relating the process of doing good work to mindfulness meditation clicks for me, and I'll be trying it. Gently putting attention back on-topic and practicing noticing when it's strayed is mindfulness, and it's addressing a key challenge in getting good work done. Treating the rest of the challenges in doing research as opportunities for learning about yourself and growing- if you pay attention to them - also makes sense.

I'm a bit suspicious of meditation as an end in itself. But getting better at research (or relationships, another suggested focus for mindfulness and growth) are worthy applications!

I wish I could attend the retreat; it would be good to really practice this attitude, and your offered rate is great. But the French Pyranees is a long expensive trip from where I'm at. So I'll try applying my own system of habit modification to this goal, and let you know how it goes.

Why do you ask? This is a somewhat interesting question, but I don't usually spend time on it. I think alignment/AI thinkers don't think about it much because we're usually more concerned with getting an AGI to reliably pursue any target. If we got it to actually have humanity's happiness as its goal, in the way we meant it and would like it, we'd just see what it does and enjoy the result. But getting it to reliably do anything at all is one problem, and making that thing something we actually want is another huge problem. See A case for AI alignment being difficult for a well-written intro on why most of us think alignment is at least fairly hard.

I don't spend much time thinking about different specific value alignment targets because I think we should first focus on how to achieve any of them. I couldn't see exactly what the world values survey was from that link at a quick glance, but I'm not sure the details matter. It's would probably produce a vastly better future than a value target like "solve hard problems" or "make me a lot of money" would; there are probably better-future-proofed targets that would be even better; but steering away from the worst and toward the better is my primary goal right now, because I don't hink we have that in hand at all.