Thanks for writing this, it's great to see people's reasons for optimism/pessimism!

My views on alignment are similar to (my understanding of) Nate Soares’.

I'm surprised by this sentence in conjunction with the rest of this post: the views in this post seem very different from my Nate model. This is based only on what I've read on LessWrong, so it feels a bit weird to write about what I think Nate thinks, but it still seems important to mention. If someone more qualified wants to jump in, all the better. Non-comprehensive list:

I think the key differences are because I don’t think there’s enough evidence to confidently predict the difficulty of future problems, and I do think it’s possible for careful labs to avoid active commission of catastrophe.

Not as important as the other points, but I'm not even sure how much you disagree here. E.g. Nate on difficulty, from the sharp left turn post:

Although these look hard, I’m a believer in the capacity of humanity to solve technical problems at this level of difficulty when we put our minds to it. My concern is that I currently don’t think the field is trying to solve this problem.

And on the point of labs, I would have guessed Nate agrees with the literal statement, just thinks current labs aren't careful enough, and won't be?

Language model interventions work pretty well

My Nate model doesn't view this as especially informative about how AGI will go. In particular:

HHH omits several vital pieces of the full alignment problem, but if it leads to AI that always shuts down on command and never causes a catastrophe I’ll be pretty happy.

If I understand you correctly, the "vital pieces" that are missing are not ones that make it shut down and never cause catastrophe? (Not entirely sure what they are about instead). My Nate model agrees that vital pieces are missing, and that never causing a catastrophe would be great, but crucially thinks that the pieces that are missing are needed to never cause a catastrophe.

Few attempts to align ML systems

In my Nate model, empirical work with pre-AGI/pre-sharp-left-turn systems can only get you so far. If we can now do more empirical alignment work, that still won't help with what are probably the deadliest problems. Once we can empirically work on those, there's very little time left.

Interpretability is promising!

Nate has said he's in favor of interpretability research, and I have no idea if he's been positively surprised by the rate of progress. But I would still guess you are way more optimistic in absolute terms about how helpful interpretability is going to be (see his comments here).

If a major lab saw something which really scared them, I think others would in fact agree to a moratorium on further capabilities until it could be thoroughly investigated.

Nate wrote a post which I understand to argue against more or less this claim.

I don’t expect a ‘sharp left turn’

Nate has of course written about how he does expect one. My impression is that this isn't just some minor difference in what you think AGI will look like, but points at some pretty deep and important disagreements (that are upstream of some other ones).

Maybe you're aware of all those disagreements and would still call your views "similar", or maybe you have a better Nate model, in which case great! But otherwise, it seems pretty important to at least be aware there are big disagreements, even if that doesn't end up changing your position much.

I expect that human extinction would follow from comprehensive disempowerment within decades.

Why would it take decades? (In contrast to scenarios where AI builds nanotech and quickly disassembles us for spare atoms.) Are you imagining a world where AI-powered corporations, governments, &c. are still mostly behaving as designed, but we have no way to change course when it turns out that industrial byproducts are slowly poisoning us, or ...?

Anecdotally, I’ve seen RLHF generalize alignmentish properties like helpfulness and harmlessness across domains at least as well as it generalizes capabilities at their present levels, and I don’t share the intuition that this is very likely to change in future. I think ‘alignment generalization failures’ are both serious and likely enough to specifically monitor and mitigate, but not that a sharp left turn is anywhere near certain.

IMO, I don't think this is a realistic hope, since from my point of view they come with unacceptable downsides.

Quoting another post:

“Discovering Language Model Behaviors with Model-Written Evaluations” is a new Anthropic paper by Ethan Perez et al. that I (Evan Hubinger) also collaborated on. I think the results in this paper are quite interesting in terms of what they demonstrate about both RLHF (Reinforcement Learning from Human Feedback) and language models in general.

Among other things, the paper finds concrete evidence of current large language models exhibiting:

convergent instrumental goal following (e.g. actively expressing a preference not to be shut down), non-myopia (e.g. wanting to sacrifice short-term gain for long-term gain), situational awareness (e.g. awareness of being a language model), coordination (e.g. willingness to coordinate with other AIs), and non-CDT-style reasoning (e.g. one-boxing on Newcomb's problem). Note that many of these are the exact sort of things we hypothesized were necessary pre-requisites for deceptive alignment in “Risks from Learned Optimization”.

Furthermore, most of these metrics generally increase with both pre-trained model scale and number of RLHF steps. In my opinion, I think this is some of the most concrete evidence available that current models are actively becoming more agentic in potentially concerning ways with scale—and in ways that current fine-tuning techniques don't generally seem to be alleviating and sometimes seem to be actively making worse.

Interestingly, the RLHF preference model seemed to be particularly fond of the more agentic option in many of these evals, usually more so than either the pre-trained or fine-tuned language models. We think that this is because the preference model is running ahead of the fine-tuned model, and that future RLHF fine-tuned models will be better at satisfying the preferences of such preference models, the idea being that fine-tuned models tend to fit their preference models better with additional fine-tuning.[1]

This is essentially saying that there's good evidence that the precursors of deceptive alignment is there, and this is something that I think no alignment plan could deal with here.

Here's a link to the post:

https://www.lesswrong.com/posts/yRAo2KEGWenKYZG9K/discovering-language-model-behaviors-with-model-written#RHHrxiyHtuuibbcCi

Thus, I think this:

There are strong theoretical arguments that alignment is difficult, e.g. about convergent instrumental goals, and little empirical progress on aligning general-purpose ML systems. However, the latter only became possible a few years ago with large language models, and even then only in a few labs! There’s also a tradition of taking theoretically very hard problems, and then finding some relaxation or subset which is remarkably easy or useful in practice – for example SMT solvers vs most NP-complete instances, CAP theorem vs CRDTs or Spanner, etc. I expect that increasing hands-on alignment research will give us a similarly rich vein of empirical results and praxis from which to draw more abstract insights.

Is somewhat unrealistic, since I think a key relaxation alignment researchers are making is pretty likely to be violated IRL.

For those of us who like accurate beliefs, could you make a second list of the truths you filtered out due to not furthering hope?

Ironically this still seems pretty pessimistic to me.  I'm glad to see something other than "AHHH!" though, so props for that.

I find it probably more prudent to worry about a massive solar flare, or an errant astral body collision, than to worry about "evil" AI taking a "sharp turn".

I put quotes around evil because I'm a fan of Nietzsche's thinking on the matter of good and evil.  Like, what, exactly are we saying we're "aligning" with?  Is there some universal concept of good?

Many people seem to dismiss blatant problems with the base premise— like the "reproducibility problem".  Why do we think that reality is in fact something that can be "solved" if we just had enough processing power, as it were?  Is there some hard evidence for that?  I'm not so sure.  It's not just our senses that are fallible.  There are some fundamental problems with the very concept of "measurement' for crying out loud, which I think it's pretty optimistic to think that super-smart AI is just going to be able to skip over.

I also think if AI gets good enough to "turn evil" as it were, it would be good enough to realize that it's a pretty dumb idea.  Humans don't really have much in common with silicon-based life forms, afaik.  You can find more rare elements, easier, in space, than you can on Earth.  What, exactly, would AI gain by wiping out humanity?

I feel that it's popular to be down on AI, and saying how scary all these "recent" advances really are, but it doesn't seem warranted.

Take the biological warfare ideas that were in the "hard turn" link someone linked in their response.  Was this latest pandemic really a valid test-run for something with a very high fatality rate? (I think the data is coming in that far more people had COVID than we initially thought, right?)

CRISPR &c. are, to me, far more scary, but I don't see any way of like, regulating that people "be good", as it were.  I'm sure most people here have read or seen Jurassic Park, right?  Actually, I think our Science Fiction pretty much sums up all this better than anything I've seen thus far.

I'm betting if we do get AGI any time soon it will be more like the movies Her or AI than Terminator or 2001, and I have yet to see any plausible way of stopping, or indeed "ensuring alignment" (again, along what axis?  Who's definition of "good"?)

The answer to any question can be used for good or ill.  "How to take over the world" is functionally the same as "how to prevent world takeovers", is it not?  All this talk of somehow regulating AI seems akin to talk of regulating "hacking" tools, or "strong maths" as it were.

Are we going to next claim that AI is a munition?

It would be neat to see some hard examples of why we should fear and why we think we can control alignment… maybe I'm just not looking in the right places?  So far I don't get what all the fear is about— at least not compared to what I would say are more pressing and statistically likely problems we face.

I think we can solve some really hard problems if we work together, so if this is a really hard problem that needs solving, I'm all for getting behind it, but honestly, I'd like to see us not have all our eggs in one basket here on Earth before focusing on something that seems, at least from what I've seen so far, nigh impossible to actually focus on.