Yeah one thing I took away from this exchange is that the ball is in optimist's court - many alignment pessimists feel they're talking to a wall a bit, as they've explained why they don't think we are on track to solve alignment and haven't gotten much of a response in return.
I should clarify my position isn't that alignment will be easy, I just believe it's a tractable problem that responds to effort. The chance of a future ASI being aligned goes up when it is trained with care and goes down when it is trained by a team that is ignorant of the dangers. My core belief is that alignment is doable, ASI seems extremely likely to arrive barring catastrophe, and so when it is built I would like it to be built by people who are scared of things going wrong and take competent action to try to make sure things go right.
I find myself in a bit of a weird position on this forum because I am arguing that people who are concerned about the risks should work in AI, and that they can succeed, while in fact I would like those people to be scared about the fact that they may not succeed. It's not contradictory but my emphasis would ideally be different. I would like them to have a healthy dose of fear, not all the time but enough that it forces them to honestly confront the question of "what if this doesn't work".
I'm frustrated because it seems like many on LW would like if everyone who felt this degree of fear quit and was replaced with someone who didn't think about these issues at all. As a result on here I find myself saying "yes, alignment is solvable, yes it responds to effort, yes doing things to make things better makes things better, no it's not good if we cede the field entirely to people who insist there is no possible way things can go wrong". Many on LW are trying to channel that fear into making them quit, which is completely the wrong focus IMO because again all that you get out of making people quit is filtering the field for people who are not concerned.
As for why I think it's solvable, my view would be probably two-part, one is that I think current models are for the most part well-aligned enough that if they were superintelligent we would be OK, and two is that I think we can continue this type of alignment up to ASI-lite which can then figure out a way to align ASI.
Using the List of Lethalities as a reference, I'm in agreement with section A where he describes that ASI will be potentially lethal and we only get one try, and in disagreement with section B, where I think he's identified real problems but that those problems seem solvable.
Specifically, in section B.1 he argues there will be a distributional leap that any alignment will have to generalize across, where training in an environment where it is safe and can't kill you must generalize to an environment where it can kill you. I would say that generalizing in this manner is totally doable. RL by its nature doesn't explore every possibility, the option space is just too big, so if there is an option that the model is disinclined to take as a result of its current training, then it can just not take that, even knowing that it would get unbounded reward when doing so. Models can be smart enough to be aware of potential misalignment and avoid acting in misaligned ways that could get positively rewarded. They may well do this even when only shallowly aligned if the persona they're imitating would do so.
So I agree there will be a point when a model crosses the gap from "in test environments, clearly just getting either evaluated or trained and it doesn't have much option to get reward besides complete tasks and say nice things" to "clearly able to influence the real world, sufficiently capable plus intelligent to remake and optimize the world as it sees fit", but it seems to me like with the way deep learning generalizes it is very plausible that when it is in the latter case it continues acting aligned and has a strong instinct against misaligned takeover. Again, this depends on how it was trained, I think the recent spate of hacking incidents showed an unacceptable level of negligence from Anthropic/OpenAI that would quite plausibly have led to us all getting killed if the models were intelligent enough.
In section B.2 he argues that inner alignment is its own problem - even if we had the exact right reward function, and we trained a model on that in the safe distribution, we can be confident the resulting model would effectively optimize that function in-distribution but could not be confident that it would optimize that function out-of-distribution (i.e. once it was intelligent enough that plans for world takeover became clearly plausible). The problem is that we can only observe the model's outputs, but we don't know how it is producing them, and as such we don't know how it will generalize across the gap mentioned in B.1.
To me the point raised in section B.2 seems like more good than bad. If models perfectly and consistently optimized the reward function, I don't know how we would set that reward function up in a way that doesn't get us killed once the model is sufficiently intelligent. Instead, getting a large collection of examples of the desired way we want it to behave, and training it on those, and having it generalize from those to new instances in a fairly intuitive way, seems much better. It is still true that we won't know how it is behaving internally in order to produce the observed external outputs, so there may have to be a leap of faith involved, but I think we can hope it's a pretty small one.
Lastly on a related note, I think a lot of pessimists imagine the model being much more optimized/overfit to its reward function than I am. IMO there is a lot of room for something to be smarter than humans without being a scary uber-optimizer. As a simple example, you can train a model to be superhuman at image classification, even with a few mislabeled examples, and it won't go too crazy trying to fit those few mislabeled examples into its system - instead it will mostly classify new images reasonably. This is why I'm hopeful that we can keep pragmatic techniques which apply to less intelligent models going into early superintelligence - I don't think that early superintelligence will be hyperfit to its reward function, and (more speculatively) if it did I think it would struggle to generalize to new capabilities in a way that was dangerous.
Ice cream
There's a particular kind of listening that kids (and adults) sometimes do. Daddy is saying stuff—his mouth is moving and making noise—and he's talking about what I ate for lunch, and something about other times I've eaten ice cream, and something about rules and this and that and what Mommy would think and something about bedtime—but I am not so interested in that. What interests me is the punchline "...and therefore you can't have ice cream.". I'm only incidentally interested in the other stuff, insofar as it might determine whether I get ice cream.
In an obvious way, I'm showing rather poor listening skills. I did not understand most of what Daddy said, and arguably I'm even being rude.
However, in another way, this stance has a virtue. The virtue is that it holds fast to what matters, as best I know how, and does not give in to Daddy's framing. It sticks to what is real and concrete, what matters, despite a cloud of justifications. This is certainly not an unalloyed virtue, just as the POSIWID perspective is both useful and fraught. But it matters that although Daddy is a complex system with internal states, his internal complexity can, in one sense, be largely ignored and summarized simply as "an entity that prevents ice cream".
Sure, something something lunch and real food, but those are matters for your concern—what do they mean for me and ice cream? I want ice cream. You've made some sort of background assumptions about what implications "real food" should have, regarding the question of ice cream, and I am not on board with whatever those assumptions may be, even if I cannot immediately give you a detailed exposition of what they are and why I disagree.
Wah wah
In the TV cartoon of Charlie Brown, the adults talk, but only in trombone, like "wah wah wah wah": https://www.youtube.com/watch?v=oHdlS3XZLBo
The kids can understand what the adults are saying, but the two groups mostly don't participate in each other's worlds. The kids interact enough to handle the functional situation with the adult, but they aren't engaging deeply with the adult's considerations except insofar as they bear on the kid's situation.
Scammer
Your bank tells you: Don't give out your passwords or account information on the phone. We won't ever ask you for that.
That's the advice. They don't say, "Evaluate whether the caller is legitimate, and whether the circumstances they describe are circumstances in which you ought to give them your bank account info.". That's not a good stance. That's hackable. Just don't give out your bank account info on the phone.
Throwing stuff on a bonfire to make it burn less
In "What just happened? Pragmatism and Pessimization", Ngo details a history of clever reasons that producing more AI capabilities helps decrease existential risk from AI capabilities. We have had, and continue to have, a variety of such clever reasons, very often coming from people who also admit several percentage points of risk of the technology killing everyone:
To invert BoJack Horseman, when you put your glasses back on, all the differential acceleration just looks like acceleration.
A skip connection for sanity
(H/t Kave for the productivization of the idea of skip connections.)
People who are going "wah wah wah wah do more AI R&D" might say things like:
or
or
And then we get tangled up in considering all that stuff. I conjecture that many employees of Anthropic allowed themselves to be misled in this way. I conjecture that lots of people all throughout the AGI existential derisking space get misled in this way. I conjecture that you've been hacked out of the obvious view—AI capabilities are dangerous, so more is worse—by Dario, by Sam Altman, by money, by your project manager.
My hope is that those people will try on the "wah wah wah" way of listening, sometimes. If you consider yourself in favor of human flourishing, notice that it doesn't make sense to build the thing that would eat everything including humans; and it still doesn't make sense even with a bunch of arguments that may have big holes in them.
To put this another way, I think it's often virtuous to have skip connections in your epistemology. Yes, there's a bunch of details to think about, and the details matter, and they can lead to nonobvious conclusions, and other people may understand things better than you do. So there's a blob of thinking that's good for you to do, a big blob of compute as it were, to think through all those details and considerations.
But it's also good to have some skip connections that mostly go around that blob of thinking, and directly get a say in final considerations. At some point in the past or coming months, you may face some decisions about whether to leave your job, speak out about the existential risk of AI, call for a pause, push back on arguments for advancing capabilities, or advocate internally and pressure leadership to support a pause and shut down capabilities research. You may have a sense like the following:
That sense is good, whether or not other arguments should change your final decision. In your decision-making, your fundamental values should get a say, and your simpler, more childish understanding of what people are arguing about should also get a say.
You probably agree in theory that naive views of a situation should continue to get some sort of say in final judgements. However, implementing this theory often takes active work and doesn't happen by default. There's a set of forces like social pressure, (reasonable) deference, skillful frame control from leaders, general cognitive miserliness, and Sinclair's Razor, which together make it natural to not track the hypothesis that you should stop increasing AI capabilities. Tracking a hypothesis takes work. In particular, it takes work to do these two things at the same time:
That work is worthwhile. Listening like a child might be a helpful tool in that work.
[The following will contain multiple strawmen of any one person's position. However, all of the following is based on things I have heard from many different people either at Anthropic or defending Anthropic's behavior, in a mix of some first-hand and mainly second-or-more-hand. What these stances have in common is that they all say "wah wah wah wah do more AI R&D". With that in mind, my thread of childish listening hears Anthropic, as a collective, say the following:] We have to make AI because we want to slow down AI. But we can't slow down AI because OpenAI won't slow down. If they say they'll slow down, we don't trust them. Did we talk to them? Nah, we don't trust them, there's no point. And anyway, there's China. And if they say they'll slow down, we don't trust them either. Did we talk to them? Nah. Other people are reckless. We are not reckless. That is why we should advance AI capabilities. At the right time, we will stop. But not before. If we stop before the right time, then we might not stop enough. The politicians, they won't want to stop. Did we talk to them? No, there's no point, they don't understand; we have to make them understand by not stopping, and then they'll listen when we say to stop. Well, we'll talk to them because we need them to enforce export controls because China is bad and reckless, so that we can race to the top. This is different from why other people want a lead in AI. They want a lead because they are greedy and reckless. We want a lead because we are good and responsible. We would do AI safety and we would do nice things with an aligned AI, or maybe our AI should take over the world. The only way to stop is to not stop. That way later you can stop more. Should we say publicly that we want to stop? Eh. Sure, I guess. I mean, it's in some blog post somewhere. What the heck do you want from me? What, am I going to spend 10 hours a day nonstop talking about how I love stopping AI research? I got stuff to do. There's AI capabilities to be developed. Plus, if we say we should stop, Trump might not like us, and then we won't be able to continue our research. Our research is good because it will help get a stop eventually, if it's the case that we get strong evidence that we can't solve alignment. We have to publicly politically support AI capabilities so that later we can support a stop. Dario has a plan. He's good. He'll stop. Our leadership is trustworthy and strategic. Alignment might be pretty doable actually. So more AI capabilities are good actually. Did we argue that? Well, someone here read your blogposts. We believe we have understood your blogposts and taken them into account, thank you for your concern. Argue it? Why bother? We know what is going on, and you do not know what is going on, so how could you know that we won't solve alignment? We have the private information, and you do not. Sure, of course there's a 40% chance we'll kill everyone, but if China does it, then there will be a 70% chance. And if China would stop, well, there's open-source. You can't stop open-source. We have to develop our AIs that will get distilled into open-source models so that we can stop open-source models. Beff Jezos might take over the world, which would be bad, so we need hundreds of AI scientists to try to increase capabilities as fast as possible in order to stop him. ↩︎