There's a possible failure mode for AI that I'm a little concerned about that I haven't seen others really acknowledge, so I'm going to talk about it.

But before I do that, I want to mention something sort of interesting. You probably know about framing what physical action to take as a decision problem: you can consider different actions to take, consider different possible consequences of them and how probable and good or bad these consequences are, and use this information to select which action has the highest expected utility^[1]

Well, you can also do that with what computational action to take. That is, when determining what physical action to take, you can't consider every single possible scenario or possible implication of an action you could take, so you need some way of prioritizing which ones to think about. And this, itself, can be framed as a decision problem: you can think about different possible implications of thinking about some consideration(s) or possible sort of plan, such has how they might affect what action you end up taking. And you can think about how good or bad this implication is, such has how much it might improve the quality of the action(s) you decide to take. And you can then use this to select the computational action(s) to take/considerations to think about that maximize expected utility. I call this sort of thing general-planning-based prioritization.

But you can't do this always, for every computational action. Remember that expected utility is computed as a function of the possible outcomes of taking an action. If you're determining what thoughts to think based on just applying your utility function to figure out the expected utility of thinking a thought, then you need some way to choose which possible outcomes of thinking a thought to think about when estimating the expected utility of thinking a certain thought. But that just leaves you with the same problem. And how do you solve this problem? By thinking about which of these thoughts would have the highest expected utility to think about by considering their implications? Then how do you do this? This becomes infinitely recursive without a base case.

So you need to, at least sometimes, have some other way of selecting what considerations to think about. This can involve general-purpose heuristics like selecting what possibilities to think about based on how probable they are and how relevant they are to your goals. You can also use your general reasoning and planning system to come up with other, more specific heuristics for what considerations to think about. For example, you might use your general reasoning ability to conclude that decreasing risk from rogue AI is an important thing to think about, and then later use the heuristic of being more likely to select considerations that are relevant to AI risk. I call using these sorts of heuristics, whether general-purpose or more specific, to be heuristic-based prioritization.

I think that ideally, you should use a combination of general-planning-based prioritization and heuristic-based prioritization. However, deciding how much to use each isn't exactly trivial.

If you use too much general-planning-based prioritization, you might spend so much time thinking about what to think about that you hardly do any object-level thinking.

If you use too much heuristic-based prioritization, you might fail to have good priorities, for example by not deeply considering the different benefits and drawbacks of prioritizing different research agendas. At worst, you may end up prioritizing thinking about something that's actively harmful to think about. Arguably, AI capabilities researchers do this by prioritizing thinking about how to increase AI capabilities.

And, to be clear, this very much does not seem like a concern that could only apply to humans. I worry it could also apply to AIs, including value-aligned ones. And I'm concerned that some AIs that are cognitively flawed, but still advanced enough to be dangerous, would be dangerously vulnerable to problems involving flawed thought-prioritization. And, due to their potentially alien psychologies and cognitions, I worry that they may be dangerously vulnerable to this in ways humans are not.

So I'll talk about a possible failure mode that might affect value-aligned AIs that don't do a good job with thought prioritization. This concern is primarily relevant to AIs that don't do enough general-planning-based prioritization, especially ones that don't do any at all. But it may also effect AIs that use planning-based prioritization, but do so with only a very basic, restrictive world model.

Consider an AI called, say, Coral, that considers different possible actions, considers possible consequences of taking said actions, and then outputs the action with the highest expected utility according to its utility function. This system could have been hard-coded by the AI's developers or could have been created in an artificial neural network via gradient descent or some other method. And suppose Coral is value-aligned with humanity.

As I've said before, Coral can't consider all possible implications of an action it takes, so it would need some way of selecting which possible implications to think. Suppose for now that Coral just uses heuristic-based prioritization.

I'm concerned that for a lot of plausible and reasonable-sounding heuristics, this might end terribly, even if the Coral is value-aligned.

(Again, this failure mode could apply to some AIs that don't just use heuristic-based prioritization, but I'm assuming Coral exclusively uses it for the sake of simplicity and concreteness.)

For example, suppose Coral selects considerations based on some function for estimating how likely thinking about the consideration is to change what action it sees as best. Then consider this: suppose Coral considers the possibility of another AI that's hostile to it. To be clear, the hostile AI doesn't have to actually exist, Coral just needs to consider the possibility that it exists.

Now, Coral considers the following possibility: the hostile AI realizes that the hardware Coral runs on, or Coral itself, has certain vulnerabilities such that, if Coral performs certain computations, the vulnerabilities can be exploited to run arbitrary code, allowing Coral to be destroyed or modified. For example, maybe if Coral thinks just the wrong thoughts in just the wrong order, a rowhammer attack would be performed that would modify Coral's utility function to be that of the hostile AI. And suppose the hostile AI can predict that such computations would be performed when Coral thinks about if a certain statement, S, is true.

(The failure mode generalizes to dangerous computations triggered by things other than thinking about the truth value of a statement. For example, there are analogous failure modes if the computation is triggered by considering a certain plan or a certain implication of one under certain circumstances. But for the sake of concreteness, I'll be assuming it's triggered by thinking about the truth value of statement S.)

Now, Coral reasons, the hypothetical hostile AI realizes this and thinks about how to get Coral to think these thoughts. So the hostile AI (hypothetically) decides to do the following: first, it comes up with an action that would have have a lot of strategic significance to Coral, for example deciding whether or not to cooperate with Coral. Then, it decides to have the policy of performing the action if and only if S is true.

Thus, since the truth value of S has high strategic significance to Coral, Coral would think about if S is true. And since S was carefully selected by the malicious AI to cause Coral to get exploited when trying to see if S is true, Coral would get hacked by the malicious AI.

Which sounds bad.

Now, you might be thinking something like this: "Wow, this seems really dumb. Wouldn't Coral just realize that thinking about S would be a really bad idea and just not do it?"

Not necessarily. Remember that Coral selects things to think about based on some estimate of how likely that thinking a thought would change what action it thinks is best in expectation. And thinking about S really would have a high probability of changing what action it thinks is best.

Again, you might still be thinking: "This is still dumb. Why doesn't Coral just select which thoughts to think based one whatever maximizes expected utility according to its utility function?"

Well, as I said before, that's because Coral sort of can't. Remember that expected utility is computed as a function of the possible outcomes of taking an action and their probabilities. If Coral determines what thoughts to think by just applying its utility function to figure out the expected utility of thinking a thought, then it needs some way to choose which possible implications of a thinking thought to think about. But that just leaves us with the same problem. And how do you solve this problem? By thinking about which of these thoughts would have the highest expected utility to think about by considering their implications? Then how do you do this? Where is the base case to this recursion?

That said, it still seems to me that the solution to this problem is to have the AI able to leverage its general-purpose reasoning and world model when determining what sorts of considerations to think about.

Again, you can't do this every time for every consideration, but you could still use general-planning-based prioritization to direct your cognitions on a high level. In this case, Coral could have considered coming up with statement S and thinking about its truth value, would realize this would have potentially devastating consequences, and thus wouldn't do it.

And being able to do this sort of general-planning-based prioritization sounds potentially very useful. And for this reason I suspect that by default advanced AIs would end up using it.

However, that doesn't mean this failure mode is irrelevant.

For one, advanced AIs that don't use general-planning-based prioritization don't sound completely implausible to me. Even if general-planning-based prioritization is ultimately better than other methods, perhaps other methods are still sufficient to create existentially-dangerous AI. And perhaps this failure mode would occur to some AI before it changes itself to have general-planning-based prioritization.

Further, it still sounds possible that this failure mode could affect AIs with general-planning-based prioritization. As I said, you can't use general-purpose cognition every time when thinking about what considerations to think about. Perhaps there would be some flaw in when it uses its general-purpose cognition for prioritization that would make it not use it nearly enough when considering thinking potentially-dangerous thoughts. It might sound ridiculous from a human perspective, but it doesn't sound completely implausible to me that an AI could come up with S and infer its truth value realizing that this would be terrible, and before it could self-modify to use a safer prioritization system. And it's not like you can necessarily just rely on reinforcement-learning-based trial and error to deal with this issue, since just a single such dangerous thought could potentially cause an existential catastrophe.

A smarter AI would potentially be able to anticipate and fix this possible failure mode more quickly than a less smart AI could, but it might also be faster at thinking up the dangerous thoughts in the first place. So it's not clear to me that you could just rely on an AI being smart to avoid this.

Also, some people have argued against the plausibility of existential risk from AI by arguing that advanced AIs based on predicting or mimicking human outputs would by default not actually use this sort of general-planning-based prioritization. This failure mode suggests such AIs could still be existentially dangerous.

Additionally, some people have considered trying to make AI safer by deliberately designing it to not really have preferences and not really be a planning-based agent in the first place, and instead just make inferences about things. General-planning-based prioritization uses planning, so such an AI that lacks planning and preferences would not use general-planning-based prioritization. So this failure mode poses an obstacle to making such systems safe.

(I'd like to thank JoeC for reviewing this article.)

1. ^{^}

   Yes, there are other ways of framing decision problems other than expected utility maximization. But this isn't really relevant for my purposes.