Chantiel's Shortform

Chantiel

This is a special post for quick takes by Chantiel. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I've thought of a way in which other civilizations could potentially "hack" Updateless Decision Theoretic agents on Earth in order to make them do whatever the other civilization wants them to do. I'm wondering if this has been discussed before, and if not, what people think about it.

Here I present a method of that would potentially aliens to take control of an AI on Earth that uses Updateless Decision theory.

Note that this crucially depends on different agents with the AI's utility function but different situations terminally valuing different things. For example, suppose the AI places special value in the welfare of its creators and other creatures in the world it's in. An AI with the same utility function in a different world would then place more terminal value the welfare of the creatures in its world. This doesn't sound like a very stringent requirement; people are not infinitely altruistic, so naturally people may want AI to favor the welfare of themselves over random aliens.

Suppose on Earth an AI is created that uses UDT and has a reasonable utility function. And suppose there are some aliens who want to acausally take over Earth. Let T be the percept-action mapping that would result in the AI conquer Earth and give control to the aliens.

So, the aliens make a world that's like Earth but has a trillion AIs on it using UDT and that have the same utility function as the AI on Earth. If any of the trillion AIs don't use T their percept-action mapping, then the Earth-like world would get destroyed by the aliens, killing all the inhabitants the AIs care about.

The chosen percept-output mapping chosen by UDT is done before updating on any evidence about the AI's current circumstances. So the AIs both on Earth and on the Earth-like alien planet reason, "For 99.99999999...% of agents with my utility, it would be essential to use T. And for only 0.000000...1% is it essential to not use it. So using it would a priori be much better for my utility function. So that's what I'll do."

And then we die.

And this would be pretty cheap for the aliens. With just a, by their standards, small amount of resources to make tons of UDT AIs, they would have the potential to take control of huge numbers of worlds. With just one world filled with AIs that need to use T, they could potentially get huge numbers of AIs on Earth-like worlds to also use T, giving the aliens control over many parts of the universe.

I think the essence of the problem is that UDT makes agents that score well on their utility function "on average" for an arbitrary UDT agent with that utility function, but what we really want is one that scores well in the circumstances we're actually in.

One way to potentially get around this problem is by changing the utility function to avoid the level of indirection. That is, instead of making the AI's utility function something like, "Value whatever my creators wanted me to value", instead first run a non-agentic AI that infers what the creators would want the AI to value, and then use that as a fixed utility function. For example, it could result in the AI finding a utility function, "Make creatures reasonably satisfied on Earth, but also give substantial moral concern to the welfare of creatures outside of Earth". That way, hopefully we wouldn't get taken over by aliens.

1TekhneMakre5y

Yeah I don't know of a clear case for those supposed properties of UDT. By pareto optimal I mean just, two UDT agents will pick a Pareto optimal policy. Whereas, say, two CDT agents may defect on each other in a PD. This isn't a proof, or even really a general argument, but one reason to suspect UDT is convergent, is that CDT would modify to be a sort of UDT-starting-now. At least, say you have a CDT agent, and further assume that it's capable of computing the causal consequences of all possible complete action-policies it could follow. This agent would replace itself with P-bot, the bot that follows policy P, where P is the one with the best causal consequences at the time of replacement. This is different from CDT: if Omega scans P-bot the next day, P-bot will win the Transparent Newcomb's problem, whereas if CDT hadn't self-modified to be P-bot and Omega had scanned CDT tomorrow, CDT would fail the TNP for the usual reason. So CDT is in conflict with its future self.

1Chantiel5y

Two UDT agents actually can potentially defect in prisoner's dilemma. See the agent simulates predictor problem if you're interested. But I think you're right that agents would generally modify themselves to more closely resemble UDT. Note, though, that the decision theory a CDT agent would modify itself to use wouldn't exactly be UDT. For example, suppose the causal decision theory agent had its output predicted by Omega for Newcomb's problem before the agent even came into existence. Then by the time the CDT agent comes to existence, modifying itself to use UDT would have no causal impact on the content of the boxes. So it wouldn't adopt UDT in this situation and would still two-box.

Well, the way the agent loses in ASP is by failing to be updateless about certain logical facts (what the predictor predicts). So from this perspective, it's a SemiUDT that does update whenever it learns logical facts, and this explains why it defects.

> So it wouldn't adopt UDT in this situation and would still two-box.

True, it's always [updateless, on everything after now].

Chantiel's Shortform

2

Chantiel's Shortform

2