# 0

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

A putative new idea for AI control; index here.

In a previous post, I discussed how one might convince an agent not to engage in acausal trade.

The idea was to reward the agent only for extra utility that accrued because the agent was turned on (by a stochastic event ). Since causally disconnected agents couldn't observe , they would "offer" the same "deals" whether or not the agent was turned on.

So the agent might be able to get a tremendous boost in utility from an acausal deal, but that boost would happen in the world as well as the world, so the agent wouldn't count that boost as a benefit.

That was effective as far as it went, but there was one kind of situation it didn't deal with: what if the agent was simulated? Then the event would be within the simulation, and the simulating 'lords of the Matrix' would be causally connected with the agent, hence the agent would act taking their preferences into account.

That in itself is still not a problem; but what if the agent had uncertainty about its own location? It might be in the "real" world, or it might be a simulation made other entities, causally disconnected from the "real" world. Then if the agent acted given that uncertainty, it would be in effect doing a form of acausal trade.

# Grounding the world

There is no costless solution, for any such solution must rule out the agent acting like it was in a simulation, which means that we incur a real cost if we are in a simulation.

But if we're willing to pay that cost, then one way of reducing the problem is to ground in our understanding of physics. So instead of ="human flourishing", we have:

• ="human flourishing in a universe that roughly follows the known laws of physics, will last at least this many trillion years, has these restrictions on how fast information moves and how causality works".

The idea is that a simulation that detailed would be indistinguishable with the real world (and the simulated humans therein would be real moral subjects).