There are two powers developing frontier AI, China and America. Do you see the elite of either country abandoning the pursuit of ever-more-potent AI? 

Let's call  the more complex agents, Agent1 and Agent2, and their purported simulations, SImulation1 and Simulation2. So in one sub-universe it's Agent1 interacting with Simulation2, and in the other sub-universe, it's Agent2 interacting with Simulation1. 

Another issue here is, what about the general sensory experience of the Agents? Agent1, while talking with Simulation2, may be subjected to the equivalent of street noise, or interruptions by colleagues, or - you get the idea. Presumably, if Simulation1 doesn't experience these same interruptions, it is no longer actually a simulation of Agent1. 

One might reasonably conclude that the whole concept of acausal interaction is dubious. Agent1 is not in any sense communicating with Agent2, they are communicating with an entirely independent agent, Simulation2, which has some limited similarity to Agent2. One could even go further and say, even if Simuation2 exactly tracked Agent2's experience, Agent1 is still not communicating with Agent2 in any sense, they are only communicating with Simulation2. 

I feel that, to save the idea of acausal interaction, you'd really have to start with a Tegmark-like omniverse in which all possibilities are real. Then, when you talk to a simulation, you can say that you're also acausally interacting with all those agents elsewhere in the omniverse for whom your simulation happens to be 100% accurate. But how do we define these equivalence classes? 

Another way to try to save the idea, is to focus on the idea of acausal coordination among identical agents. It is true by definition that all the perfect simulations of an agent, throughout the omniverse, will behave the same as the original. So maybe you can construct some scenario in which you are speaking to a simulation of an entity that is powerful within its own simulation, and elsewhere in the omniverse there will be perfect copies of that interaction in which the roles are reversed, and this can be the basis for acausal trade or acausal blackmail. 

Yet another way to try to save the idea of acausal interaction, is to think of it as an interaction between classes of entities which are not identical, but simply share some trait. For example, there have been attempts to argue that voting in an election, even though your individual vote is extremely unlikely to matter, can be justified as acausal cooperation among voters. I suppose this would be a version of superrationality as defined by Hofstadter. 

There seems to be ample room for a thoroughly skeptical assessment of whether acausal communication, etc, makes any sense (especially given the arguments that you can get some of the wins of timeless decision theories from causal decision theories plus uncertainty as to which copy you are). But as far as I know, there's no essay-length discussion of this nature. 

The current leader at Manifold ("rules clarification...") doesn't have a link to it from here?

I am not familiar with this debate, but it seems to me that "creating a happy person" is not even something we know how to accomplish by choice. You can choose to create a person, create a new life, but you certainly can't guarantee that they will be happy. The human condition is hazardous, disappointing, with terrible events and fates scattered throughout it. For me, the hazard is great enough to be an antinatalist. 

I concede the bare possibility that (as you have suggested elsewhere) this life could just be a prelude to a bigger one that somehow makes up for what happens here, or the possibility that, after some friendly singularity, we could know enough about the being of our world that creating a definitely happy life really is an option. 

But neither of these corresponds to the life that we know and endure right now, which to a great extent is still about the struggle to survive rather than the pursuit of happiness. 

When you start wondering if one of your heroes is playing 11D chess when they do things that run counter to your idealized notion of who they are... it probably means you've idealized them a bit too much. Eliezer is still "just a man", as the Merovingian might say. 

You may also underestimate the kind of pressures to which he is subjected. Are you familiar with /r/sneerclub at all? This was a group of redditors united by contempt for Less Wrong rationalism, who collected dirt on anything to do with the rationalist movement, and even supplied some of it to a New York Times journalist. And that was before the current AI boom, in which even billionaires can feel threatened by him, to the point that pioneer MIRI donor Peter Thiel now calls him a legionnaire of Antichrist. 

Add to that 10+ years of daily debate on social media, and it shouldn't be surprising that any idea of always being diplomatic, has died a death of a thousand cuts. 

This seems continuous with CEV rather than being an alternative to it. CEV wants to extrapolate human values but didn't specify a way to identify them. You propose that human values can be identified via a perfected evolutionary psychology, but then acknowledge that they'll need to be extrapolated, in order to be extended to situations outside the ancestral distribution...

It may be hard to believe today, but for most of the 20th century it was considered standard wisdom that companies which had stood the test of time were wiser investments than inexperienced startups.

Aren't stock markets now dominated by massive institutional investors engaged in automated very-short-term speculation?

I'm confused by this. First of all you talk about situations in which there is a text containing multiple persons interacting, and you say an AI, in predicting the words of one of the persons, will inappropriately use information that this person would not possess in real life. But you don't give any examples of this. 

Then, we switch to situations in which an AI is not extrapolating a story, but is explicitly, all the time, in a particular persona (that of an AI assistant). And the claim is that this assistant will have a poor sense of where it ends, and the user, or the universe, begins. 

But in the scenario of an AI assistant talking with a user, the entire conversation is meant to be accessible to the assistant, so there's no information in the chat that "in real life" the assistant couldn't access. So I don't even know what the mechanism of bleed-through in an AI assistant is supposed to be?

I don't have much to say except that I think it would be good to create a bloc with the proposed goals and standards, but that it would be hard to adhere to those standards and get anywhere in today's politics. 

Also, if I was an American and interested in the politics of AI, I would be interested in the political stories surrounding the two movements that actually made a difference to executive-branch AI policy, namely effective altruism during the Biden years and e/acc during Trump 2.0. I think the EAs got in because the arrival of AI blindsided normie society and the EAs were the only ones who had a plan to deal with it, and then the e/accs managed to reverse that because the tech right wanted to get rich from a new technological revolution, and were willing to bet on Trump. 

Also, for the record, the USA actually had a state politician who was a rationalist, ten years ago. 

As the paper notes, this is part of Terry Tao's proposed strategy for resolving the Navier-Stokes millennium problem.