Here's a (messy, haphazard) list of ways a group of idealized agents could merge into a single agent:

Proposal 1: they merge into an agent which maximizes a weighted sum of their utilities. They decide on the weights using some bargaining solution.

Objection 1: this is not Pareto-optimal in the case where the starting agents have different beliefs. In that case we want:

Proposal 2: they merge into an agent which maximizes a weighted sum of their utilities, where those weights are originally set by bargaining but evolve over time depending on how accurately each original agent predicted the future.

Objection 2: this loses out on possible gains from acausal trade. E.g. if a paperclip-maximizer finds itself in a universe where it's hard to make paperclips but easy to make staples, it'd like to be able to give resources to staple-maximizers in exchange for them building more paperclips in universes where that's easier. This requires a kind of updateless decision theory:

Proposal 3: they merge into an agent which maximizes a weighted sum of their utilities (with those weights evolving over time), where the weights are set by bargaining subject to the constraint that each agent obeys commitments that logically earlier versions of itself would have made.

Objection 3: this faces the commitment races problem, where each agent wants to make earlier and earlier commitments to only accept good deals.

Proposal 4: same as proposal 3 but each agent also obeys commitments that they would have made from behind a veil of ignorance where they didn't yet know who they were or what their values were. From that position, they wouldn't have wanted to do future destructive commitment races.

Objection 4: as we take this to the limit we abstract away every aspect of each agent—their values, beliefs, position in the world, etc—until everything is decided by their prior from behind a veil of ignorance. But when you don't know who you are, or what your values are, how do you know what your prior is?

Proposal 5: all these commitments are only useful if they're credible to other agents. So, behind the veil, choose a Schelling prior which is both clearly non-cherrypicked and also easy for a wide range of agents to reason about. In other words, choose the prior which is most conducive to cooperation across the multiverse.

Okay, so basically we've ended up describing not just an ideal agent, but the ideal agent. The cost of this, of course, is that we've made it totally computationally intractable. In a later post I'll describe some approximations which might make it more relevant.