Vladimir_Nesov's Shortform

Vladimir_Nesov

This is a special post for quick takes by Vladimir_Nesov. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

The physical world is a simple thing that's not useful to study too much, once intelligence scales well past the human level. Almost all cognition is going to be about the computations designed by cognition (or their semantics), not about the physical world, which is likely to be almost entirely relegated to the role of compute substrate for the intentionally designed computations. All of the physical world might one day be about as relevant to most people as the arcane details of material manufacturing processes necessary to make a specialized industrial tool they were never aware of.

So I don't think decision theory for agent foundations should be concerned with the kinds of uncertainty that are central to thinking about the physical world. Instead, it should study reasoning about known computations, and influence of decisions on the behavior of known computations. These are not toy examples that in some ways pose unusual difficulties and in other ways are easier to handle, but rather most of what matters.

7jessicata1h

I feel like you're trying to speculate here about what happens after someone already won the war for the lightcone and is a singleton. Sure, there are 'simulation hypothesis' type considerations here that are decision relevant, but it also matters what happens before someone has conquered everything (if that's even likely). What happens at the intermediate stage where there are smarter-than-human AIs (bootstrapped by centaur systems) and they are using foreseeable quantum tech, which surely includes quantum RNG, and probably includes Bell-nonlocality (already can be done in the lab) and quantum computing (can't yet, but seems foreseeable)? The decision / game theory of this could affect who, if anyone, gets a singleton locking down a bunch of the world, and restricting who has access to which quantum operations. And what about aliens? They are probably very far away if they exist, but even an Earth-originating singleton needs to plan on the game theory of future alien encounters, and assume aliens have quantum tech. There is the "try to run away" thing and if that is the plan, why not include a quantum RNG (or other quantum tech, if available) on board of the runaway space ship? Regarding applications of quantum computing, Grover's algorithm is highly general (though not an exptime speedup), there could be more like that. And the physical world still has important consequences about which computations are important to run, in order to make good use of material / space resources. Quantum computers help with simulating physical systems.

2Vladimir_Nesov26m

The idea with the substrate/civilization split is not that the substrate owns the civilization, but that the civilization is the updateless core for the substrate, it defines the policies that the substrate follows (like in a UDT agent), without knowing some of the things the substrate gets to observe in the physical world. The civilization governance on the inside could take any form (doesn't need to be a singleton), and the substrate governance on the outside doesn't carry the historically familiar menacing connotations even if it runs the civilization in a perfect sandbox, because one way or another alignment is going to be settled at that point (which was never historically the case, and so direct control was always relevant). [...] Right, that's a thing for the outer governance to worry about (like that arcane material manufacturing process for the industrial tool), but not necessarily for the civilization.

Ok, to make my opinion on this clearer:

(a) We don't really know much about what the eventual state looks like, whether it is a singleton, how locked down, how many have access to quantum ops.

(b) My personal guess for (a) is that "stochastic Turing machines + Bell nonlocality" is a more relevant model than "deterministic Turing machines", though both are relevant.

(c) How do we figure out about (a)? It seems like we need some theory of agent foundations to think about this well. That theory shouldn't start out with strong assumptions like "only deterministic... (read more)

Vladimir_Nesov's Shortform

11

Ω 4

Vladimir_Nesov's Shortform

11

Ω 4