To those whose formalisms still depend on simulating the universe from the beginning (those who don't, ignore this post, I'm not trying to correct connectionism here):

It doesn't matter how smart a being is; Even temporarily assuming superdeterminism[1], Laplace's demon[2] still cannot fit in the universe it's attempting to simulate - it'd be even more extreme than Maxwell's demon[3], which is also forbidden even in a classical deterministic universe, I think. There's cryptographic amounts of chaos-induced entropy in the way. Impressively cryptographic amounts, actually.

There aren't simplifications that let you get around this, you have to simulate probabilistically to deal with the fact that you can never exactly guess what chaos did in the blurred-out details. Everything has sensitive dependence on initial conditions - the amount you can get around that by being smart is significant, but bumping a butterfly to intentionally change the course of history in a specific, intentional way ain't ever happening without the bump transferring nanites, because the atmosphere is far too chaotic. You need active control to make small energy transfers accumulate into large ones when trying to do open-loop control through a system which is in a chaotic fluid regime, eg the atmosphere; you can't pre-model it and bump it in ways that accumulate into perfection, there's too much chaos and the slightest amount of noise in your sim will result in accumulation into error.

A key point here: it really doesn't matter how smart you are. You can do all sorts of crazy stuff, don't get me wrong, intelligence lets you simplify systems where there isn't chaos. But you can't simulate 100 copies of the universe in your head using 100 watts. or 100 gigawatts. or 100exawatts, and expect to get something that exactly matches. It cannot be done, no matter how big your brain is. There's too much chaos.

Though there might be larger scale structures in things like life, since life is in fact built out of resistance to error accumulation. But it's still noisy, and the maximum you can constrain claims about it is still not favorable towards formalisms that simulate from the beginning.

If this changes your plans at all

I'd suggest looking into whether you can generalize your formalism so it works on arbitrarily small or large patches of spacetime with boundary conditions, and then focus on how to ensure that you find the bubbles of spacetime that the ai and user are in assuming you're starting inference from the middle of a patch of spacetime. Eg, Carado's plan for how to do that seems like a starter, but it's underestimating some problems. I've talked to her about it and made this claim directly. Just thought I'd also mention it in public.

Again, you can do ridiculously well at optimized cause-effect simulation, but if your formalism requires simulating from the beginning in order to identify a specific being, you're going to have a bad time trying to make it useful to finite superintelligences, who can never simulate the universe from the beginning, even if they are very very large.

  1. ^

    From wikipedia, "Superdeterminism":

    In quantum mechanics, superdeterminism is a loophole in Bell's theorem. By postulating that all systems being measured are correlated with the choices of which measurements to make on them, the assumptions of the theorem are no longer fulfilled. A hidden variables theory which is superdeterministic can thus fulfill Bell's notion of local causality and still violate the inequalities derived from Bell's theorem.[1] This makes it possible to construct a local hidden-variable theory that reproduces the predictions of quantum mechanics, for which a few toy models have been proposed.[2][3][4] In addition to being deterministic, superdeterministic models also postulate correlations between the state that is measured and the measurement setting.

  2. ^

    From wikipedia, "Laplace's demon":

    In the history of science, Laplace's demon was a notable published articulation of causal determinism on a scientific basis by Pierre-Simon Laplace in 1814.[1] According to determinism, if someone (the demon) knows the precise location and momentum of every atom in the universe, their past and future values for any given time are entailed; they can be calculated from the laws of classical mechanics.[2]

    Discoveries and theories in the decades following suggest that some elements of Laplace's original writing are wrong or incompatible with our universe. For example, irreversible processes in thermodynamics suggest that Laplace's "demon" could not reconstruct past positions and momenta from the current state.

  3. ^

    From wikipedia, "Maxwell's demon":

    Maxwell's demon is a thought experiment that would hypothetically violate the second law of thermodynamics. It was proposed by the physicist James Clerk Maxwell in 1867.[1] In his first letter Maxwell called the demon a "finite being", while the Daemon name was first used by Lord Kelvin.[2]

    In the thought experiment, a demon controls a small massless door between two chambers of gas. As individual gas molecules (or atoms) approach the door, the demon quickly opens and closes the door to allow only fast-moving molecules to pass through in one direction, and only slow-moving molecules to pass through in the other. Because the kinetic temperature of a gas depends on the velocities of its constituent molecules, the demon's actions cause one chamber to warm up and the other to cool down. This would decrease the total entropy of the system, without applying any work, thereby violating the second law of thermodynamics.



New Comment
15 comments, sorted by Click to highlight new comments since: Today at 6:17 PM

(cross-posted as a top-level post on my blog)

QACI and plausibly PreDCA rely on a true name of phenomena in the real world using solomonoff induction, and thus talk about locating them in a theoretical giant computation of the universe, from the beginning. it's reasonable to be concerned that there isn't enough compute for an aligned AI to actually do this. however, i have two responses:

  • isn't there enough compute? supposedly, our past lightcone is a lot smaller than our future lightcone, and quantum computers seem to work. this is evidence that we can, at least in theory, build within our future lightcone a quantum computer simulating our past lightcone. the major hurdle here would be "finding out" a fully explanatory "initial seed" of the universe, which could take exponential time, but also could maybe not.
  • we don't need to simulate past lightcone. if you ask me what my neighbor was thinking yesterday at noon, the answer is that i don't know! the world might be way too complex to figure that out without simulating it and scanning his brain. however, i have a reasonable distribution over guesses. he was more likely to think about french things than korean things. he was more likely to think about his family than my family. et cetera. an aligned superintelligence can hold an ever increasingly refined distribution of guesses, and then maximize the expected utility of utility functions corresponding to each guess.

you've thoroughly convinced me that your formalisms do not break in the way I thought they did because of the limitation I'm referencing in this post.

That said, I still think point 1 is invalid: no, I don't think any computational system until the end of time will know to 5 decimal places what temperature it was 500 feet in the air off the instantaneous shape of the surface of the ocean the exact moment that the apple made contact with newton's scalp skin. it's just too chaotic.

Maybe you can manage three, or something. Or maybe I'm wrong and you can actually get more than 5. but then I can go further and say you definitely can't get ten decimal places, or fifteen. There's just no way you can collect all of the thermal noise from that moment in time and run it backwards. I think. well, I guess I can imagine that this might not be true; there are ways the past might be unique and possible to exactly infer, and I definitely believe the past is quite unique at the level of humans who had any significant impact on history at all, so ancestor sims are in fact probably possible.

But as you say - gotta use simplifying abstraction for that to work.

"just too chaotic" is covered by "entire past lightcone". no matter how chaotic, the past lightcone is one whole computation which you can just run naively, if you've got room. you get all decimal places, just like an agent in conway's game of life can build a complete exact simulation of its past if it's got room in the future.

(yes, maybe quantum breaks this)

Can you link specifically an example of what you are criticizing here? Which formalisms require simulating the universe from the beginning to identify a specific being? (as opposed to, e.g. some probabilistic approximation?)

(trying to think of examples, maybe Aumann's theorem qualifies? But I'm not sure, and not sure what else does?)

PreDCA and QACI are the main things I have in mind.

Just reviewed PreDCA, and while I consider it to be a clever solution to the wrong* problem, I don't see how it requires perfect physical modelling? (maybe geting a perfect answer would require perfect modelling, but an approximate answer does not).

QACI seemed obviously overcomplicated to me, don't care enough to really check if it requires perfect modelling, though I'm not sure I would call that MIRI-style.

*to clarify, I think Pre-DCA is trying to solve both a) "how does the AI identify a target to help" and b) "how does it help them", and it is a clever solution to problem (a) for a clean sheet from-scratch AI but that is the wrong problem since it's more likely an AI would be trained as a non-agent or weaker agent before being made a strong agent so would already know what humans are, and it is a clever but inadequate attempt to solve (b) which is one of the right problems. 

THIS. I have been trying to make this point for a while now: there are limits to what intelligence can accomplish. Many of the plans I hear about AGIs taking over the world assume that the power is unlimited and anything can be computed. 

Don't get confused now though, not being able to simulate the universe from the beginning doesn't mean an AI can't take over the world unexpectedly. It does mean the diamondoid concern is probably a bit overhyped, but there's plenty of other concerning nanoscale life that could be generated by a superplanner and would wipe out humanity.

Oh yes, I don't deny that, I think we agree. I simply think it is a good sanity practice to call bullshit on those overhyped plans. If people were more sceptical on those SciFi scenarios they would also probability update to lower P(doom) estimates.

I do not agree, and this is coming from someone who has much lower P(doom) estimates and has serious issues with Yudkowsky's epistemics.

The real issue I have the the nanotechnology plans and fast takeoff plans is that they require more assumptions than Yudkowsky realizes, and he has a problem of overweighting their probabilities compared to what we actually see today.

They're not magical, just way overweighted on their probability mass, IMO.

I don't see how we disagree here? Maybe it's the use of the word magical? I don't intend to use it in the sense "not allowed by the laws of physics", I am happy to replace that by overweighted probability mass if you think that's more accurate.

Maybe it's the use of the word magical? I don't intend to use it in the sense "not allowed by the laws of physics"

Yes, that was my issue. From my vantage point, the tone of the comment implied that Yudkowskian foom scenarios were downright impossible, which wasn't the case.

That stated, it looks like we came to an agreement here, so thanks for talking.

Would your argument hold if the world were partially coarse-grained predictable from inside?

I'm specifically criticizing something I've seen in the formal alignment formalisms. coarse graining is not relevant - only things that need to find the ai in a world sim from scratch have the problem I'm describing. if you can find the ai in a finite sim that just stores 4d boundary conditions at the edges in all directions, then you don't have this problem.

"Generative Models of Huge Objects" looks interesting from the perspective of approximating the universe...