All of M. Y. Zuo's Comments + Replies

Variables Don't Represent The Physical World (And That's OK)

Ah, I understand what your getting at now dxu, thanks for taking the time clarify. Yes, there likely are not extra bits of information hiding away somewhere, unless there really are hidden parameters in space-time (as one of the possible resolutions to Bell’s theorem). 

When I said ‘there will always be’ I meant it as ‘any conceivable observer will always encounter an environment with extra bits of information outside of their observational capacity’, and thus beyond any  model or mapping. I can see how it could have been misinterpreted.


In r... (read more)

Variables Don't Represent The Physical World (And That's OK)

Well, if we were to know that assertion is unprovable, or undecidable, then we can treat it as any other unprovable assertion. 

Announcing the Nuclear Risk Forecasting Tournament

Interesting idea, though the first round seems unverifiable. “How many nuclear weapons will states possess on December 31, 2022?” 

1MichaelA4moNot sure what you mean by that being unverifiable? The question says: FAS update their estimates fairly regularly - here [] are their estimates as of May (that link is also provided earlier in the question text). Though I do realise now that they're extremely unlikely to update their numbers on December 31 specifically, and maybe not even in December 2022 at all. I'll look into the best way to tweak the question in light of that. If that's what you meant, thanks for the feedback! (I do expect there'll be various minor issues like that, and we hope the community catches them quickly so we can tweak the questions to fix them. This was also one reason for showing some questions before they "open".)

Referencing Leo Szilard is amusing for this topic as his moment of genius insight, that the atomic nucleus can be split to generate enormous explosions, is one of those few ideas so genuinely beyond the then current paradigm (1930’s) that it seems like real precognition.  

Allegedly he spent a significant fraction of every day sitting in a hotel bathtub in rumination, and he lived permanently in hotels for decades. I assume that is how he developed that depth of thinking. 

2bluefalcon4moSzilard got his key insight precisely by being pissed at Rutherford's hubris.
Variables Don't Represent The Physical World (And That's OK)

It seems that your comment got cut off at the end there.

How to make errands fun

“All Nature is but art, unknown to thee
All chance, direction, which thou canst not see;
All discord, harmony not understood; 
All partial evil, universal good.”

  • Alexander Pope
Variables Don't Represent The Physical World (And That's OK)

When you said ’not directly extensible’ I understood that as meaning ‘logistically impossible to perfectly map onto a model communicable to humans’. With the fishes fluctuating in weight, in reality, between and during every observation, and between every batch. So even if, perfect, weight information was obtained somehow, that would only be for that specific Planck second. And then averaging, etc., will always have some error inherently. So every step on the way there is a ’loose coupling’, so that the final product, a mental-model of what we just read, i... (read more)

1JBlack4moThe way I use "extensibility" here is between two different models of reality, and just means that one can be obtained from the other merely by adding details to it without removing any parts of it. In this case I'm considering two models, both with abstractions such as the idea that "fish" exist as distinct parts of the universe, have definite "weights" that can be "measured", and so on. One model is more abstract: there is a "population weight distribution" from which fish weights at some particular time are randomly drawn. This distribution has some free parameters, affected by the history of the tank. One model is more fine-grained: there are a bunch of individual fish, each with their own weights, presumably determined by their own individual life circumstances. The concept of "population weight distribution" does not exist in the finer-grained model at all. There is no "abstract" population apart from the actual population of 100 fish in the tank. So yes, in that sense the "population mean" variable does not directly represent anything in the physical world (or at least our finer-grained model of it). This does not make it useless. Its presence in the more abstract model allows us to make predictions about other tanks that we have not yet observed, and the finer-grained model does not.

You are misunderstanding the post. There are no "extra bits of information" hiding anywhere in reality; where the "extra bits of information" are lurking is within the implicit assumptions you made when you constructed your model the way you did.

As long as your model is making use of abstractions--that is, using "summary data" to create and work with a lower-dimensional representation of reality than would be obtained by meticulously tracking every variable of relevance--you are implicitly making a choice about what information you are summarizing and how ... (read more)

Variables Don't Represent The Physical World (And That's OK)

Right, for determinism to work in practice, some method of determining that ‘previous world state’ must be viable. But if there are no viable methods, and if somehow that can be proven, then we can be confident that determinism is impossible, or at the very least, that determinism is a faulty idea.

4TAG4moWhat you are talking about is prediction. Determinism also needs to be distinguished from predictability. A universe that unfolds deterministically is a universe that can be predicted by an omniscient being which can both capture a snapshot of all the causally relevant events, and have a perfect knowledge of the laws of physics. The existence of such a predictor, known as a Laplace's demon is not a prerequisite for the actual existence of determinism, it is just a way of explaining the concept. It is not contradictory ro assert that the universe is deterministic but unpredictable.

You‘ve really put some thought into this, thanks for sharing.

Though I don’t want to make a critique I would like to save you a bit of future trouble as a courtesy from someone who has trodden down the same path.

The issue with basing a philosophy on Mozi is that there are no ‘fixed standards’. All standards, like the rest of the universe, are forever in flux. Universal frameworks can not exist.

For the next stage I found reading Liezi was helpful.

3Bae's Theorem4moThanks for the feedback! Basing it on Mohism is more of an aesthetic decision than anything; if classical Mohism has an issue then Neo-Mohism should set out to solve it. :) I think there's a difference between "no fixed standards" and "the ability to update standards in light of new evidence". Neo-Mohism is definitely about "strong opinions, weakly held" kind of thing. The standards it sets forth are only to be overturned by failing a test, and until then should be treated as the best answer so far.
Variables Don't Represent The Physical World (And That's OK)

Therefore, determinism is impossible? You’ve demonstrated quite a neat way of showing that reality is of unbounded complexity whereas the human nervous system is of course finite and as such everything we ‘know’, and everything that can be ‘known’, necessarily is, in some portion, illusory.

3TAG4moDeterminism usually refers to a world state being determined by a previous one, not the ability to make prefect maps of the world.
3JBlack4moI don't see any implications for determinism here, or even for complexity. It's just a statement that these abstract models (and many others) that we commonly use are not directly extensible into the finer-grained model. One thing to note is that in both cases, there are alternative abstract models with variables that are fully specified by the finer-grained reality model. They're just less convenient to use.
Reward Is Not Enough

That is true, the desired characteristics may not develop as one would hope in the real world. Though that is the case for all training, not just AGI. Humans, animals, even plants, do not always develop along optimal lines even with the best ‘training’, when exposed to the real environment. Perhaps the solution you are seeking for, one without the risk of error, does not exist. 

Reward Is Not Enough

Could the hypothetical AGI be developed in a simulated environment and trained with proportionally lower consequences?

2Steven Byrnes4moI'm all for doing lots of testing in simulated environments, but the real world is a whole lot bigger and more open and different than any simulation. Goals / motivations developed in a simulated environment might or might not transfer to the real world in the way you, the designer, were expecting. So, maybe, but for now I would call that "an intriguing research direction" rather than "a solution".
The Generalized Product Rule

“We can group projects into subprojects without changing the overall return”

What if this were not true? Would that make the problem intractable?

Reward Is Not Enough

Interesting post!

Query: How do you define ‘feasibly’? as in ‘Incentive landscapes that can’t feasibly be induced by a reward function’

As from my perspective all possible incentive landscapes can be induced by reward, with sufficient time and energy. Of course a large set of these are beyond the capacity of present human civilization.

2Steven Byrnes4moRight, the word "feasibly" is referring to the bullet point that starts "Maybe “Reward is connected to the abstract concept of ‘I want to be able to sing well’?”". Here's a little toy example we can run with: teaching an AGI "don't kill all humans". So there are three approaches to reward design that I can think of, and none of them seem to offer a feasible way to do this (at least, not with currently-known techniques): 1. The agent learns by experiencing the reward. This doesn't work for "don't kill all humans" because when the reward happens it's too late. 2. The reward calculator is sophisticated enough to understand what the agent is thinking, and issue rewards proportionate to the probability that the current thoughts and plans will eventually lead to the result-in-question happening. So the AGI thinks "hmm, maybe I'll blow up the sun", and the reward calculator recognizes that merely thinking that thought just now incrementally increased the probability that the AGI will kill all humans, and so it issues a negative reward. This is tricky because the reward calculator needs to have an intelligent understanding of the world, and of the AGI's thoughts. So basically the reward calculator is itself an AGI, and now we need to figure out its rewards. I'm personally quite pessimistic about approaches that involve towers-of-AGIs-supervising-other-AGIs, for reasons in section 3.2 here [] , although other people would disagree with me on that (partly because they are assuming different AGI development paths and architectures than I am [] ). 3. Same as above, but instead of a separate reward calculator estimating the probability that a thought or plan will lead to the result-in-question, we all