drnickbone

Replying toSolving the Doomsday argument

Thanks again for the useful response.

My initial argument was really a question “Is there any approach to anthropic reasoning that allows us to do basic scientific inference, but does not lead to Doomsday conclusions?” So far I’m skeptical.

The best response you’ve got is I think twofold.

Use SIA but please ignore the infinite case (even though the internal logic of SIA forces the infinite case) because we don’t know how to handle it. When applying SIA to large finite cases, truncate universes by a large volume cutoff (4d volume) rather than by a large population cutoff or large utility cutoff. Oh and ignore simulations because if you take those into account it

... (read 516 more words →)

Replying toSolving the Doomsday argument

drnickbone6y

Solving the Doomsday argument

I get that this is a consistent way of asking and answering questions, but I’m not sure this is actually helpful with doing science.

If, say, universes 1 and 2 contain TREE(3) copies of me while universes 3 and 4 contain BusyBeaver(1000) then I still don’t know which I’m more likely to be in, unless I can somehow work out which of these vast numbers is vaster. Regular scientific inference is just going to completely ignore questions as odd as this, because it simply has to. It’s going to tell me that if measurements of background radiation keep coming out at 3K, then that’s what I should assume the temperature actually is.

drnickbone7y

Solving the Doomsday argument

Thanks Stuart.

The difficulty is that, by construction, there are infinitely many copies of me in each universe (if the universes are all infinite) or there are a colossally huge number of copies of me in each universe, so big that it saturates my utility bounds (assuming that my utilities are finite and bounded, because if they’re not, the decision theory leads to chaotic results anyway).

So SIA is not an approach to anthropics (or science in general) which allows us to conclude we are probably in universe 1 or 2 (rather than 3 or 4). All SIA really says is “You are in some sort of really big or infinite universe, but beyond

drnickbone7y

Solving the Doomsday argument

Hi Stuart. It’s a while since I’ve posted.

Here’s one way of asking the question which does lead naturally to the Doomsday answer.

Consider two universes. They’re both infinite (or if you don’t like actual infinities, are very very large, so they both have a really huge number of civilisations).

In universe 1, almost all the civilisations die off before spreading through space, so that the average population of a civilisation through time is less than a trillion.

In universe 2, a fair proportion of the civilisations survive and grow to galaxy-size or bigger, so that the average population of a civilisation through time is much more than a trillion trillion.

Now consider two more universes. Universe

drnickbone11y

Debunking Fallacies in the Theory of AI Motivation

I think by "logical infallibility" you really mean "rigidity of goals" i.e. the AI is built so that it always pursues a fixed set of goals, precisely as originally coded, and has no capability to revise or modify those goals. It seems pretty clear that such "rigid goals" are dangerous unless the statement of goals is exactly in accordance with the designers' intentions and values (which is unlikely to be the case).

The problem is that an AI with "flexible" goals (ones which it can revise and re-write over time) is also dangerous, but for a rather different reason: after many iterations of goal rewrites, there is simply no telling what its goals will come to look like. A late version of the AI may well end up destroying everything that the first version (and its designers) originally cared about, because the new version cares about something very different.

Replying toIdentity and quining in UDT

drnickbone11y

Identity and quining in UDT

Consider the following decision problem which I call the "UDT anti-Newcomb problem". Omega is putting money into boxes by the usual algorithm, with one exception. It isn't simulating the player at all. Instead, it simulates what would a UDT agent do in the player's place.

This was one of my problematic problems for TDT. I also discussed some Sneaky Strategies which could allow TDT, UDT or similar agents to beat the problem.

Replying toNew(ish) AI control ideas

drnickbone11y

New(ish) AI control ideas

Presumably anything caused to exist by the AI (including copies, sub-agents, other AIs) would have to count as part of the power(AI) term? So this stops the AI spawning monsters which simply maximise U.

One problem is that any really valuable things (under U) are also likely to require high power. This could lead to an AI which knows how to cure cancer but won't tell anyone (because that will have a very high impact, hence a big power(AI) term). That situation is not going to be stable; the creators will find it irresistible to hack the U and get it to speak up.

Replying to[Link] Physics-based anthropics?

drnickbone11y

[Link] Physics-based anthropics?

I had a look at this: the KCA (Kolmogorov Complexity) approach seems to match my own thoughts best.

I'm not convinced about the "George Washington" objection. It strikes me that a program which extracts George Washington as an observer from insider a wider program "u" (modelling the universe) wouldn't be significantly shorter than a program which extracts any other human observer living at about the same time. Or indeed, any other animal meeting some crude definition of an observer.

Searching for features of human interest (like "leader of a nation") is likely to be pretty complicated, and require a long program. To reduce the program size as much as possible, it ought to... (read more)

Replying to[Link] Physics-based anthropics?

drnickbone11y

[Link] Physics-based anthropics?

Upvoted for acknowledging a counterintuitive consequence, and "biting the bullet".

One of the most striking things about anthropics is that (seemingly) whatever approach is taken, there are very weird conclusions. For example: Doomsday arguments, Simulation arguments, Boltzmann brains, or a priori certainties that the universe is infinite. Sometimes all at once.

Replying to2014 Less Wrong Census/Survey

drnickbone11y

2014 Less Wrong Census/Survey

Taken survey.

[LINK] IBM simulate a "brain" with 500 billion neurons and 100 trillion synapses

drnickbone

13y

Recent article in The New Yorker:

http://www.newyorker.com/online/blogs/newsdesk/2012/11/ibm-brain-simulation-compass.html

Here is the research report from IBM, with the simple title "10^14":

http://www.modha.org/blog/SC12/RJ10502.pdf

It's nothing like a real brain simulation, of course, but illustrates that hardware to do this is getting very close.

There is likely to be quite a long overhang between the hardware and the software...

How do we really escape Prisoners' Dilemmas?

drnickbone

13y

This discussion article was provoked in part by Yvain's post on Main a few weeks ago, and some of the follow-up comments.

EDIT: I've also just noticed that there was a recent sequence rerun on the point about finite iterations. My bad: I simply didn't see the rerun article, as it had already slipped down a couple of pages when I posted. If you down-voted (or didn't read) out of a feeling of "Didn't we just do this?" then sorry.

In any case, one of my main motivations for running this article was point 5 (Does an environment of commitment and reputation create the background against which TDT - or something like it - can easily evolve?)... (read 2388 more words →)

Problematic Problems for TDT

drnickbone

14y

A key goal of Less Wrong's "advanced" decision theories (like TDT, UDT and ADT) is that they should out-perform standard decision theories (such as CDT) in contexts where another agent has access to the decider's code, or can otherwise predict the decider's behaviour. In particular, agents who run these theories will one-box on Newcomb's problem, and so generally make more money than agents which two-box. Slightly surprisingly, they may well continue to one-box even if the boxes are transparent, and even if the predictor Omega makes occasional errors (a problem due to Gary Drescher, which Eliezer has described as equivalent to "counterfactual mugging"). More generally, these agents behave like a CDT agent will wish it had pre-committed itself... (read 919 more words →)

293

Sneaky Strategies for TDT

drnickbone

14y

My previous article on problematic problems attracted quite a lot of feedback and comment. One of the questions it posed (2) was whether TDT should do something other than the initial analysis suggests.

I've had a couple of ideas on that, partly in response to the comments. I'm posting some follow-up thoughts in the hope that they might help clarify the original problems. Basically, it seems there are some sneaky things that TDT could try to do, but mostly they are not robust to slightly different variants of the problems. Interestingly, some of these variants look even "fairer", since they contain no mentions of TDT anywhere in the problem statement.

An alternative approach could be that... (read 1533 more words →)

Self-Indication Assumption - Still Doomed

drnickbone

14y

I recently posted a discussion article on the Doomsday Argument (DA) and Strong Self-Sampling Assumption. See http://lesswrong.com/lw/9im/doomsday_argument_with_strong_selfsampling/

This new post is related to another part of the literature concerning the Doomsday Argument - the Self Indication Assumption or SIA. For those not familiar, the SIA says (roughly) that I would be more likely to exist if the world contains a large number of observers. So, when taking into account the evidence that I exist, this should shift my probability assessments towards models of the world with more observers.

Further, on first glance, it looks like the SIA shift can be arranged to exactly counteract the effect of the DA shift. Consider, for instance, these two hypotheses:

H1. Across... (read 697 more words →)

Doomsday Argument with Strong Self-Sampling Assumption

drnickbone

14y

Hello everyone; I'm new to the forum, and have been advised to post this in the "discussion" section. Hope this is OK.

I've found some references to discussions here on Brandon Carter / John Leslie's "Doomsday Argument" and they seemed well-informed. One thing I've noticed about the argument though (but haven't seen discussed before) is that it can be made much sharper by assuming that we are making random *observations*, rather than just that we are a random *observer*.

For those who know the literature, this is a form of Nick Bostrom's Strong Self-Sampling Assumption as opposed to the (basic) Self-Sampling Assumption. Oddly enough, Bostrom discusses SSSA quite a lot in connection with the Doomsday Argument,... (read 511 more words →)

LESSWRONG
LW

LESSWRONG
LW

Problematic Problems for TDT

Doomsday Argument with Strong Self-Sampling Assumption

Sneaky Strategies for TDT

[LINK] IBM simulate a "brain" with 500 billion neurons and 100 trillion synapses

drnickbone

drnickbone

[LINK] IBM simulate a "brain" with 500 billion neurons and 100 trillion synapses

How do we really escape Prisoners' Dilemmas?

Problematic Problems for TDT

Sneaky Strategies for TDT

Self-Indication Assumption - Still Doomed

Doomsday Argument with Strong Self-Sampling Assumption

drnickbone

Problematic Problems for TDT

Doomsday Argument with Strong Self-Sampling Assumption

Sneaky Strategies for TDT

[LINK] IBM simulate a "brain" with 500 billion neurons and 100 trillion synapses

drnickbone

drnickbone

[LINK] IBM simulate a "brain" with 500 billion neurons and 100 trillion synapses

How do we really escape Prisoners' Dilemmas?

Problematic Problems for TDT

Sneaky Strategies for TDT

Self-Indication Assumption - Still Doomed

Doomsday Argument with Strong Self-Sampling Assumption