All of siIver's Comments + Replies

Questions about AGI's Importance

It doesn't really matter whether the AI uses their full computational capacity. If the AI has a 100000 times larger capacity (which is again a conservative lower bound) and it only uses 1% of it, it will still be 1000 as smart as the human's full capacity.

AGI's algorithm will be better, because it has instant access to more facts than any human has time to memorize, and it will not have all of the biases that humans have. The entire point of the sequences is to list dozens of ways that the human brain reliably fails.

If the advantage is speed, then in one year an AI that thinks 10,000x faster could be as productive as a person who lives for 10,000 years. Something like that. Or as productive as one year each from 10,000 people. But a person could live to 10,000 and not be very productive, ever. That's easy, right? Because they get stuck, unhappy, bored, superstitious ... all kinds of things can go wrong with their thinking. If AGI only has a speed advantage, that won't make it immune to dishonesty, wishful thinking, etc. Right? Humans have fast access to facts via google, databases, and other tools, so memorizing isn't crucial. I thought they talked about things like biases. Couldn't an AGI be biased, too?
Questions about AGI's Importance


"[the brain] is sending signals at a millionth the speed of light, firing at 100 Hz, and even in heat dissipation [...] 50000 times the thermodynamic minimum energy expenditure per binary swtich operation"

AI will be quantitatively smarter because it'll be able to think over 10000 times faster (arbitrary conservative lower bound) and it will be qualitatively smarter because its software will be built by an algoirthm far better than evolution

My calculator can add large numbers much, much faster than I. That doesn't make it "quantitatively smarter". Given that no one has any idea about what that algorithm might look like, statements like this seem a bit premature.
i think humans don't use their full computational capacity. why expect an AGI to? in what way do you think AGI will have a better algorithm than humans? what sort of differences do you have in mind?
Instrumental Rationality Sequence Finished! (w/ caveats)

"Less than a third of students by their own self-appointed worst-case estimate *1."

missing a word here, I think.

Thanks! Will edit.
Is life worth living?

re-live. Although I'd rather live the same amount of time from now onward.

Sam Harris and Scott Adams debate Trump: a model rationalist disagreement

First question: I know you admire Trump's persuasion skills, but what I want to know is why you think he's a good person/president etc.

Answer: [talks about Trump's persuasion skills]

Yeah, okay.

Daniel Dewey on MIRI's Highly Reliable Agent Design Work

This is an exceptionally well reasoned article, I'd say. Particular props to the appropriate amount of uncertainty.

Against lone wolf self-improvement

Well, if you put it like that I fully agree. Generally, I believe that "if it doesn't work, try something else" isn't followed as often as it should. There's probably a fair number of people who'd benefit from following this article's advice.

Against lone wolf self-improvement

I don't quite know how to make this response more sophisticated than "I don't think this is true". It seems to me that whether classes ore lone-wolf improvement is better is a pretty complex question and the answer is fairly balanced, though overall I'd give the edge to lone-wolf.

Yeah, that seems to be the biggest flaw in the post. I shouldn't have addressed it to everyone, it's intended mostly for people suffering from "akrasia". I.e. if lone wolf is working for you, ignore the post. If it isn't, notice that and change course.

I don't know what our terminal goals are (more precisely than "positive emotions"). I think it doesn't matter insofar as the answer to "what should we do" is "work on AI alignment" either way. Modulo that, yeah there are some open questions.

On the thesis of suffering requiring higher order cognition in particular, I have to say that sounds incredibly implausible (for I think fairly obvious reasons involving evolution).

Announcing AASAA - Accelerating AI Safety Adoption in Academia (and elsewhere)

This looks solid.

Can you go into a bit of detail on the level / spectrum of difficulty of the courses you're aiming for, and the background knowledge that'll be expected? I suspect you don't want to discourage people, but realistically speaking, it can hardly be low enough to allow everyone who's interested to participate meaningfully.

Thank you! Difficulty/prerequisites is one of the uncertainties that will have to be addressed. Some AI safety only requires algebra skills while other stuff needs logic/ML/RL/category theory/other, and then there is stuff that isn't formalized at all. But there are other applied mathematics fields with this problem, and I expect that we can steal a solution by having a look there.
Destroying the Utility Monster—An Alternative Formation of Utility

Yeah, you're of course right. In the back of my mind I realized that the point I was making was flawed even as I was writing it. A much weaker version of the same would have been correct, "you should at least question whether your intuition is wrong." In this case it's just very obvious to me me that there is nothing to be fixed about utilitarianism.

Anyway, yeah, it wasn't a good reply.

Destroying the Utility Monster—An Alternative Formation of Utility

This is the ultimate example of... there should be a name for this.

You figure out that something is true, like utilitarianism. Then you find a result that seems counter intuitive. Rather than going "huh, I guess my intuition was wrong, interesting" you go "LET ME FIX THAT" and change the system so that it does what you want...

man, if you trust your intuition more than the system, then there is no reason to have a system in the first place. Just do what is intuitive.

The whole point of having a system like utilitarinism is that we can fig... (read more)

Sometimes when explicit reasoning and intuition conflict, intuition turns out to be right, and there is a flaw in the reasoning. There's nothing wrong with using intuition to guide yourself in questioning a conclusion you reached through explicit reasoning. That said, DragonGod did an exceptionally terrible job of this.
The term you're looking for is "apologist".
No. I am improving the existing system. All individuals have the same capacity for desire.
That looks like a category error. What does it mean for utilitarianism to be "true"? It's not a feature of the territory. Trust is not all-or-nothing. Putting ALL your trust into the system -- no sanity checks, no nothing -- seems likely to lead to regular epic fails.
I think the name you are looking for is ad hoc hypothesis [].
Existential risk from AI without an intelligence explosion

This seems like something we should talk about more.

Although, afaik there shouldn't be a decision between motivation selection and capability controlling measures – the former is obviously the more important part, but you can also always "box" the AI in addition (insofar as that's compatible with what you want it to do).

AGI and Mainstream Culture

That sounds dangerously like justifying inaction.

Literally speaking, I don't disagree. It's possible that spreading awareness has a net negative outcome. It's just not likely. I don't discourage looking into the question, and if facts start pointing the other way I can be convinced. But while we're still vaguely uncertain, we should act on what seems more likely right now.

I would never argue for inaction. I think this line of thinking would argue for efforts being made to make sure any AGI researchers were educated but no efforts were made to make sure other people were (in the most extreme case). But yep we may as well carry on as we are for the moment.
AGI and Mainstream Culture

I guess it's a legit argument, but it doesn't have the research aspect and it's a sample size of one.

(Un)luckily we don't have many examples of potentially world destroying arms races. We might have to adopt the inside view. We'd have to look at how much mutual trust and co-operation there is currently for various things. Beyond my current knowledge. By the research aspect, I think research can be done without the public having a good understanding of the problems. E.g. cern/CRISPR. I can also think of other bad outcomes of the the public having an understanding of AIrisk. It might be used as another stick to take away freedoms, see the war on terrorism and drugs for examples of the public's fears. Convincing the general public of AIrisk seems like shouting fire in crowded movie theatre, it is bound to have a large and chaotic impact on society. This is the best steelman of this argument, that I can think of at the moment. I'm not sure I'm convinced. But I do think we should put more brain power into this question.
AGI and Mainstream Culture

This just seems like an incredibly weak argument to me. A) it seems to me that prior research will be influenced much more than the probability for an arms race, because the first is more directly linked to public perception, B) we're mostly trying to spread awareness of the risk not the capability, and C) how do we even know that more awareness on the top political levels would lead to a higher probability for an arms race, rather than a higher probability for an international cooperation?

I feel like raising awareness has a very clear and fairly safe upside, while the downside is highly uncertain.

Why do you think this time is different to the nuclear arms race? The federation of atomic scientists [] didn't prevent it. It only slackened because russia ran ouf of steam.
Reaching out to people with the problems of friendly AI

Pretty sure it is. You have two factors, increasing the awareness of AI risk and of AI specifically. The first is good, the second may be bad but since the set of people caring about AI generally is so much larger, the second is also much less important.

There are roughly 3 actions: 1) Tell no one and work in secret 2) Tell people that are close to working on AGI 3) Tell everyone Telling everyone has some benefits in maybe getting people that are close to working on AGI that you wouldn't get otherwise and maybe making it more convincing. It might be most efficient as well. While lots of people care about AI I think establishment is probably still a bit jaded from the hype before the AI winters. I think the number of people who think about artificial general intelligence is a small subset of the number of of people involved in weak AI. So I think I am less sure than you and I'm going to think about what the second option might look like.
Reaching out to people with the problems of friendly AI

I whole-heartedly agree with you, but I don't have anything better than "tell everyone you know about it." On that topic, what do you think is the best link to send to people? I use this, but it's not ideal.

This is the exact topic I'm thinking a lot about, thanks for the link! I've wrote my own essay for a general audience but it seems ineffective. I knew about the Wait but why blog post, but there must be better approaches possible. What I find hard to understand is that there have been multiple best-selling books about the topic, but still no general alarm is raised and the topic is not discussed in e.g. politics. I would be interested in why this paradox exists, and also how to fix it. Is there any more information about reaching out to a general audience on Lesswrong? I've not been able to find it using the search function etc. The reason I'm interested is twofold: 1) If we convince a general audience that we face an important and understudied issue, I expect them to fund research into it several orders of magnitude more generously, which should help enormously in reducing the X-risk (I'm not working in the field myself). 2) If we convince a general audience that we face an important and understudied issue, they may convince governing bodies to regulate, which I think would be wise. I've heard the following counterarguments before, but didn't find them convincing. If someone would want to convince me that convincing the public about AGI risk is not a good idea, these are places to start: 1) General audiences might start pressing for regulation which could delay AI research in general and/or AGI. That's true and indeed a real problem, since all the potential positive aspects of AI/AGI (which may be enormous) cannot be applied yet. However, in my opinion the argument is not sufficient because: A) AGI existential risk is so high and important that reducing it is more important than AI/AGI delay, and B) Increased knowledge of AGI will also increase general AI interest, and this effect could outweigh the delay that regulation might cause. 2) AGI worries from the general public could make AI researchers more secretive and less cooperative in working together wi
Why do we think most AIs unintentionally created by humans would create a worse world, when the human mind was designed by random mutations and natural selection, and created a better world?


Q: Evolution is a dumb algorithm, yet it produced halfway functional minds. How can it be that the problem isn't easy for humans, who are much smarter than evolution?

A: Evolution's output is not just one functional mind. Evolution put out billions of different minds, an extreme minority of them being functional. If we had a billion years of time and had a trillion chances to get it right, the problem would be easy. Since we only have around 30 years and exactly 1 chance, the problem is hard.

Evolution also had 1 chance, in the sense that the first intelligent species created would take over the world and reform it very quickly, leaving no time for evolution to try any other mind-design. I'm pretty sure there will be no other intelligent species that evolves by pure natural selection after humanity - unless it's part of an experiment run by humans. Evolution had a lot of chances to try to create a functional intelligence, but as for the friendliness problem, it had only one chance. The reason being, a faulty intelligence will die out soon enough, and give evolution time to design a better one, but a working paperclip maximizer is quite capable of surviving and reproducing and eliminating any other attempts at intelligence.
There is No Akrasia

I often ask myself the question of "is this really a thing" when it comes to high level concepts like this. I'm very unsure on Akrasia, and you make a decent enough argument. It could very well not actually be a thing (beyond referring to sub-things).

More importantly, though, even if it were a thing, I agree that the strategy you suggest of focusing on the smaller issues is likely the better one.

Nate Soares' Replacing Guilt Series compiled in epub Format

I read the first post, which is excellent. Thanks for sharing.

MIRI: Decisions are for making bad outcomes inconsistent

Thanks! So UDT is integrated. That's good to hear.

MIRI: Decisions are for making bad outcomes inconsistent

Can someone briefly explain to me the difference between functional and updateless decision theory / where FDT performs better? That would be much appreciated. I have not yet read FDT because it does not mention UDT (I checked) and I want to understand why UDT needs "fixing" before I invest the time.

4Rob Bensinger5y
"UDT" is ambiguous and has been used to refer to a number of different approaches; "FDT" is a new name for the central cluster of UDT-ish theories (excluding some similar theories like TDT), intended to be less ambiguous and easier to explain (especially to working decision theorists). In part it's easier to explain because it's formulated in a more CDT-like fashion (whereas Wei Dai's formulations are more EDT-like), and in part it's easier to explain because it builds in less content: accepting FDT doesn't necessarily require a commitment to some of the philosophical ideas associated with updatelessness and logical prior probability that MIRI, Wei Dai, or other FDT proponents happen to accept. In particular, some of Nate's claims in the linked post are stronger than is strictly required for FDT.
My impression (based in part on this humorous piece [] ) is that FDT is primarily a better formulation than UDT & TDT, but doesn't necessarily make better decisions. Ah. And the paper [] says (emphasis mine)
OpenAI makes humanity less safe

Ugh. When I heard about this first I naively thought it was great news. Now I see it's a much harder question.

Alien Implant: Newcomb's Smoking Lesion

In my situation, it is the same: you can "determine" whether your dial is set to the first or second position by making a decision about whether to smoke.


You can not. You can't.

I'm struggling with this reply. I almost decided to stop trying to convince you. I will try one more time, but I need you to consider the possibility that you are wrong before you continue to the next paragraph. Consider the outside view: if you were right, Yudkowksy would be wrong, Anna would be wrong, everyone who read your post here and didn't upvote this revoluti... (read more)

Also, about this: This outside view is too limited; there are plenty of extremely intelligent people outside Less Wrong circles who agree with me. This is why I said from the beginning that the common view here came from the desire to agree with Eliezer. Notice that no one would agree and upvote without first having to disagree with all those others, and they are unlikely to do that because they have the limited outside view you mention here: they would not trust themselves to agree with me, even if it was objectively convincing. Scott Alexander is probably one of the most unbiased people ever to be involved with Less Wrong. Look at this comment []: Likewise, if they don't like not smoking in the situation here, they can die of cancer.
"You have the option to change the kind of person you are, i.e. decide you'll one-box in Newcomb's problem at any given moment before you are confronted with it (such as right now), therefore you causally determine how much money you will receive once you play it in the future." If I am not the kind of person who would accept this reasoning, I can no more make myself into the kind of person who would accept this reasoning (even right now), than I can make myself into a person who has the dial set to the second position. Both are facts about the world: whether you have the dial set in a certain position, and whether you are the kind of person who could accept that reasoning. And on the other hand, I can accept the reasoning, and I can choose not to smoke: I will equally be the kind of person who takes one box, and I will be a person who would have the dial in the second position.
Alien Implant: Newcomb's Smoking Lesion

it will mean that everyone in the future had their dial set to the second position.

No it won't. Nothing you wrote into the story indicates that you can change the box (in case of no forecaster). If you could, that would change everything (and it wouldn't be the smoking lesion anymore).

I don't think you understood. Consider Newcomb's problem by itself. Omega has already flown away. The million is either there, or it is not. The only sense that you can change whether the million is there is this: if you decide to take two boxes, you are basically deciding to have been a person who would take two boxes, and therefore deciding that Omega would have not put the million. If you decide to take one box, you are basically deciding to have been a person who would take one box, and therefore deciding that Omega would have put the million there. In my situation, it is the same: you can "determine" whether your dial is set to the first or second position by making a decision about whether to smoke. Now consider the Omega situation above, except that after Omega has left, Super-Omega steps in, who cannot be predicted by Omega. Super-Omega changes your decision to the opposite of what it was going to be. If this happens, you can two-box and still get the million, or one-box and get nothing, depending on what your original decision was. In my situation, it is the same: if someone can actually persuade someone to do the opposite of his dial setting, that persuader is basically like Super-Omega here. In other words, this would be exactly what you were talking about, the situation where convincing someone does not help. What I was saying was this: in the Alien Implant world, the currently existing people have their dials set to the first or second position in a certain proportion. Let's say that 90% of people have their dials set to the second position (so that most people don't die of cancer), and 10% have their dials set to the first position. I agree that the story says their dials never change place. But new people are constantly being born, and nothing in the story says that the proportion among the new people cannot be different. Assuming the non-existence of Super-Omegas, it is true that the proportion of people who choose to smoke will never be diffe
Alien Implant: Newcomb's Smoking Lesion

I know it was the intention, but it doesn't actually work the way you think.

The thing that causes the confusion is that you introduced an infallible decision maker into the brain that takes all autonomy away from the human (in case of there being no forecaster). This is basically a logical impossibility, which is why I just said "this is newcomb's problem". There has to be a forecaster. But okay, suppose not. I'll show you why this does make a difference.

In Newcomb's problem, you do in fact influence the contents of the opaque box. Your decision ... (read more)

I'm not sure what you mean by "autonomy" here. The scientists guess that the device is reading or writing, but a third possibility is that it is doing both, and is a kind of brain computer interface. In essence you might as well say it is part of the thing there: the human-black box combination has just as much autonomy as normal humans have. "Suppose the decision maker was infallible. Everyone who smokes dies. Sooner or later people would all stop smoking. And this is where the scenario doesn't work anymore. Because the number of people dying can't go down. So either it must be impossible to convince people – in that case, why try? – or the decision maker becomes fallible, in which case your whole argument breaks apart. You don't smoke and still die." In real life it does seem impossible to convince people; there are plenty of people who are stubborn two-boxers, and plenty of people stubbornly insisting on smoking in the smoking lesion, like yourself. So nothing in my experience rules out it being impossible to convince everyone because of the box. Nonetheless, if the box is writing people's choices, that does not mean it will permanently be impossible to persuade people. It will be impossible to persuade people who already have the opposite written; but if we succeed in the future in persuading everyone, it will mean that everyone in the future had their dial set to the second position. Nothing says that the proportion of people in the population with the dial set one way or another can't change; it may be being beamed in by aliens, and perhaps you are cooperating with them by trying to persuade people. "Think about this fact again: no forecaster means there is a fixed percentage of the population who has their box on state 1." So what. The proportion of the population who will in real life die of cancer is a fixed proportion; everyone who is going to die of cancer, is going to die of it, and everyone who isn't, isn't. That doesn't mean the proportion can't ch
Alien Implant: Newcomb's Smoking Lesion

I'm afraid you misunderstand the difference between the Smoking Lesion and Newcomb's problem. In the Smoking Lesion, if you are the kind of person who is affected by the thing which causes lung cancer and the desire to smoke, and you resist this desire, you still die of cancer. Your example is just Newcomb's problem with an infallible forecaster, where if you don't smoke you don't die of cancer. This is an inherent difference. They are not the same.

My example may or may not have a forecaster. The story doesn't say, and that's the point. Even if it turns out that the box is not forecasting anything, but simply making people do things, the winning move is the same. The Smoking Lesion is used as a counterexample to evidential decision theory. But understood in the way that you just did, it would not be a counterexample. You have the desire to smoke. So you know you have that desire, and you already know that you likely have the lesion. So if you resist the desire, it does not become less probable that you have the lesion. In order to be a counterexample, your estimate of the probability that you have the lesion has to change depending on whether you decide to smoke or not smoke. This is different from the situation that you just described.
AI Research Discussed by Mainstream Media

I'm pretty happy with this article... though one of my concerns is that the section on how exactly AI could wipe out humanity was a bit short. It wants to cure cancer, it kills all humans, okay, but a reader might just think "well this is easy, tell it not to harm humans." I'd have liked if the article had at least hinted at why the problem is more difficult.

Still, all in all, this could have been much worse.

Also a book excerpt. Books tend to be better.
Stupidity as a mental illness

I feel like I am repeating myself. Here is the chain of arguments

1) A normal person seeing this article and its upvote count will walk away having a very negative view of LessWrong (reasons in my original reply)

2) Making the valid points of this article is in no way dependent on the negative consequences of 1). You could do the same (in fact, a better job at the same) without offending anyone.

3) LessWrong can be a gateway for people to care about existential risk and AI safety.

4) AI safety is arguably the biggest problem in the world right now and extremel... (read more)

I feel it very much depends on your idea of a "normal person". Someone I consider a "normal person" would zone out after the first couple of paragraphs and go do something else. People who are sufficiently abnormal to finish that post (but still someone I'd consider "close to normal") would NOT walk away with a very negative view of LW. Clearly we have a different idea of what's normal or close-to-normal. Citation needed. Especially in 2017. I think you're mistaken about the direction of causality. Oh, boy. First let me point out that people who I would consider as close-to-normal, on hearing that chain of logic would make an rude gesture (physically or mentally, depending on how polite they are) and classify you as a crank they should probably keep away from. What did you call it? ah! "harming the reputation of LW". Second, do you really believe that the best way to attract people to LW is to be as... milquetoast as possible? Third, let's look at me. Here I am, snarking at everyone and generally unwilling to give out gold stars and express benevolence and empathy towards clueless newbies (and not only newbies). Doesn't it follow that I'm a great threat to the safety of humanity? Something must be done! Think of the children!
Stupidity as a mental illness

No, I fully acknowledge that the post tries to do those things, see the second half of my reply. I argue that it fails at doing so but is harmful for our reputation etc.

So if both you and me clearly understand the main point, and if the main point seems reasonably uncontroversial (everyone agrees that it's better to be smart than to be dumb, right?), then why do you describe this post as an epic fail? I'm sure that it makes some people's undergarments undergo interesting topological transformations, but that's hardly unusual or cause for such a.. forceful rejection.
Stupidity as a mental illness

It's about a set of mannerisms which many people on LW have that are really bad. I don't know what you mean by woke.

Well, there are these words and expression sprinkled throughout your comment: All of this seems to go deeper than "mannerisms". Your basic beef with the post seems to be that it is mean and insensitive and I think such an approach missed the post's main point. It seems that you think the main point is to stigmatize stupid people, label them sub-human, and, possibly, subject them to mandatory treatments with drugs and such. I think the main point is to stress that stupidity is not an unchanging natural condition ("sky is blue, water is wet, some people are stupid") but something that could be changed.
Me: We could be more successful at increasing general human intelligence if we looked at low intelligence as something that people didn't have to be ashamed of, and that could be remedied, much as how we now try to look at depression and other mental illness as illness--a condition which can often be treated and which people don't need to be ashamed of. You: YOU MONSTER! You want to call stupidity "mental illness", and mental illness is a bad and shameful thing!
Well said! a thought/idea can only go so far before they fall on deaf ears. Does not matter how "rational" a thought is...if you cannot convey it to just have an idea that is in your head.
I agree it isn't nice. I upvoted it anyway, because it is a very original idea that merits a discussion with not entirely predictable outcomes. This isn't just the most-upvoted submission in a while, it is also the most-discussed in an even longer while.
I am a bit confused by this comment. Is it, basically, a rant how LW is not woke enough?
I don't think that's all what the article is about. There's also the fact that our society only allows people to take drugs to fix illnesses. If you redefine what happens to be an illness you redefine what can be treated with drugs. You redefine what drugs get developed by Big Pharma. You redefine what our insurance system pays for. There's a reason about why we care about whether the FDA sees aging as a disease. It might be that the present administration completely deregulated the FDA so that we can treat things that aren't illnesses with drugs, but that's not where we are at the moment.
Open thread, Feb. 06 - Feb. 12, 2017

I agree that it's clear that you should one box – I'm more talking about justifying why one-boxing is in fact correct when it can't logically influence whether there is money in the box. Initially I found this to be unnerving initially, but maybe I was the only one.

The correct solution is not to one-box. It is to decide based on the flip of a coin. Take that, Omega. Seriously, the problem is over-constrained to the point of being meaningless, not representing reality at all. Part of the problem that leads to intuition breakdown is that the setup deals with omniscient knowledge and infinite computation, which surprise surprise has weird results. "Infinities in math problems leads to paradox: News at 11." The setup of the problem assumes that Omega has full knowledge about your decision making process and that you have no reciprocal insight into its own, other than assumption that its simulation of you is correct. Well, of course the correct answer then is to one-box, if you insist on deterministic processes, because by definition two-boxing results in empty boxes. This only feels weird because it seems acausal. But the solution is equivalent as Dagon said to eliminating free will -- without the intuitive assumption of free will the outcome is predictable and boring. Imagine the "you" in the setup was replaced with a very boring robotic arm with no intelligence that followed a very strict program to pick up either one or both of the boxes, but was explicitly programmed to do one or the other. Omega walks up, checks the source code to see whether it is configured to one-box or two-box, and fills the boxes accordingly. The weirdness comes when we replace the robot with "you", except a version of you that is artificially constrained to be deterministic and for which the physically implausible assumption is made that Omega can accurately simulate you. It's a problem of bad definitions, the sort of thing the Human's Guide to Words warns us against. Taboo the "you" in the setup of the problem and you find something more resembling the robotic arm than an actual person, for the purposes of the problem setup. However if you change the setup to be two Omegas of finite capability -- "you" have full access to the decision-making facili
That's pretty much the heart of the issue, isn't it? Clearly, by the omniscience of Omega's prediction, your choice is extremely correlated with what's in the box. So whether your choice determines the box contents, the box contents determines your choice, or some other thing determines both your choice and the box contents, there is a "logical influence" between your choice and the money in the box. The assumption that your choice is independent of the state of the universe is flawed.
Open thread, Feb. 06 - Feb. 12, 2017

Reposting this from last week's open thread because it seemed to get buried

Is Newcomb's Paradox solved? I don't mean from a decision standpoint, but the logical knot of "it is clearly, obviously better two one-box, and it is clearly, logically proven better to two-box". I think I have a satisfying solution, but it might be old news.

Thanks for starting the discussion, but please ALSO post your solution. Pretty much everything on the topic is old news, so no harm even if yours is already known to some.
I don't see an issue besides the decision standpoint that matters (and that could be solved). Depending on where you see the issue it's likely dependent on the assumptions you make about the problem.

It's solved for anyone who doesn't believe in magical "free will". If it's possible for Omega to correctly predict your action, then it's only sane to one-box. Only decision systems that deny this ability to predict will two-box.

Causal Decision Theory, because it assumes single-direction-causality (a later event can't cause an earlier one), can be said to deny this prediction. But even that's easily solved by assuming an earlier common cause (the state of the universe that causes Omega's prediction also causes your choice), as long as you don't demand actual free will.

What we have is a list of proposed decision theories (Evidential Decision Theory, Causal Decision Theory, Timeless Decision Theory, Updateless Decision Theory), each of which acts the same on standard decisions, but which deal with Newcomb-like problems differently. Some of these decision theories satisfy nice general properties which we would want a decision theory to satisfy. There's argument about which decision theory is correct, but also about what the various decision theories actually do in various situations. For example CDT is normally thought of as being the two-boxing theory that people intuitively use, but some people argue that it should take into account the possibility that it is in Omega's simulation and hence it even people following CDT should actually one-box. So the discussion is more nuanced than "What is the correct thing to do in Newcomb's problem?", it's more "By what general criteria should we judge a decision theory?". Of course any particular insight you have about Newcomb's problem might generalise to this way of looking at things.
Are we running out of new music/movies/art from a metaphysical perspective?

But what makes you think that more complex story types allow many more possibilities?

Isn't that an inherent property of complexity? A larger set of elements -> a larger powerset of elements -> more possibilities. In fact the size of the powerset grows at 2^x. I think a second game of thrones would be less groundbreaking, but doesn't have to be worse... and the same goes for the 1000th GoT.

There seems to be a slowdown in more arty / complex stories this decade (than compared to the 90's for example).

With film and television creation being more de

... (read more)
Yes I see what you're saying. I think a larger set of elements does not mean that all combination of those elements "works" as a movie story. It seems better to view possibilities as limited and sparse distinct points. A movie like Star Wars requires the correct combination of thousands of factors, and if you only had the right balance of half of the factors then there wouldn't necessarily be another workable story there. I guess the point I'm trying to make is that there seems to be a certain number of distinct point possibilities of movie stories that isn't directly a factor of the set of elements involved. I agree with you though, I'm not sure there is an iron-clad proof of this idea. I think it being proved right will depend on reaching a point where many people start to view our greatest works as being behind us, and wonder why that is the case.
Are we running out of new music/movies/art from a metaphysical perspective?

Well, there is a provably finite possibility space for stories. You only have so many ways to arrange letters in a script. The question is whether it's meaningful.

To use some completely made-up numbers, I think the current possibility space for movies produced by the bottom 80% of people with the current style may be 90% covered. The space for the top 2%, on the other hand, is probably covered for less than 0.1% (and I resisted putting in more zeros there).

To get more concrete, I'll name some pieces (which I avoided doing in my initial post). Take Game of ... (read more)

I always thought it's because of blood and boobs. No?
Interesting points, yea you're getting at the heart of what I'm trying to figure out. I think you're right, that it's easy to see how the story possibilities that use the simplest story types (Hero's Journey, etc) have possibly been ~90% completed. But what makes you think that more complex story types allow many more possibilities? Along the lines of your point, Game of Thrones is a fantasy epic with a much darker tone that breaks storytelling conventions, but wouldn't any fantasy epic series with similar attributes in the future seem less groundbreaking than Game of Thrones? I agree that you could apply similar attributes to a Sci Fi epic series, or another type of series, but it seems like that type of story would begin to get old in the near future as well. On the television front, there are so many shows being created that it's hard to see how they can keep being groundbreaking. With arty / more complex films like Being John Malkovich or Adaptation or Eternal Sunshine (I'm a fan of those Kaufman movies), does complexity lead to more possibilities of these types of movies or less? There seems to be a slowdown in more arty / complex stories this decade (than compared to the 90's for example). With film and television creation being more democratized than ever, I don't see a reason why the creation of these type of films would slow down apart from the remaining stories requiring more complexity and skill to write than ever. I think we agree on the necessity of higher skill in writing currently. But, it seems to me that a slowdown in the category of non-traditional or unique stories would mean that we are running out of those story possibilities as well.
Are we running out of new music/movies/art from a metaphysical perspective?

I'd say no to both. I don't think any genre has come meaningfully close to completion, though I don't know classic of jazz very well.

Let's talk film. If I take a random movie that I didn't like, I find it very similar to others. If, however, I take one that I really like, I find that frustratingly few movies exist that are even similar.

I consider the possibility space to be a function of creativity/intelligence/competence (let's call it skill) of writing, and one that grows faster-than-linearly. The space of medium-skill writing may be nearing completion (... (read more)

The reason that I bring up classical and jazz, is that there has been a clear slowdown in meaningful additions to the genres over the past few decades. So, if music genres reach a limit of possibilities, then it seems likely to apply to other areas of art as well. Yes, I agree that there are more intelligent (or less simple) stories that haven't been written yet. I'm not sure if you are saying that you agree that there is a limit of possible stories, or that you think there is no limit? If there is a limit, what do you think would be the signs that we are reaching it? I would agree that it seems from intuition that there are a lot of available stories still left to be written, but what would explain the slowdown in original properties being created or finding an audience currently (than in previous decades)?
A question about the rules

While I would agree that those kinds of accusations are used unfairly at times, I don't think it's unreasonable to assign Yudkowsky's statements a higher a priori chance of being true.

Open thread, Jan. 30 - Feb. 05, 2017

Do people feel like the Newcomb paradox (one-boxing yields the better result, it is clearly preferable; two-boxing only means taking an additional 1000$ through a decision that can't possibly have an effect on the 1 million, it is clearly preferable) been resolved through Anna's post in the Sequences (or others)? I strongly feel that I have a solution with no contradictions, but don't want to post it if it's obvious.

First impressions...

In being ironically guilty of not addressing your actual argument here, I'll point out that flaws of LW, valid or otherwise, aren't flaws of rationality. Rationality just means avoiding biases/fallacies. Failure can only be in the community.

I was being facetious, of course I still believe in rationality. But you know, I was reading Slate Star Codex, which basically represents the rationalist community as an amazing group of people committed to truth and honesty and the scientific approach - and though I appreciate how open these discussions are, I am a bit disappointed at how pedantic some of the comments are.
Yeah, this is particularly important. LW right now is a bit of a disgrace to rationality if I'm honest.
Willpower duality

Yeah, this is pretty much my conclusion, too. If I had read this article a couple of years ago, it'd have helped me a lot.

I'd add that you should still overrule system 1 in some really important and rare cases, it's just not practical for recurring things.

[LINK] EA Has A Lying Problem

I'm pretty split on this. I found the quotes from Ben Todd and Robert Wiblin to be quite harmless, but the quotes from Jacy Reese to be quite bad. I don't think it's possible to judge the scope of the problem discussed here based on the post alone. In either case, I think the effort to hold EA to high standards is good.

Disjunctive AI scenarios: Individual or collective takeoff?

I don't find this convincing.

“Human intelligence” is often compared to “chimpanzee intelligence” in a manner that presents the former as being so much more awesome than, and different from, the latter. Yet this is not the case. If we look at individuals in isolation, a human is hardly that much more capable than a chimpanzee.

I think the same argument has been made by Hanson, and it doesn't seem to be true. Humans seem significantly superior based on the fact that they are capable of learning language. There is afaik no recorded instance of a chimpanzee... (read more)

It occurred to me too that a strong human advantage is the verbal world.
The "turning knobs on an oil rig" analogy is particularly unconvincing. Even a smart human can read the engineering schematics and infer what the knobs do without needing to be shown.
Open thread, Jan. 09 - Jan. 15, 2017

Question: Regardless of the degree to which this is true, if everyone collectively assumed that Valence Utilitarianism (every conscious experience has value (positive or negative, depending on pleasantness/unpleasantness), each action's utility is the sum of all value it causes / changes / prevents) was universally true, how much would that change about Friendly AI research?

It would surely create new failure modes for Friendliness: e.g. kill all humans to spawn a trillion barely sentients copy of an AI. But on the other hand I would say: not much. You'll still have the hard problems of instilling a very fragile ontology (what is a conscious being, from the bottom up?), analyzing the consequences of the actions performed by the AI, judging carefully which values it is fulfilling or breaking and how much, etc.
Epilogue: Atonement (8/8)

Well, I don't think this is even complicated. The super happies are right... it is normal for them to forcefully reform us, and it is moral for us to erase the babyeater species.

Suffice to say I preferred the normal ending.

0Deku-shrub6y []
Apparently it's supposed to link here [] .
Expected Error, or how wrong you expect to be

One thing to keep in mind is that, just because something already exists somewhere on earth, doesn't make it useless on LW. The thing that – in theory – makes this site valuable in my experience, is that you have a guarantee of content being high quality if it is being received well. Sure I could study for years and read all content of the sequences from various fields, but I can't read them all in one place without anything wasting my time in between.

So I don't think "this has already been figured out in book XX" implies that it isn't worth reading. Because I won't go out to read book XX, but I might read this post.

Superintelligence: The Idea That Eats Smart People

Frankly I think that most people have no business having confident beliefs about any controversial topics. It's a bit weird to argue what an average IQ person "should" believe, because, applying a metric like "what is the average IQ of people holding this belief" is not something they're likely to do. But it would probably yield better results than whatever algorithm they're using.

Your first sentence isn't really a sentence so I'm not sure what you were trying so say. I'm also not sure if you're talking about the same thing I was talkin... (read more)

I didn't think you were talking about means of two different populations.... I was mainly making the point that having a population of smarter people than just an individual believing in an idea, wasn't great evidence for that idea for that individual. I didn't get that impression. But if you expand on what you mean by stuff we can try and get evidence for it one way or another.
Load More