All of Mestroyer's Comments + Replies

A couple questions- what portion of the workshop attendees self-selected from among people who were already interesting in rationality, compared to the portion that randomly stumbled upon it for some reason?

Don't know, sorry.

Hi. Checking back on this account on a whim after a long time of not using it. You're right. 2012!Mestroyer was a noob and I am still cleaning up his bad software.

I would need a bunch of guarantees about the actual mechanics of how the AI was forced to answer before I stopped seeing vague classes of ways this could go wrong. And even then, I'd assume there were some I'd missed, and if the AI has a way to show me anything other than "yes" or "no", or I can't prevent myself from thinking about long sequences of bits instead of just single bits separately, I'd be afraid it could manipulate me.

An example of a vague class of ways this could go wrong is if the AI figures out what my CEV would want usin... (read more)

It doesn't have to know what my CEV would be to know what I would want in those bits, which is a compressed seed of an FAI targetted (indirectly) at my CEV.

But there are problems like, "How much effort is it required to put into it?" (clearly I don't want it to spend far more compute power than it has trying to come up with the perfect combination of bits which will make my FAI unfold a little bit faster, but I also don't want it to spend no time optimizing. How do I get it to pick somewhere in between without it already wanting to pick the optimal amount of optimization for me?) "What decision theory is my CEV using to decide those bits? (Hopefully not something exploitable, but how do I specify that?)"

Ok, so your request would really be along the lines of "please output a seed AI that would implement indirect normativity", or something aong those lines?

Given that I'm turning the stream of bits, 10KiB long I'm about to extract from you into an executable file, through this exact process, which I will run on this particular computer (describe specifics of computer, which is not the computer the AI is currently running on) to create your replacement, would my CEV prefer that this next bit be a 1 or a 0? By CEV, would I rather that the bit after that be a 1 or a 0, given that I have permanently fixed the preceding bit as what I made it? By CEV, would I rather that the bit after that be a 1 or a 0, given that I have permanently fixed the preceding bit as what I made it? ...

(Note: I would not actually try this.)

Why not?
The AI is not omnipotent. How does it know what your coherent extrapolated volition would be?
Nay, all credit (okay, not actually all credit) goes to the blue-minimizing robot.

This reminds me of this SMBC. There are fields (modern physics comes to mind too) that no one outside of them can understand what they are doing anymore, yet that appear to have remained sane. There are more safeguards against postmodernists' failure mode than this one. In fact, I think there is a lot more wrong with postmodernism than that they don't have to justify themselves to outsiders. Math and physics have mechanisms determining what ideas within them get accepted that imbue them with their sanity. In math, there are proofs. In physics, there are ex... (read more)

CFAR seems to be trying to be using (some of) our common beliefs to produce something useful to outsiders. And they get good ratings from workshop attendees.

A couple questions- what portion of the workshop attendees self-selected from among people who were already interesting in rationality, compared to the portion that randomly stumbled upon it for some reason? And even if it were from outsiders... I suppose that guards against the specific post-modernist failure mode. I think the checking by having to explain to outsiders isn't the most important thing that checks engineering, though: the most important one is having to engineer things that actually work. So rationality producing people who are better at accomplishing their goals would be the ideal measure.
True. CFAR is anything but insular. Their (excellent) workshops are based on outside research and they do very well to reach out to outsiders. They have Slovic and Stanovich as advisors, Kahneman has visited them, etc.

The last point is particularly important, since on one hand, with the current quasi-Ponzi mechanism of funding, the position of preserved patients is secured by the arrival of new members.

Downvoted because if I remember correctly, this is wrong; the cost of preservation of a particular person includes a lump of money big enough for the interest to pay for their maintenance. If I remember incorrectly and someone points it out, I will rescind my downvote.

I use text files. (.txt, because I hate waiting for a rich text editor to open, and I hate autocomplete for normal writing) It's the only way to be able to keep track of them. I sometimes write paper notes when I don't have a computer nearby, but I usually don't keep those notes. Sometimes if I think of something I absolutely have to remember as I'm dozing off to sleep, I'll enter it in my cell phone because I use that as an alarm clock and it's always close to my bed. But my cell phone's keyboard makes writing notes really slow, so I avoid it under normal... (read more)

In response to your first paragraph,

Human morality is indeed the complex unfolding of a simple idea in a certain environment. It's not the one you're thinking of though. And if we're talking about hypotheses for the fundamental nature of reality, rather than a sliver of it (because a sliver of something can be more complicated than the whole) you have to include the complexity of everything that contributes to how your simple thing will play out.

Note also that we can't explain reality with a god with a utility function of "maximize the number of copie... (read more)

The principle of sufficient reason rides again....or, if not, God can create a small numbers of entities thatbThe really wants to, and roll a die for the window dressing.

I agree. "AGI Safety"/"Safe AGI" seems like the best option. if people say, "Let me save you some time and tell you right now that's impossible" half of the work is done. The other half is just convincing them that we have to do it anyway because otherwise everyone is doomed. (This is of course, as long as they are using "impossible" in a loose sense. If they aren't, the problem can probably be fixed by saying "our definition of safety is a little bit more loose than the one you're probably thinking of, but not so much more loose that it becomes easy").

Time spent doing any kind of work with a high skill cap.

Edit: Well, okay not any kind of work meeting that criterion, to preempt the obvious LessWrongian response. Any kind you can get paid for is closer to true.

One of my old CS teachers defended treating the environment as adversarial and knowing your source code, because of hackers. See median of 3 killers. (I'd link something, but besides a paper, I can't find a nice link explaining what they are in a small amount of googling).

I don't see why Yudkowsky makes superintelligence a requirement for this.

Also, it doesn't even have to be source code they have access to (which they could if it was open-source software anyway). There are such things as disassemblers and decompilers.

[Edit: removed implication that Yudkowsky thought source code was necessary]

Because often when we talk about 'worst-case' inputs, it would require something of this order to deliberately give you the worst-case, in theoretical CS, at least. I don't think Eliezer would object at all to this kind of reasoning where there actually was a plausible possibility of an adversary involved. In fact, one focus of things like cryptography (or systems security?) (where this is assumed) is to structure things so the adversary has to solve as hard a problem as you can make it. Assuming worst-case input is like assuming that the hacker has to do no work to solve any of these problems, and automatically knows the inputs that will screw with your solution most.

A lot of stuff on LessWrong is relevant to picking which charity to donate to. Doing that correctly is of overwhelming importance. Far more important than working a little bit more every week.

This is the kind of thing that when I take the outside view about my response, it looks bad. There is a scholarly paper refuting one of my strongly-held beliefs, a belief I arrived at due to armchair reasoning. And without reading it, or even trying to understand their argument indirectly, I'm going to brush it off as wrong. Merely based on the kind of bad argument (Bad philosophy doing all the work, wrapped in a little bit of correct math to prove some minor point once you've made the bad assumptions) I expect it to be, because this is what I think it wou... (read more)

To be fair to yourself, would you reject it if it were a proof of something you agreed with?

If they had gone out and 'proven' mathematically that sentient robots ARE possible, I'd be equally skeptical - not of the conclusion, but of the validity of the proof, because the core of the question is not mathematical in nature.

Actually, if you do this with something besides a test, this sounds like a really good way to teach a third-grader probabilities.

we're human beings with the blood of a million savage years on our hands. But we can stop it. We can admit that we're killers, but we're not going to kill Today.

Captain James Tiberius Kirk dodging an appeal to nature and the "what the hell" effect, to optimize for consequences instead of virtue.

That clip is a brilliant example of Shatner's much-mocked characteristic acting-speak.

My impression is that they don't, because I haven't seen people who do this as low status. But they've all been people who are clearly high status anyway, due to their professional positions.

This is a bad template for reasoning about status in general, because of countersignaling.

Omniscience and omnipotence are nice and simple, but "morally perfect" is a word that hides a lot of complexity. Complexity comparable to that of a human mind.

I would allow ideal rational agents, as long as their utility functions were simple (Edit: by "allow" I mean they don't get the very strong prohibition that a human::morally_perfect agent does) , and their relationship to the world was simple (omniscience and omnipotence are a simple relationship to the world). Our world does not appear to be optimized according to a utility func... (read more)

I think this is eminently arguable. Highly complex structures and heuristics can be generated by simpler principles, especially in complex environments. Humans don't currently know whether human decision processes (including processes describable as 'moral') are reflections of or are generated by elegant decision theories, or whether they "should" be. To my intuitions, morality and agency might be fundamentally simple, with 'moral' decision processes learned and executed according to a hypothetically simple mathematical model, and we can learn the structure and internal workings of such a model via the kind of research program outlined here. Of course, this may be a dead end, but I don't see how one could be so confident in its failure as to judge "moral perfection" to be of great complexity with high confidence. By hypothesis, "God" means actus purus, moral perfection; there is no reason to double count. The rules of moral perfection are found implicit in the definition of the ideal agent, the rules don't look like a laundry list of situation-specific decision algorithms. Of course humans need to cache lists of context-dependent rules, and so we get deontology and rule consequentialism; furthermore, it seems quite plausible that for various reasons we will never find a truly universal agent definition, and so will never have anything but a finite fragment of an understanding of an infinite agent. But it may be that there is enough reflection of such an agent in what we can find that "God" becomes a useful concept against which to compare our approximations.

A first approximation to what I want to draw a distinction between is parts of a hypothesis that are correlated with the rest of the parts, and parts that aren't, so that and adding them decreases the probability of the hypothesis more. In the extreme case, if a part of a hypothesis is logically deduced from the other parts, then it's perfectly correlated and doesn't decrease the probability at all.

When we look at a hypothesis, (to simplify, assume that all the parts can be put into groups such that everything within a group has probability 1 conditioned o... (read more)

Abrahamic Gods are suppposed to be eternal, have minds and not be made of atoms, or other moving parts. That may be hard thing to sell, but complex gods are a mixture of natural and supernatural assumptions.
So you mean like a mind that's omniscient, omnipotent, and morally perfect? Why? AIXI is very easy to specify. The ideal decision theory is very easy to specify, hard to describe or say anything concrete about, but very easy to specify. If you're willing to allow electro-magnatism which is based on the mathematical theory of partial differential equations, I don't see why you won't allow ideal agents based on decision/game theory. Heck, economists tend to model people as ideal rational agents because ideal rational agents are simpler than actual humans.

Perhaps I'm misusing the phrase "ontologically basic," I admit my sole source for what it means is Eliezer Yudkowsky's summary of Richard Carrier's definition of the supernatural, "ontologically basic mental things, mental entities that cannot be reduced to nonmental entities." Minds are complicated, and I think Occam's razor should be applied to the fundamental nature of reality directly. If a mind is part of the fundamental nature of reality, then it can't be a result of simpler things like human minds appear to be, and there is no lessening the complexity penalty.

I don't think "ontologically basic" is a coherent concept. The last time I asked someone to describe the concept he ultimately gave up. So could you describe it better than EGI?

It seemed pretty obvious to me that MIRI thinks defenses cannot be made, whether or not such a list exists, and wants easier ways to convince people that defenses cannot be made. Thus the part that said: "We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years. So limited involvement of advanced nanotechnology and quantum computers would be appreciated. "

This is quibbling over semantics, but I would count "don't let the AI get to the point of existing and having an Internet-connected computer" as a valid defense. Additional defenses after that are likely to be underwhelming, but defense-in-depth is certainly desirable.
Yes. I assume this is why she's collecting these ideas. Katja doesn't speak for all of MIRI when she says above what "MIRI is interested in". In general MIRI isn't in favor of soliciting storytelling about the singularity. It's a waste of time and gives people a false sense that they understand things better than they do by incorrectly focusing their attention on highly salient, but ultimately unlikely scenarios.

I think theism (not to be confused with deism, simulationism, or anything similar) is a position only a crazy person could defend because:

  1. God is an ontologically basic mental entity. Huge Occam penalty.

  2. The original texts the theisms these philosophers probably adhere to require extreme garage-dragoning to avoid making a demonstrably false claim. What's left after the garage-dragoning is either deism or an agent with an extremely complicated utility function, with no plausible explanation for why this utility function is as it is.

  3. I've already listened

... (read more)
Thanks for engaging with me. I'm afraid you'll have to say more than that though to convince me. Of course I know about Occam's Razor and how it can be applied to God. So do the theist philosophers. My uncertainty comes from general uncertainty about whether or not that is the right way to approach the question, especially given that Occam's Razor is currently (a) unjustified and (b) arbitrary. Also, I think that it is better to be too open-minded and considerate than to be the opposite. Such explanations are easy to come by. For example, on any politically tinged issue, we have a good explanation for why anyone might be wrong. So would you say we shouldn't take seriously expert opinions if they are on a politically sensitive topic? You would advise me against e.g. asking a bunch of libertarian grad students why they were libertarians? Your conclusion from this is that the philosophers are the problem, and not the questions they are attempting to answer? You think, not that these questions are difficult and intractable, but that philosophers are stupid or irrational? That seems to me to be pretty obviously wrong, though I'd love to be convinced otherwise. (And if the questions are difficult and intractable, then you shouldn't be as confident as you are!)

Theistic philosophers raised as atheists? Hmm, here is a question you could ask:

"Remember your past self, 3 years before you became a theist. And think, not of the reasons for being a theist you know now, but the one that originally convinced you. What was the reason, and if you could travel back in time and describe that reason, would that past self agree that that was a good reason to become a theist?"

Mestroyer keeps saying this is a personality flaw of mine

An imaginary anorexic says: "I don't eat 5 supersize McDonalds meals a day. My doctor keeps saying this is a personality flaw of mine."

I don't pay attention to theistic philosophers (at least not anymore, and I haven't for a while). There's seeking evidence and arguments that could change your mind, and then there's wasting your time on crazy people as some kind of ritual because that's the kind of thing you think rationalists are supposed to do.

Why do you think they are crazy? They are, after all, probably smarter and more articulate than you. You must think that their position is so indefensible that only a crazy person could defend it. But in philosophical matters there is usually a lot of inherent uncertainty due to confusion. I should like to see your explanation, not of why theism is false, but of why it is so obviously false that anyone who believes it after having seen the arguments must be crazy. If you don't pay attention to theistic philosophers, are there any theists to whom you pay attention? It seems to me that theistic philosophers are probably the cream of the theist crop. Note that I honestly think you might be right here. I am open to you convincing me on this matter. My own thoughts on theism are confused, which is why I give it a say even though I don't believe in it. (I'm confused because the alternative theories still have major problems, problems which theism avoids. In a comparison between flawed theories it is hard to be confident in anything.)

If a few decades is enough to make an FAI, we could build one and either have it deal with the aliens, or have it upload everyone, put them in static storage, and send a few Von Neumann probes faster than it would be economical for aliens to send them to catch us if they are interested in maximum spread, instead of maximum speed, to galaxies which will soon be outside the aliens' cosmological horizon.

It is unlikely that the FAI would be able to deal with the aliens. The aliens would have (or be) their own "FAIs" much older and therefore more powerful. Regarding probes to extremely far galaxies: theoretically might work, depending on economics of space colonization. We would survive at the cost of losing most of potential colonization space. Neat.

Can't answer any of the bolded questions, but...

When you did game programming how much did you enjoy it? For me, it became something that was both productive (relatively, because it taught me general programming skills) and fun (enough that I could do it all day for several days straight, driven by excitement rather than willpower). If you are like me and the difference in fun is big enough, it will probably outweigh the benefit of doing programming exercises designed to teach you specific things. Having a decent-sized codebase that I wrote myself to refac... (read more)

I often had to force myself to do/learn game programming, but it still could be very enjoyable and motivating. This is a near-perfect description of it. Around 5-7 months in I was skilled enough to start working on more difficult projects. I often became completely obsessed with what I was working on, trying desperately to maximize my time spent working on it. This ranged from being super exciting to painfully disruptive. It allowed me to get things done, but often made it hard to continue studying after I finished the project. As for including AI in games, that's something important that I have considered. I won't be giving up game development completely, partially for that reason. There are books like 'Programming Game AI by Example' that could greatly increase my programming skill and knowledge of AI-related topics. Also, having a portfolio of completed game projects, especially ones with AI in them, would be highly favored by colleges. I linked the recommended MIRI courses page in my discussion; you must have missed it (that's okay, it was in the most boring part.) I have already considered that several other topics, aside from programming, are essential to know about when dealing with the creation of AI. So I already plan to learn functional programming; I will begin dedicated study of it within a year, whenever it will fit into my short-term studying plans.

The downside to not reading what I write is that when you write your own long reply, it's an argument against a misunderstood version of my position.

I am done with you. Hasta nunca.

Oops. Fixed.

Two regular attendees. Two people who sometimes show up. One person who's new and whose attendance rate hasn't been well-established.

Roughly similar to what our meetup was like. We once had 8 people show up, but 4 was about average.

You, personally, probably don't care about all sentient beings. You probably care about other things. It takes a very rare, very special person to truly care about "all sentient beings," and I know of 0 that exist.

I care about other things, yes, but I do care quite a bit about all sentient beings as well (though not really on the level of "something to protect", I'll admit). And I have cared about them before I even heard of Eliezer Yudkowsky. In fact, when I first encountered EY's writing, I figured he did not care about all sentien... (read more)

I'm not going to read all of that. Responding to a series of increasingly long replies is a negative-sum game. I'll respond to a few choice parts though: Most of this thread: But why do you care about that? It's grossly improbable that out of the vast space of things-to-protect, you were dispotitioned to care about that thing before hearing of the concept. So you're probably just shopping for a cause in exactly the way EY advises against. To put it another way... for the vast majorty of humans, their real thing-to-protect is probably their children, or their lover, or their closest friend. The fact that this is overwhelmingly underrepresented on LW indicates something funny is going on.

The problem with "Act on what you feel in your heart" is that it's too generalizable. It proves too much, because of course someone else might feel something different and some of those things might be horrible.

It looks like there's all this undefined behavior, and demons coming out the nose from the outside because you aren't looking at the exact details of what's going on in with their feelings that are choosing the beliefs. Though a C compiler given an undefined construct may cause your program to crash, it will never literally cause demons... (read more)

If utility functions are those constructed by the VNM theorem, your utility function is your wants; it is not something you can have wants about. There is nothing in the machinery of the theorem that allows for a utility function to talk about itself, to have wants about wants. Utility functions and the lotteries that they evaluate belong to different worlds. Are there theorems about the existence and construction of self-inspecting utility functions?
I very much doubt that. At least with present technology you cannot self-modify to prefer dead babies over live ones; and there's presumably no technological advance that can make you want to.

Is Year 2000-era computing power your true estimate for a level of computing power that is significantly safer than what comes after?

This quote seems like it's lumping every process for arriving at beliefs besides reason into one. "If you don't follow the process I understand and is guaranteed not to produce beliefs like that, then I can't guarantee you won't produce beliefs like that!" But there are many such processes besides reason, that could be going on in their "hearts" to produce their beliefs. Because they are all opaque and non-negotiable and not this particular one you trust not to make people murder Sharon Tate, does not mean that they all have the same pr... (read more)

I don't think it's lumping everything together. It's criticizing the rule "Act on what you feel in your heart." That applies to a lot of people's beliefs, but it certainly isn't the epistemology of everyone who doesn't agree with Penn Jillette. The problem with "Act on what you feel in your heart" is that it's too generalizable. It proves too much, because of course someone else might feel something different and some of those things might be horrible. But if my epistemology is an appeal to an external source (which I guess in this context would be a religious book but I'm going to use "believe whatever Rameses II believed" because I think that's funnier), then that doesn't necessarily have the same problem. You can criticize my choice of Rameses II, and you probably should. But now my epistemology is based on an external source and not just my feelings. Unless you reduce me to saying I trust Rameses because I Just Feel that he's trustworthy, this epistemology does not have the same problem as the one criticized in the quote. All this to say, Jillette is not unfairly lumping things together and there exist types of morality/epistemology that can be wrong without having this argument apply.
That means you can actually make people less harmful if you tell them to listen to their hearts instead of listening to ancient texts. The person who's completely in their head and analyses the ancient text for absolute guidance of action is dangerous. A lot of religions also have tricks were the believer has to go through painful exercises. Just look at a Christian sect like Opus Dei with cilices. The kind of religious believer who wears a cilice loses touch with his heart. Getting someone who's in the habit of causing his own body pain with a cilice to harm other people is easier.
I'd have to disagree here; I think that "faith" is a useful reference class that pretty effectively cleaves reality at the joints, which does in fact lump together the epistemologies Penn Jilette is objecting to. The fact that some communities of people who have norms which promote taking beliefs on faith do not tend to engage in acts of violence, while some such communities do, does not mean that their epistemologies are particularly distinct. Their specific beliefs might be different, but one group will not have much basis to criticize the grounds of others' beliefs. The flaw he's arguing here is not "faith-based reasoning sometimes drives people to commit acts of violence," but "faith-based reasoning is unreliable enough that it can justify anything, in practice as well as principle, including acts of extreme violence."

To have a thing to protect is rare indeed. (Aside: If your thing-to-protect is the same as a notable celebrity, or as the person you learned the concept from, it is not your thing-to-protect.)

Really? What if the thing you protect is "all sentient beings," and that happens to be the same as the thing the person who introduced it to you or a celebrity protects? There're some pretty big common choices (Edited to remove inflationary language) or what a human would want to protect.

Beware value hipsterism.

Or, if by "thing to protect", you ... (read more)

You, personally, probably don't care about all sentient beings. You probably care about other things. It takes a very rare, very special person to truly care about "all sentient beings," and I know of 0 that exist. I find it very convenient that most of Less Wrong has the same "thing-to-protect" as EY/SigInst, for the following reasons: 1. Safe strong AI is something that can only be worked on by very few people, leaving most of LW free to do mostly what they were doing before they adopted that thing-to-protect. 2. Taking the same thing-to-protect as the person they learned the concept from prevents them from having to think critically about their own wants, needs, and desires as they relate to their actual life. (This is deceptively hard -- most people do not know what they want, and are very willing to substitute nice-sounding things for what they actually want.) Taken in concert with this quote from the original article: seems obvious to me that most people on LW are brutally abusing the concept of having a thing-to-protect, and thus have no real test for their rationality, making the entire community an exercise in doing ever-more-elaborate performance forms rather than a sparring ground.

Thou shalt never engage in solipsism or defeatism, nor wallow in ennui or existential angst, or in any other way declare that thy efforts are pointless and that exerting thyself is entirely without merit. For just as it is true that matters may never get to the point where they cannot possibly get any worse, so is it true that no situation is impossible to improve upon. Verily, the most blessed of silver linings is the fact that the inherent incertitude of one’s own beliefs also implies that there is never cause for complete hopelessness and despair.

Ab... (read more)

Okay, I acknowledge that "no situation is impossible to improve upon" is not strictly speaking true for literally every conceivable situation, but if ever there is a time where it's acceptable to leave out the ol' BOCTAOE for the sake of prose, I'd say a post including this many thees and thous would be it. I don't think I conflated map and territory though. The statement "There is never cause for complete hopelessness and despair" is a policy recommendation (read it as: "complete hopelessness and despair is never useful"), not a statement about the territory.

Overthinking issues that are really very simple

Counter-signallign as a smart-person mistake

Valuing intelligence above all other qualities

Rigidly adhering to rules -- compare the two endings of "Three Worlds Collide" and the decision by which they diverge.

Expecting other people to always be rational

Got nothing for the last two. I don't think the last one is a mistake that very many people at all make. (I think being right about things has surprising benefits well past the point that most people can see it having benefits).

Other smart person mistak... (read more)

Adding to this: Directly pointing out people's flaws I also don't see the problem with valuing being right highly. I can see the problem with letting people know that you are right too much.

Context: Aang ("A") is a classic Batman's Rule (never kill) hero, as a result of his upbringing in Air Nomad culture. It appears to him that he must kill someone in order to save the world. He is the only one who can do it, because he's currently the one and only avatar. Yangchen ("Y") is the last avatar to have also been an Air Nomad, and has probably faced similar dilemmas in the past. Aang can communicate with her spirit, but she's dead and can't do things directly anymore.

The story would have been better if Aang had listened to her advice, in my opinion.

And anyhow, why didn't they forcibly sedate every human until after the change? Then if they decided it wasn't worthwhile they could choose to die then.

It wouldn't be their own value system making the decision. It would be the modified version after the change.

Unrelatedly, you like Eliezer Yudkowsky's writing, huh? You should read HPMOR.

Something that's helped me floss consistently: (a) getting a plastic holder thing, not the little kind where it's still extremely difficult to reach your back teeth, but a reusable one with a long handle that you wrap floss onto, and (b) keeping it next to my computer, within arms reach.

If you are told a billion dollars hasn't been taxed from people in a city, how many people getting to keep a thousand dollars (say) do you imagine? Probably not a million of them. How many hours not worked, or small things that they buy do you imagine? Probably not any.

But now that I think about it, I'd rather have an extra thousand dollars than be able to drink at a particular drinking fountain.

But I don't think fairness the morality center is necessarily fairness over differing amounts of harm. It could be differing over social status. You could have an inflated sense of fairness, so that you cared much more than the underlying difference in what people get.

You're familiar with the idea of anthropomorphization, right? Well, by analogy to that, I would call what you did here "rationalistomorphization," a word I wish was added to LessWrong jargon.

This reaction needs only scope insensitivity to explain, you don't need to invoke purity. Though I actually agree with you that liberals have a disgust moral center.

How so?

What is the best textbook on datamining? I solemnly swear that upon learning, I intend to use my powers for good.

This sounds like bad instrumental rationality. If your current option is "don't publish it in paperback at all", and you are presented with an option you would be willing to take, publishing at a certain quality, if that quality was the best quality, then the fact that there may be better options you haven't explored should never return your "best choice to make" to "don't publish it in paperback at all." Your only viable candidates should be: "Publish using a suboptimal option" and "Do verified research about what is the best option and then do that."

As they say, "The perfect is the enemy of the good."

Sure, but I'm not even sure at this stage that publishing a paperback version with CreateSpace is a better use of 2 hours of Alex's time than the other stuff he's doing. Are there hidden gotchas which make publishing worse than not-publishing even if it was totally free? (I've encountered many examples of this while running MIRI.) Will it actually take 5 hours of time rather than 2? I don't know the answers to these questions, and this isn't a priority. Deciding whether to publish a paperback copy of Smarter Than Us is, like, the 20th most important decision I'll make this week. I'm not even sure that explaining all the different considerations I'm weighing for such a minor decision is worth the time I've spent typing these sentences. Anyway, I don't mean to be rude and I understand why you and Alicorn are engaging me about this, it's just that the decision is more complicated and less important (relative to all the invisible-to-LWers things we're doing) than you might realize, and I don't have time to explain it all. Again: if somebody can save us time on the initial research to figure out what's a good idea, it might become competitive with the other things Alex is doing with his MIRI time.

Downvoted for the fake utility function.

"I wont let the world be destroyed because then rationality can't influence the future" is an attempt to avoid weighing your love of rationality against anything else.

Think about it. Is it really that rationality isn't in control any more that bugs you, not everyone dying, or the astronomical number of worthwhile lives that will never be lived?

If humanity dies to a paperclip maximizer, which goes on to spread copies of itself through the universe to oversee paperclip production, each of those copies being rational beyond what any human can achieve, is that okay with you?

4Ilverin the Stupid and Offensive10y
Thank you, I initially wrote my function with the idea of making it one (of many) "lower bound"(s) of how bad things could possibly get before debating dishonestly becomes necessary. Later, I mistakenly thought that "this works fine as a general theory, not just a lower bound". Thank you for helping me think more clearly.

Whether or not the lawful-goods of the world like Yvain are right, they are common. There are tons of people who want to side with good causes, but who are repulsed by the dark side even when used in favor of those causes. Maybe they aren't playing to win, but you don't play to win by saying you hate them for for following their lawful code.

For many people, the lawful code of "I'm siding with the truth" comes before the good code of "I'm going to press whatever issue." When these people see a movement playing dirty, advocating arguments... (read more)

Could you post a screenshot or archived version of your Facebook link?

Science is tailored to counteract human cognitive biases. Aliens might or might not have the same biases. AIs wouldn't need science.

For example, science says you make the hypothesis, then you run the test. You're supposed to make a prediction, not explain why something happened in retrospect. This is to prevent hindsight bias and rationalization from changing what we think is a consequence of our hypotheses. But the One True Way does not throw out evidence because humans are too weak to use it.

That isn't really clear to me. Science wasn't intelligently designed; it evolved. While it has different ideals and functions from other human institutions (such as religions and governments), it has a lot in common with them as a result of being a human institution. It has a many features that contribute to the well-being of its participants and the stability of their organizations, but that don't necessarily contribute much to its ostensible goal of finding truth. For instance, it has been commonly observed that wrong ideas in science only die when their adherents do. Senior scientists have influence proportional to their past success, not their current accuracy. This serves the interests of individual humans in the system very well, by providing a comfortable old age for successful scientists. But it certainly does not counteract human cognitive biases; it works with them! Yes, science has the effect of finding quite a lot of truth. And philosophers and historians of science can point to good reasons to expect science to be much better at this than other claimed methods such as mysticism or traditionalism. But science as an institution is tailored at least as much to self-sustenance through human biases, as to counteracting them.

But how do you avoid those problems? Also, why should contemplating tradeoffs between how much we can get values force us to pick one? I bet you can imagine tradeoffs between bald people being happy, and people with hair being happy, but that doesn't mean you should change your value from "happiness" to one of the two. Which way you choose in each situation depends on how many bald people there are, and how many non-bald people there are. Similarly, with the right linear combination, these are just tradeoffs, and there is no reason to stop caring about one term because you care about the other more. And you didn't answer my last question. Why would most people meta-reflectively endorse this method of reflection?

1, as you said, can be destroyed by the truth (if they're actually wrong), so it's part of a learning process. 2 isn't a problem once you isolate the principle by itself, outside of various emotional factors. 3 is a counterargument against any kind of decisionmaking, it means that we should be careful, not that we shouldn't engage in this sort of reflection. 4 is the most significant of these problems, but again it's just something to be careful about, same is in 3. As for 5, that's to be solved by realizing that there are no sacred values. It doesn't, you're right. At least, contemplating tradeoffs doesn't by itself guarantee that people would choose only one value, But it can force people to endorse conclusions that would seem absurd to them - preserving one apparent value at the expense of another. Once confronted, these tensions lead to the reduction to one value. As for why people would meta-reflectively endorse this method of reflection - simply, because it makes sense.
Load More