Open Thread: September 2009

In light of how important it is (especially in terms of all the decision theory posts!) to know about Judea Pearl's causality ideas, I thought I might share something that helped me get up to speed on it: this lecture on Reasoning with Cause and Effect.

For me, it has the right combination of detail and brevity. Other books and papers on Pearlean causality were either too pedantic or too vague about the details, but I learned a lot from the slides, which come with good notes. Anyone know what proceedings papers it refers to though?

2Eliezer Yudkowsky14y

Hm, on first browse this does seem like a good intro.

[-]Kaj_Sotala14y10

Does anyone feel the occasional temptation to become religious?

I do, sometimes, and push it away each time. I doubt that I could, really, fall for that temptation - even if I tried to, the illogicality of the whole thing would very likely prevent me from really believing in even a part of it very seriously. And as more and more time passes, religious structures begin to seem more and more ridiculous and contrived. Not that I'd have believed that them starting to feel more contrived would even have been possible.

And yet... occasionally I remember the time back in my teens, when I had some sort of a faith. I remember the feeling of ultimate safety it brought with it - the knowledge that no matter what happens, everything will turn out well in the end. It might be a good thing that I spend time worrying over existential risks, and spend time thinking about what I could do about them, but it sure doesn't exactly improve my mental health. The thought of returning to the mindset of a believer appeals on an emotional level, in the same way stressed adults might longingly remember the carefree days of childhood. But while you can't become a child again, becoming a believer is at least theoretically possible. And sometimes I do play around with the idea of what it'd be like, to adopt a belief again.

6Jack14y

Uh, here is a confession. Twice in the last 6-7 years, at moments of extreme psychological distress, I talked to the God of my Catholic youth. Once I went to an empty church after a series of coincidences (running into two people from my Catholic grade school separately) that I thought was a sign from God. Really, embarrassing, right? It was like my mind segmented and the rationalist was put aside and the devout Catholic school boy was put in charge. The last time this happened was about four years ago. I don't know if this could still happen today (my atheism is probably more entrenched now). There have been plenty of periods of distress where this didn't happen, so I don't know what triggered it in particular. I think my brain must have really needed a God figure at that moment and didn't know how to deal with the pain without one so it hacked itself and turned off the rationalist defenses. Or something, it seems so screwed up looking back on it. Interestingly, I told my theistic-non religious girlfriend about this who in turn told her Christian best friend. Talking to me on the phone for the first time, the friend something along the lines of "I know you say you're an atheist but B(my girlfriend) told me that you sometimes pray when you're upset so I know you're really a good person/God loves you." In other words, I'm least wrong when I'm thinking least clearly.

[-]Alicorn14y9

On the consolidation of dust specks and the preservation of utilitarian conclusions:

Suppose that you were going to live for at least 3^^^3 seconds. (If you claim that you cannot usefully imagine a lifespan of 3^^^3 seconds or greater, I must insist that you concede that you also cannot usefully imagine a group of 3^^^3 persons. After all, persons are a good deal more complicated than seconds, and you have experienced more seconds than people.)

Suppose that while you are contemplating how to spend your 3^^^3-plus seconds, you are presented with a binary ch... (read more)

3Unknowns14y

If this choice was actually presented to someone, my guess would be that he would first choose the speck, and then after an extremely long time (i.e. much, much longer than 50 years, giving him a sense of proportion) he would undergo a preference reversal and ask for the torture.

2Eliezer Yudkowsky14y

What if the TORTURE occurs during a random time in the next 3^^^3 seconds, not right at the beginning? Also, I think we definitely require a limit on sanity damage because otherwise the scenario is being tortured for 50 years and then spending the next 3^^^3 seconds being insane which Vastly outweighs the ordinary scenario of being tortured for 50 years.

0Alicorn14y

In the original scenario, where just some random person got tortured, no constraints were specified about eir sanity or lifespan post-torture.

1Eliezer Yudkowsky14y

I think I did specify that no one would die who would otherwise be immortal; eternal insanity or 3^^^3 years of insanity ought to be implicitly included, I'd think.

2Larks14y

This phrase makes the difference for me- the 3^^^3 other people in the original argument weren't mad- or at least, no more than would have been mad anyway. Additionally, in your scenario, we have to consider discount rates- it's certainly conceivable that someone might choose the dust specks over torture now, but be willing to forgo the dust specks in return for torture in 3^^^3 seconds time.

4Alicorn14y

Does it seem likely to you that out of 3^^^3 people chosen with no particular safeguards, not one of them will find a dust speck in the eye to be maddening? It could be the last straw in a string of misfortunes; it could set off some causal chain that will lead to other maddening events, etc.

1Larks14y

3^^^3 is such a huge number, some must find it maddening, but the proportion will be a lot lower than the odds that 50 years of torture breaks you mentally.

2Emily14y

That's interesting: I have much, much less hesitation in saying TORTURE to this one. With the original, I can grudgingly concede that I suppose I possibly ought to choose TORTURE, but I still can't ever quite convince myself to feel that it's a good answer. This one, I think I can.

7Wei Dai14y

Suppose you are given a button that you can press at any time during the 50 years of torture, that will stop the torture (and erase your memory of it if you wish), but you'll have to live with the dust speck from then on. I predict that you'll press the button after actually being tortured for a couple of hours, maybe days, but at most weeks. Even professional spies/soldiers/terrorists who have trained to resist torture end up betraying their cause, so I find it hard to believe that you can hold out for 50 years. But if you really prefer TORTURE now, that brings up an interesting question: whose preferences are more important, the current you, or the hypothetical future you? It could be argued that the future you is in a better position to decide, since she knows what it actually feels like to be tortured for a significant period of time, whereas you don't. But I don't consider that a knock-down argument, so what do you think? Suppose you can also commit to not pressing the button (say by disabling your arm/hand muscles for 50 years), would you do so? (This is related to a recent comment by Rolf Andreassen, which I think applies better to this scenario.)

0Emily14y

Yes, I think you're almost certainly right about the button, which thought does indeed put a dent in my lesser hesitation in choosing TORTURE. I think I would definitely not commit to not pressing the button if I were able to "try out" the SPECKS scenario for some short period of time first (say, a week). That way I could make a comparison. Absent that condition... I don't know. I can't imagine having the guts to commit to not pressing it. The fact is that while I certainly don't see a momentary dust speck as torture, I can easily imagine beginning to see it as torture after a day, never mind 3^^^3 seconds, which is rather longer than 50 years. But I can't be certain of that, nor of to what extent I would get used to it, nor of how it would compare with much worse kinds of torture. (But then again, there's a spanner in the works: knowing that you won't have any lasting physical damage... perhaps that would lend some kind of strength of mind to a torturee?)

[-]Furcas14y7

Sean Carroll and Carl Zimmer are leaving Bloggingheads, mostly because it's started playing nice with creationists. Click their names to read their full explanations.

1Peter_Twieg14y

Thanks for the link. I saw the original Behe podcast and was surprised when it suddenly disappeared with a brief note from an admin. I haven't been able to follow up on this matter much since it hasn't generated much buzz, but I'll look forward to reading these links - if people are boycotting Bloggingheads for giving platforms to people whom they don't like.... well, that's an interesting precedent to set which should be scrutinized carefully.

1Eliezer Yudkowsky14y

Hm. (Reads.) Well... if BHTV has me on, an "anti-accomodationist" as I've recently heard my diavlogs called (what a lovely term!), then I don't think it's unfair for them to give Behe a slot. It is extremely foolish that they paired him up with a linguist. (Reads.) Looks like Robert Wright agrees with that last part.

2Furcas14y

You're one of the last people I would have expected to be concerned with 'fairness' when one of the sides is blatant crackpottery. I suppose you wouldn't have a problem if BHTV invited an astrologer, as long as they paired him up with an astronomer?

0Peter_Twieg14y

What harm is done by bringing on an astrologer? At worst it fail to amuse. But it's obvious you're not talking about the diavlog's impact on you... you're concerned with the poor, unwashed masses who might actually be left to form their own opinions from the available information. Well, that's very nice of you, but I don't think it's unreasonable to believe that it's safe to expose people to views which might be labeled as "crackpottery" by some.

5Vladimir_Nesov14y

High-status serious people want to associate with high-status others. Allowing crackpots on the same venue dramatically decreases its attractiveness for quality participants. You can't really give an elaborate justification for why in this particular case it's OK, because signals are shallow.

5Furcas14y

The "unwashed masses", as you call them, are already getting plenty of exposition to crackpottery, much more than they get to real science, so that a few crackpots on BHTV are barely a drop in the ocean. That's not what concerns me, no. What concerns me is that BHTV has a reputation as a respectable website thanks to the participation of respectable academics and experts. It's reasonable to assume that such a respectable website wouldn't invite crackpots to promote their brand of crackpottery; in fact, that's an assumption I made myself until I read Sean's and Carl's posts. Inviting crackpots therefore gives the impression that these people should be taken seriously, even if we think they're wrong. In any event, your own motives are suspect, to say the least. Characterizing creationist nuts as "people whom [Sean and Carl] don't like", as if creationism was merely a distasteful political opinion, or something, makes you sound like a crackpot yourself, or worse, a postmodernist.

-1Peter_Twieg14y

So BHTV can't both enjoy the participation of respectable academics and also host the occasional crackpot? There exists no such universe where the two could possibly coincide? Is there some implicit assumption here that there's a fixed amount of BHTV episodes, each of which will feature either crackpots or respectable academics? Even if this were so, wouldn't the reasonable response be to skip over the crackpots rather than avoiding the entire medium? The only justifiable rationale I can see for skipping over BHTV because of this is if you just watched diavlogs at random and having crackpots degraded the signal:noise ratio of the site. But I doubt that you, I, Sean, Carl, or your average Bloggingheads viewer navigates the site in this manner. Even though I profoundly degree with Behe's epistemology (and theology), which should go without saying in these parts, I found the debate interesting (I think irreducible complexity is a neat topic), certainly moreso than I've enjoyed other diavlogs. Can anyone honestly say that Behe's presence is less valuable than any other podcast on the website? I doubt it, and thus it strikes me as disingenuous that the unique response his presence generates can be explained away purely through outrage at the notion that somewhere, someone's time may be wasted. And I didn't include the "people whom others don't like" line as a defense of creationism per se, but as a broader point about silencing views found in contempt. To rip off Will Wilkinson, I'd probably venture to assert that unrepentant Marxists are just as high on the crackpottery scale as Creationists, but I highly doubt we'd see people abandon the site in protest in BHTV hosted some of them. "Respectability" in this context is a tricky term to use, since "respectability" tends to be conferred by social fashions just as much as actual correspondence to whatever virtues we've deemed to be worthy of respect. On a more base level, I suspect that many participants in this community

2eirenicon14y

Should BHTV invite Perez Hilton to debate the fearsome Man Bat? Michael Behe is as credible an author as Pamela Anderson, although not quite as illuminating. I used to think that the worst kind of ignorance was when you knew you were wrong and refused to accept it. Now I think the worst kind is when you know you're capable of knowing when you're wrong but refuse to let yourself. Michael Behe wants to be ignorant of his own ignorance. Let him do so in the peace and quiet of his own sad little world.

1Peter_Twieg14y

You're shifting the goalposts some. I'm not defending the original decision to invite Behe. I'm questioning the notion that inviting Behe is such an egregious offense against BHTV's "respectability" that it should be boycotted. I wouldn't boycott BHTV if 90% of the diavlogs were replaced by midget porn, if it meant that I would get the occasional episode of Free Will. I think Behe's critics should just admit that what's really motivating the reaction is the notion that Creationists not only should not be given forums to speak, but those who do grant Creationists forums to speak should be actively identified and boycotted in a way which is reserved for an arguably arbitrarily-defined set of social undesirables. This isn't an indefensible position, but people have to admit to holding this belief (or some similar belief which is constructed in a more-charitable manner) before a meaningful debate can be enjoined. [Edit] Reading over the comments section of the CV posts, it looks like a lot of people are quick to point to Megan McArdle as the political crackpot equivalent of Behe. Should her presence be boycotted too as detrimental to the site? Where should the line be drawn? Where do you actually think the line would be drawn, if not along questionable ideological lines? Why have a line at all?

2eirenicon14y

What critic will not admit that? It's hardly a fringe opinion in the scientific community that Creationists should not be given forums to speak on the thoroughly unscientific topic of Creationism, and that those who do so and call it science are being absurdly and unnecessarily tolerant. Creationism has never been more or less than an attack on science. It's extremely toxic, and while I would never try to "silence" anyone, I don't think it deserves more publicity. I grew up being taught that dreck in a fundamentalist Christian school and I'm more familiar with Behe than I'd care to be. Frankly, he's an idiot, and his life purpose seems to be toward making more idiots. He doesn't need anyone's help. As for McArdle, I don't really care. Politics is not a hard science, and while she's something of a crackpot, she's not that way because somebody proved her map doesn't follow the territory. It's the difference between someone who thinks the earth is flat and someone who thinks it's run by the Illuminati. The former is just wrong, the latter is just crazy. I don't mind crazy, because crazy isn't nearly as dangerous as wrong.

-1Peter_Twieg14y

I haven't seen BhTV endorse Creationism as science in any official capacity.

3XFrequentist14y

We been known to shoot us some subjectivists 'round these here parts, y'hear? Sean Carrol concurs

-2timtyler14y

It seems like science snobbery. BHTV has loads of political commentary - and other non-science. It just isn't remotely like a peer-reviewed science journal.

1timtyler14y

PZ Myers says no to BHTV: http://scienceblogs.com/pharyngula/2009/09/phil_plait_ditches_blogginghea.php

-1timtyler14y

Phil Plait pulls the plug on BHTV: http://blogs.discovermagazine.com/badastronomy/2009/09/04/bloggingheads-capo-non-grata/

-1timtyler14y

Robert Wright discusses the whole issue on BHTV: http://bloggingheads.tv/diavlogs/22300

-1timtyler14y

He says 48 minutes in that the Templeton foundation "were no longer" and that they only supported the site for four months. However, their ads are still on the sponsored videos beneath: http://bloggingheads.tv/percontations ...so the Templeton foundation link apparently remains. Maybe there will be no more percontations, though.

-3timtyler14y

Wright gives much the same arguments about ID as I give - in the section on "Viable Intelligent Design Hypotheses" - on: http://originoflife.net/intelligent_design/

[-]Rain14y6

Since the topic of atheism, morality, and the like often come up here, I would like to point people to the free online book Secular Wholeness by David Cortesi. He approaches the topic of religion by trying to determine what benefits it can provide (community, challenges to improve oneself, easier ethical decisions, etc.), then tries to describe how to achieve these same benefits without resorting to religion. It's not very heavy on detail but seems very well sourced and with some good pointers on why people choose religions and what they get out of them.

ETA: This could have been a reply to the thread on Scientology, had I seen it before posting.

[-]spriteless14y5

It's been awhile since this came up. But hay, look, arguments about sex and race and stuff, as applicable to atheism as to rationality in general.

0JoeSchmoe14y

Boo to whoever voted this down in the first place. Great link.

0spriteless14y

It's obviously a counter-conspiracy against my conspiracy to get people to see rationality as something that exists outside of this site. And also to make you look at girls.

[-]anonym14y5

Scott Aaronson announced Worldview Manager, "a program that attempts to help users uncover hidden inconsistencies in their personal beliefs".

You can experiment with it here. The initial topics are Complexity Theory, Strong AI, Axiom of Choice, Quantum Computing, Libertarianism, Quantum Mechanics.

2Nubulous14y

Mostly agree is a higher degree of agreement than Agree ? To Somewhat agree that everyone should have the vote and Disagree that children should have the vote is inconsistent ? Obviously this is the work of the Skrull "Scott Aaronson", whose thinking is not so clear.

0Nubulous14y

Also, almost every question is so broken as to make answering it completely futile. So much so that it's hard to believe it was an accident.

2anonym14y

I find it hard to believe that you could really think the most likely explanation of the flaws you perceive are that Aaronson and the students that implemented this purposely introduced flaws and are trying to sabotage the work. So why do you utter such nonsense? And did it not occur to you that disagreeing that children should have the vote could be resolved by being neutral on everybody having the vote, which is what I did after realizing that there are plausible interpretations under which I would disagree and plausible interpretations under which I would agree.

2Nubulous14y

Whether you consider this as sabotage or not depends on what you think the goal of the site's authors was. It certainly wasn't to help find inconsistencies in people's thinking, given the obvious effort that went into constructing questions that had multiple conflicting interpretations. Quite.

2DS361814y

I just tried the one for AI and I think its not quite accurate. One of the biggest issues is that I think some of the terms need to be precisely defined and they are not. The other issue I found was that the analysis of my beliefs was not completely accurate because it did not take into account all the answers properly. Its an interesting idea but needs work.

-2[anonymous]14y

I didn't find the lack of precise definitions a problem.

1Larks14y

I got this conflict between my acceptance of the draft in the unlikely event it would be useful, and my belief that all acts I think the Government should be allowed to do are currently allowed. It doesn't seem to know of the existence of this supreme court ruling

1wedrifid14y

Interesting link. I played with it for a while. It kept misunderstanding the nuances of my responses, telling me I was wrong when I wasn't then refusing to listen to my replies. So I stopped playing with it. Two in one day. What are chances?

1cousin_it14y

Good idea, bad implementation. Right now it thinks I have this "tension", but I'm pretty sure it's not a tension.

3JulianMorrison14y

Versus: It's the [...] that hurts. "It is possible for one's mind to exist outside of one's material body." does not imply "the mind is physically independent of the material body". It's physically dependent and abstractly independent.

1anonym14y

I did have some difficulty resolving all tensions, but I was able to do so. I found that there were often alternate interpretations of a statement that would resolve a tension but were still plausible interpretations. For example, one that I remember was interpreting some of the questions about "physical body" more generally as "physical substrate". Sometimes the tension page didn't offer the question that needed reinterpretation, in which case I deferred the tension until I saw a tension that contained the statement to be reinterpreted. It definitely does need a lot of work, but I can imagine a tool like this having profound effects on people when all the bugs are worked out and it is applied to mind killers and beliefs/habits where cognitive biases figure prominently. One major thing that needs to be improved if they intend normal people to use it for normal issues like politics, abortion, etc., is to make the tension page much friendlier. Most LWers have probably studied logic, and can pretty easily interpret the tension explanation, but most people have no clue about logic and won't understand the implicit implications that aren't explained (like that contrapositive of "A -> B" is valid).

0[anonymous]14y

I wouldn't call it a bad implementation for the occasional wrong reported tension, especially if it's not completely clear for why it reports such a wrong tension. To me, the purpose of the service is not to provide 100% coherent and consistent questionnaire. The idea is that it points to conceptions that might contradict themselves. Whether or not they in fact do contradict should be up to a closer investigation. But merely pointing the user to these possible contradictions should prove to be useful, because it's so difficult find these inconsistencies by oneself. It seems clear to me that it will generate some false positives. It will also come up chains of logic that aren't obviously true or false (because it's impossible to create statements that are completely free of differing interpretations). Of course, the better the implementation in whole (both the logic system and the sets of statements) the less it will generate these false positives and other inconsistencies, but I do think that it's impossible to remove them all. Instead, the service should perhaps be considered more like a probing machine. To claim that it's a bad implementation sounds to me like it's not a useful implementation at all. Sure, it'll probably have a relatively many glitches and bugs, but the above comment doesn't give any particular evidence that the implementation as such doesn't work correctly. It seems almost equally likely that such possible inconsistencies are an inherent part of this kind of implementation. If the implementation would constantly point to tensions that are obviously not real tensions (or useful observations in general), then I'd be more inclined to call it a bad implementation. After all, such claim will discourage people from trying out the service and I don't see reason for such claim in the example cousin_it gave. The other common complaint seems to be the lack of precise definitions. Again, I see this more like a feature than a bug. When taking the questio

-6[anonymous]14y

[-]byrnema14y4

Is there an easy way to access the first comment of someone without looking at their comment page and uploading "next" zillions of times?

You're probably wondering why I would want to do that. I have been motivated occasionally to read someone's comments from beginning to end, and today I found myself wondering what my first comment was about.

My mind frequently returns to and develops the idea that sometime in the future, a friendly artificial intelligence is going to read Less Wrong. It uploads all the threads and is simultaneously able to (a) ... (read more)

0wedrifid14y

Earlier today I was assigning probabilities to whether or not someone on this site was someone I used to date. If they aren't then I really should introduce them...

[-]eirenicon14y4

Readers of Less Wrong may be interested in this New Scientist article by Noel Sharkey, titled Why AI is a dangerous dream, in which he attacks Kurzweil's and Moravec's "fairy tale" predictions and questions whether intelligence is computational ("[the mind] could be a physical system that cannot be recreated by a computer").

[edit] I thought this would go without saying, but I suspect the downvotes speak otherwise, so: I strongly disagree with the content of this article. I still consider it interesting because it is useful to be aware o... (read more)

5SilasBarta14y

I strongly disagree. First, on the grounds that LW readers have strong reason to believe this: to be false, and so treat it similarly to a proof that 2=1. But instead of just being a grouch this time, I decided to save you guys the effort and read it myself to see if there's anything worth reading. There isn't. It's just repetitions of skepticism you've already heard, based on Sharkey's rejection of "the assumption that intelligence is computational" (as opposed to what, and which is different and uncreatable why?), which "It might be, and equally it might not be". Other than that, it's a puff piece interview without much content.

1byrnema14y

(Phase 1) Agreed, I don't see why the mind isn't a type of "computer", and why living organisms aren't "machines". If there was something truly different and special about being organic, then we could just build an organic AI. I don't get the distinction being made. (Phase 2) Oh: sounds like dualism of some kind if it is impossible for a machine to have empathy, compassion or understanding. Meaning beings with these qualities are more than physical machines, somehow. (Phase 3) Reading through some of the comments to the article, it sounds like the objection isn't that intelligence is necessarily non-physical, but that "computation" doesn't encompass all possible physical activity. I guess the idea is that if reality is continuous, then there could be some kind of complexity gap between discrete computation and an organic process. Phases 1-3 is the sequential steps I've taken to try to understand this point of view. A view can't be rejected until its understood...I'm sure people here have considered the AI-is-impossible view before, but I hadn't. What is the physical materialist view on whether reality is discrete? (I would guess it's agnostic.) What is the AI view on whether computations must be discrete? (I would guess AI researchers wouldn't eschew a continuous computation as as a non-computational thing if it were possible?)

2SilasBarta14y

I agree it's important to apply the principle of charity, but people have to apply the principle of effort too. If Sharkey's point is about some crucial threshold that continuous systems possess, he should say so. The term "computational" is already taken, so he needs to find another term. And he can't be excused on the grounds that "it's a short interview", considering that he repeated the same point several times and seemed to find enough space to spell out what (he thinks) his view implies.

4HalFinney14y

"[the mind] could be a physical system that cannot be recreated by a computer" Let me quote an argument in favor of this, despite the apparently near universal consensus here that it is wrong. There is a school of thought that says, OK, let's suppose the mind is a computation, but it is an unsolved problem in philosophy how to determine whether a given physical system implements a given computation. In fact there is even an argument that a clock implements every computation, and it has yet to be conclusively refuted. If the connection between physical systems and computation is intrinsically uncertain, then we can never say with certainty that two physical systems implement the same computation. In particular, we can never know that a given computer program implements the same computation as a given brain. Therefore we cannot, in principle, recreate a mind on a computer; at least, not reliably. We can guess that it seems pretty close, but we can never know. If LessWrongers have solved the problem of determining what counts as instantiating a computation, I'd like to hear more.

-4SilasBarta14y

Sure thing. I solved the problem here and here in response to Paul Almond's essays on the issue. So did Gary Drescher, who said essentially the same thing in pages 51 through 59 of Good and Real. (I assume you have a copy of it; if not, don't privately message me and ask me how to pirate it. That's just wrong, dude. On so many levels.)

-1billswift14y

This was linked on Hacker News http://news.ycombinator.com/item?id=797871 I left this comment there: I thought this might be something like Eliezer's arguments against developing a GAI until it could be made provably Friendly AI, instead I just got an argument exactly like the ones in 1903 that said heavier than air flight by men was impossible - go back and read some of them, some of the arguments were almost identical. Some of the arguments are currently true, but some of them amount to "I can't do it, and no one else has done it, therefore there must be some fundamental reason it can't be done".

[-]cousin_it14y3

Just found this note in Shalizi's notebooks which casts an interesting shadow on the Solomonoff prior:

The technical results say that a classification rule is simple if it has a short description, measured in bits. (That is, we are in minimum description length land, or very close to it.) The shorter the description, the tighter the bound on the generalization error. I am happy to agree that this is a reasonable (if language-dependent) way of defining "simplicity" for classifier rules. However, so far as I can tell, this really isn't what makes

... (read more)

4Johnicholas14y

Here's my understanding of the dialog, which (as I read it) is not particularly critical of the Solomonoff prior, if that is what cousin_it meant by "casts a shadow". (Background knowledge) Shalizi understands "Occam's Razor" to be something like "In order to reach the truth, among the theories compatible with the evidence, chose the simplest". There is a claim that he wishes to refute. The claim is that a certain result is an explanation or proof of Occam's Razor. The result says that if one finds a simple classification rule which works well in-sample, then it is highly probable that it will continue to work well out-of-sample. This is a failure of relevance. Occam's Razor, as Shalizi understands it, is a way of obtaining TRUTH, but the proof only concludes something about GENERALIZATION PERFORMANCE. To illustrate the difference, he points to an example where, in order to increase generalization performance, one might decrease truth. Shalizi contrasts the algorithmic information theory proof with Kevin T. Kelly's Ockham Efficiency Theorem, which seems to Shalizi more productive. In particular, Kevin T. Kelly's formalization does talk about truth rather than generalization performance. Finally, Shalizi provides an alternative ending to the algorithmic information theory proof. If instead of choosing the simplest classification rule, one chose the simplest rule within a sparse random subset of rules (even a non-computable random subset), then you could still conclude a bound on generalization performance. By providing an alternative ending, he has constructed an alternative proof. Presumably this alternative proof does NOT seem like it is a formalization of Occam's Razor. Therefore, the interpretation of the original proof as demonstrating some version of Occam's Razor must also be mistaken. To sum up, Shalizi is arguing that a certain rhetorical/motivational/interpretational notion which often occurs near a specific proof is wrong. I don't think he's concludi

3Wei Dai14y

Looks like I almost missed a very interesting discussion. Hope I'm not too late in joining it. As far as I can tell, you still need the Solomonoff prior for decision making. Kevin T. Kelly's Ockham Efficiency Theorem says that by using Ockham's Razor you minimize reversals of opinion prior to finding the true theory, but that seems irrelevant when you have to bet on something, especially since even after you've found the truth using Ockham's Razor, you don't know that you've found it. Also, I think there's a flaw in Shalizi's argument: But if you're working with a sparse random subset of the rules, why would any of them correctly classify all the data? In algorithmic information theory, the set of rules is universal so one of them is guaranteed to fit the data (assuming the input is computable, which may not be a good assumption but that's a separate issue).

0cousin_it14y

All good points if the universe has important aspects that are computable, which seems uncontroversial to me. Thanks for the link, I'd lost it sometime ago, that thread is pretty epic.

1Daniel_Burfoot14y

The point of Kolmogorov complexity is that there is some limit to how baroque your rules can ever become, or how baroque you can make a fundamentally simple rule by choosing a really weird prior. In algorithmic information theory, the problem of choosing a prior is equivalent to choosing a particular universal Turing machine. If you pick a really weird Turing machine, you will end up assigning low prior probability to things that "normal" people would consider simple - like a sine wave, for example. But because of universal computation, there's a limit to how low a probability you can assign. Your weird machine is still universal, so somehow or other it's got to be able to produce a sine wave, after whatever translation-prefix machinations you have to perform to get it to act like a normal programming language. Another way of viewing this is just to eschew the notion of generalization and state that the goal of learning is compression. If you do this, you end up doing all the same kinds of work, with many fewer philosophical headaches. Now the great deep problem of picking a prior boils down to the rather more quotidian one of picking a data format to use to transmit/encode data.

0cousin_it14y

I don't understand this claim. Surely we can make the probability of a specific program as low as we want by restricting the programming language in ad hoc ways, while letting it stay Turing complete.

1Daniel_Burfoot14y

I agree there is some philosophical greyness here. But let me try again. Let's say we are adversaries. I am going to choose a data set which I claim is simple, and you are going to try to prove me wrong by picking a weird Turing machine which assigns the data a low probability. I generate my data by taking T samples from a sine wave. You pick some strange Turing machine which is designed to be unable to produce sine waves. But regardless of the choice you make, I can always just crank up T to a high enough value so that the compression rate of the data set is arbitrarily close to 100%, proving its simplicity.

0Richard_Kennaway14y

cousin_it quoting Shalizi: But a particular style of baroque elaboration is one that has a short description.

0cousin_it14y

Not necessarily. (Or did I misread your comment?) The particular style can have an arbitrarily long/complex description, and learning will still work as long as the class of described rules is small enough. This observation seems to imply that algorithmic simplicity doesn't play the central role I'd imagined it to play. This is precisely the point where I'd like to hear LW's informed replies.

1Richard_Kennaway14y

Fixed overheads are ignored in the definition of Kolmogorov complexity. A "particular style of baroque elaboration" is simply a program through whose eyes certain rules look short, which look elaborate and baroque through the eyes of another program.

0cousin_it14y

The PAC-learning scenario doesn't have any parameter that goes to infinity, so I'm not sure why you dismiss "fixed" overheads :-)

0Richard_Kennaway14y

Once you've chosen them, they're fixed, and don't run off to infinity. This definition-only-up-to-a-constant is one of the weaknesses of minimum description length. (The other is its uncomputability. Shalizi somewhere else remarks that in discussions of algorithmic complexity, it is traditional to solemnly take out Kolmogorov complexity, exhibit its theoretical properties, remark on its uncomputability, and put it away again before turning to practical matters.) FWIW, this is mine, informed or otherwise. Anyone else have light to shed?

1cousin_it14y

No, let me try nailing this jelly to the wall once again. The definition-only-up-to-a-constant is a weakness of MDL, but this weakness isn't relevant to my question at all! Even if we had some globally unique variant of MDL derived from some nice mathematical idea, learning theory still doesn't use description lengths, and would be perfectly happy with rules that have long descriptions as long as we delineate a small set of those rules. To my mind this casts doubt on the importance of MDL.

4Johnicholas14y

Consider this alternative characterization. Someone wants to fit a polynomial to some data. They pre-selected a sparse set of polynomials, which are in general ridiculously complex. Against all odds, they get a good fit to the training data. This theorem says that, because they haven't examined lots and lots of polynomials, they definitely haven't fallen into the trap of overfitting. Therefore, the good fit to the training data can be expected to generalize to the real data. Shalizi is saying that this story is fine as far as it goes - it's just not Occam's Razor.

2Daniel_Burfoot14y

Good characterization. It's worth noting that learning theory never gives any kind of guarantee that you will actually find a function that provides a good fit to the training data, it just tells you that if you do, and the function comes from a low-complexity set, it will probably give good generalization.

0Daniel_Burfoot14y

Any delineation of a small set of rules leads immediately to a short description length for the rules. You just need to encode the index of the rule in the set, costing log(N) bits for a set of size N. Note that MDL is not the same as algorithmic information theory (definition-up-to-a-constant comes up in AIT, not MDL), though they're of course related.

0cousin_it14y

I see everyone's assuming that some things, by their nature, always go to infinity (e.g. number of samples) while others stay constant (e.g. rule set). This is a nice convention, but not always realistic - and it certainly wasn't mentioned in the original formulation of the problem of learning, where everything is finite. If you really want things to grow, why don't you then allow the set itself to be specified by increasingly convoluted algorithms as N goes to infinity? Like, exponential in N? :-) Learning can still work - in theory, it'll work just as well as a simple rule set - but you'll have a hard time explaining that with MDL. (If there's some theoretical justification why this kind of outrageous bloatware won't be as successful as simple algorithms, I'd really like to hear it...)

2Johnicholas14y

You might find what you're looking for in Kevin T. Kelly's work: http://www.andrew.cmu.edu/user/kk3n/ockham/Ockham.htm Dr. Shalizi mentioned it as an alternative formalization of Occam's Razor.

0cousin_it14y

Johnicolas, thanks. This link and your other comments in those threads are very close to what I'm looking for, though this realization took me some time.

0Richard_Kennaway14y

I think its uncomputability already does that. When you make a computable version by limiting attention to some framework of descriptive capabilities smaller than universal computation, different choices of that framework will give you different measures of simplicity. What is simple in one framework may seem elaborate and baroque in another. Or as some military strategist once put it: "To the foot-soldier, the strategy of a general may seem obscure, shrouded in shadows and fog, but to the general himself, his way is as plain as if he were marching his army down a broad, straight highway."

[-]AnnaSalamon14y2

Any Santa Fe or Albuquerque lesswrong-ers out there, who might want to chat for an hour? I'll be in Santa Fe for a conference from 9/13 to 9/17, and am flying in and out of Albuquerque, and will have some free time Sunday 9/13.

[-]PeerInfinity14y2

I'm going to use this open thread to once again suggest the idea of a Less Wrong video game.

( Here's a link to the post I made last month about it )

After some more thought, I realized that making a game with fancy graphics and complex gameplay would probably not be a good idea for a first project to try.

A better idea would be a simple text-based game you play in your browser, probably running on either PHP or Python.

This might not have as much fun appeal as a traditional video game, since it would probably look like a university exam, but it could still b... (read more)

3DanielLC13y

This could be used to make a game based off of Dungeons and Discourse. When you attack, you have to select an argument without a flaw, or it gets blocked. When the opponent attacks, if you find a flaw, it deals no damage.

2gwern14y

This has already been done many times as part of critical thinking courses; people don't use free sites like http://www.wwnorton.com/college/phil/logic3/ because they're boring and hard. I think the problem is that we lack a good game mechanic. Come up with a mechanic, and the goals can follow, but it's hard to go from a goal like 'calibrate yourself to avoid overconfidence' to a fun game. We need to think about how to borrow games like Zendo and repurpose them.

4PeerInfinity14y

Thanks for the link. That LogicTutor site you linked to provides a good, basic introduction to a few concepts and fallacies. However, the practice problems just ask you to identify which fallacy is in the sentence they give you. They're missing the other half of the game, which is spotting the fallacy in a block of text that's deliberately designed to hide the fallacy. I'll keep looking in case someone has already made a game that contains this part. One way to make the game more fun would be to have interesting text to find the fallacy in. Eliezer's short stories are a good example of this. Though for the purpose of the game, we would need just short segments of stories, which contain one clear example of a fallacy. Preferably one that's well-hidden, but obvious once you see it. Also, to keep players on their toes, we could include segments that don't actually contain a fallacy, and players would have the option of saying that there is no fallacy. And as I mentioned before, another idea is to flesh out the stories even more, so that it could be expanded into a mystery game, or an adventure game, or an escape-the-room game, where in order to continue you need to talk to people, and some of these people will give inaccurate information, because they didn't notice a flaw in their own reasoning, and you will need to point out the flaw in their reasoning before they will give you the accurate information. You would also have to choose your replies during the conversation, and have to choose a reply that doesn't introduce a new fallacy and send the conversation off in the wrong direction. Many of the possible responses would be to question why the person believes specific things that they just said. Maybe there could also be a feature where you could interrupt the person in the middle of what they're saying, to point out the problem. Optionally, score the player based on how long they took, and how many wrong paths they went down before finding the correct path. And

1Eliezer Yudkowsky14y

Well that would be an interesting game mechanic.

4CronoDAS14y

How about a courtroom drama?

1gwern14y

I object! (Alternate Higurashi ending: /me claws throat open to get at insects on me)

0gwern14y

Incidentally, another link right up your alley: http://projects.csail.mit.edu/worldview/about (Starting to think maybe we could use a wiki page, even if only for links and ideas. This game discussion is now spread out over something like 5 LW articles...)

0PeerInfinity14y

5 articles? I only know of 2 articles where this topic was discussed. Can you post a link to the other 3, please? Anyway, I went ahead and created a wiki page for discussing the game idea, and posted links to the two threads I know of: http://wiki.lesswrong.com/wiki/The_Less_Wrong_Video_Game

0gwern14y

It certainly feels like it's been spread out over more than 2 open threads!

[-]taw14y2

Out of pure curiosity, what's your probability distribution of Scientology (or some other such group) being useful? Not the Xenu part, but is it possible that they've discovered some techniques to make people happier, more successful, etc.?

We already have some limited evidence that conventionally religious people are happier, and conventional religions are quite weak.

2conchis14y

But see Will Wilkinson on this too (arguing that this only really holds in the US, and speculating that it's really about "a good individual fit with prevailing cultural values" rather than religion per se).

1taw14y

That's a good counter-argument, but the linked post doesn't actually measure religion-happiness correlation within those other countries (which is the relevant factor), and it's very plausible that European monopolistic religions are far less effective than American freely competing religions for creating happiness.

1conchis14y

The Snoep paper Will linked to measured the correlation for the US, Denmark and the Netherlands (and found no significant correlation in the latter two). The monopolist religion point is of course a good one. It would be interesting to see what the correlation looked like in relatively secular, yet non-monopolistic countries. (Not really sure what countries would qualify though.)

1taw14y

I'm going to completely ignore "statistical significance", as scientific papers are well known to have no idea how to do statistics properly with multiple hypotheses, and can be assumed to be doing it wrong until proven otherwise. If null hypothesis were false, the chance of all almost signs pointing in the same direction would be very low. As far as I can tell what the paper finds out is that religion is less effective in Denmark and Netherlands than in US, but it increases happiness, and it's extremely unlikely to be a false positive result due to chance.

1Jonathan_Graehl14y

I know a scientist who's spent significant money on Landmark Education (nee "est"). He's happy with what he got out of it, but doesn't feel the need for any more of it now.

[-]nerzhin14y2

I'm curious about how Less Wrong readers would answer these questions:

What is your probability estimate for some form of the simulation hypothesis being true?
If you received evidence that changed your estimate to be much higher (or lower), what would you do differently in your life?

3Jonathan_Graehl14y

To answer both, there's no consequence. So I choose not to invent a completely arbitrary prior. I do enjoy fantasizing about possible measurable consequences of particular types of simulations. Perhaps if I'm interesting enough, I'll be copied into other simulations; perhaps we can discover some artifact of variably approximate simulation when no important observer is near, etc.

1[anonymous]14y

A simulation hypothesis such as "our universe is a simulation" is not falsifiable even given perfect knowledge of the universe at some point in time; maybe the universe has a definite beginning and end and it's simulated perfectly the whole way through. Therefore, I'll use the following definition of the simulation hypothesis: "The best description of the universe as we are capable of observing it describes our observations as happening entirely within a simulation crafted by optimizing processes." Let's assume for the sake of convenience that "the" priors for the laws of physics are P, and let's call the distribution of universes that optimizing processes would simulate P'. The only necessary difference between P and P' is that P' is biased toward universes that are easy and/or useful to simulate. How easy a universe is to simulate in general can probably be estimated by how easy a universe is to simulate in itself. We have quantum mechanics but quantum computers have been late in coming, suggesting that our universe would be difficult to simulate. Now, as for utility, evolution optimizes for things that themselves optimize for reproduction, but it also produces optimization for pretty much random things. We can ignore the random things, and ask how useful our universe is for reproduction. I'm guessing that the universe, as it seems to involve lots of pointless computation, is not good for that. So, given the above, I'd estimate the probability as being... oh, how does 20% sound? Now, of course, the other thing to look for in a simulated universe is simulation artifacts: things that seem to not follow the laws of physics, and behaviors that are only approximations to how things should behave. Suffice to say, we haven't seen any of those.

0JGWeissman14y

Quantum computers are computers which use quantum superposition for parallel processing, and are not required for simulating quantum mechanics. And our "classical" computers do in fact take advantage of quantum mechanics, as classical physics does not allow for solid state transistors.

0[anonymous]14y

I don't understand what point you're trying to make here, but classical physics allows for mechanical computers.

0[anonymous]14y

It seems that quantum computers are required for simulating quantum mechanics in sub-exponential time, though.

1JGWeissman14y

When discussing asymptotic algorithmic complexity, you should specify the varying parameter of problem complexity.

0pengvado14y

The usual default parameter is number of bits it takes to write down the problem. It could also be number of particles. Either one works in this case.

0JGWeissman14y

What quantum algorithm for simulating quantum mechanics takes sub-exponential time with respect to the number of particles?

0pengvado14y

I didn't have a particular algorithm in mind when I said that, but since you ask I went and found this one.

0byrnema14y

I consider any evidence that a truly random/spontaneous process occurs is evidence that the universe isn't closed, because something is happening without an internal mechanism to arbitrate it. And here we have the 2008 Nobel prize in physics, "for the discovery of the mechanism of spontaneous broken symmetry in subatomic physics".

2Douglas_Knight14y

I do not think that phrase means what you think it means.

-2byrnema14y

I thought it meant what you linked to, and after checking I'm pretty sure that was what the prize was about. So what do you think about the possibility of a physical mechanism being able to make a free choice? Perhaps some better examples: * spontaneous creation of particles in a vacuum * spontaneous particle decay

0Douglas_Knight14y

QM is deterministic. Spontaneous symmetry breaking also occurs in stat mech, which applies to deterministic classical systems.

0byrnema14y

QM could be interpreted in a deterministic way, but this is not a common view. I would like to learn more about it from you and others here on LW. "Spontaneous" means that something happens without precursor; without any apparent cause. It is orthogonal in meaning to "determined". When you write that spontaneous symmetry breaking is deterministic, perhaps you mean that its description is analytic -- wholly described by a set of deterministic mathematical equations?

0Douglas_Knight14y

Spontaneous symmetry breaking is part of stat mech. It has practically nothing to do with QM. Stat mech can be interpreted probabilistically, but it is not at all controversial to apply it to deterministic systems. Maybe that's a reasonable definition, but you contrasted "spontaneous" with "closed," which is not orthogonal to "determined."

-1byrnema14y

My point was that true randomness of any kind would be evidence that a system is not closed. This might be a novel observation (I haven't heard it before) but I think it is a logical one. It is relevant to reductionism (we wouldn't want supernatural processes swooping down to make choices for our free processes) and whether we are in a simulation. When applied to deterministic systems, the spontaneous symmetry breaking isn't really spontaneous, just apparently so. The idea is that the direction of breaking is determined by the initial conditions, but we may not have enough information about the initial conditions to predict it. It sounds like you want like to argue with whoever is responsible for, "spontaneous symmetry breaking in subatomic physics". I didn't mention QM apart from that.

0[anonymous]14y

All of your examples count as random events with a collapse postulate, but not with many worlds, and hidden-variables have been formulated both ways. Based on your past comments, I assume you already know that. Still, since your examples don't suffice to distinguish interpretations of QM, they also don't suffice to distinguish a universe with randomness from one without. Or are you just pointing out that we should assign higher probability to randomness than we would have if we hadn't observed anything that looked like collapse?

0billswift14y

I'm not much interested in creating bogus/useless "probability estimates". The simulation hypothesis I rate, as I do religion, as "false, barring further evidence". Evidence that the simulation hypothesis is true could be a "physically impossible" inconsistency, like in Heinlein's story "Them". If I became convinced that this was a simulation, I'd become a complete hedonist, why bother with anything else when you are completely under the thumb of whatever's running the simulation.

6Vladimir_Nesov14y

You are completely under the thumb of the physical laws.

1billswift14y

Sorry, but that is a reification of "physical laws"; physical laws aren't a thing, they are simply our description of "how things work".

1Vladimir_Nesov14y

Stuff with which you interact is part of the rules of the game applied to you. The more generally applicable of these rules you call "physical laws". Those are the rules that can't be helped. If you are in the domain of a singleton, then its preference is one of the inescapable laws. You can analyze the raw content of the laws applied to you, just as you can analyze sensory input, and see patterns such as individual agents making decisions that affect your condition. Maybe such patterns are there, maybe they are not, but the judgment of what to do under the given rules must depend on what exactly those patterns are, not just a fact of "their existence".

0byrnema14y

That's interesting: Vladimir_Nesov chastised me for exactly the same thing a month ago. While I probably do reify physical laws, I'm sure he was just applying a metaphor.

1Vladimir_Nesov14y

My point in both cases is more that the concept of "existence" is very low on meaningfulness, that you shouldn't act on mere "existence" of "nonexistence" of something, you must instead understand what that something is.

[-][anonymous]14y1

So, I was thinking about how people conclude stuff. We tend to think of ourselves as having about two levels of conclusion: the "rational" level, which is the level we identify with, considering its conclusions to be our conclusions, and the "emotional" level, which is the one that determines our behavior. (Akrasia is disagreement between the two levels.)

Now, there doesn't seem to be any obvious rule for what becomes a rational level conclusion. If you go outside and wonder at nature, have you proven that God exists? For some people, it... (read more)

[-]Vladimir_Nesov14y1

There is an audio interview with EY on "Make A Public Commitment".

[-]CannibalSmith14y1

Eclipse Phase

Does it deserve a top level post?

3cousin_it14y

I see it tries to push the right buttons, but does it contain any novel insights?

1Kaj_Sotala14y

The plenty of in-group references that are scattered in are almost as good. An excerpt from the backstory: That group sounds kinda familiar. I wonder if they might be inspired by some real-life organization? ETA: Oh, and it gets better. Later on, they mention within the same sentence the "Lifeboat Institute" and the "Singularity Foundation" as two organizations that existed before the Singularity. Hmmmmmmmm...

0CannibalSmith14y

Almost certainly not. After all, that's not the goal. Edit: After 72 pages I retract my previous two statements. Bought, downloaded, read intro. Basically it's Shadowrun IN SPACE! and mixed with Paranoia. A good analogy would be that this tabletop RPG is to Transhumanism what I, Robot the movie is to the book.

0Kaj_Sotala14y

Not really related to rationality, so no top-post if you ask me, but it's a very good setting and an excellent product.

[-]AndrewKemendo14y1

In reading the Singularity Institute's research goals, and the ruminations of Yudkowski, Wei Dai, Nesov et al. in postings here, the approach to developing friendly AI which stands out the most, and from my perspective seems to just always have been the case, seems to be exclusively logic based in the vein of John McCarthy.

I am wondering how the decision was made to focus research for SIAI on the pure logic side, rather than, for example building a synthetic conscious which uses the brain as a model?

To be sure, nearly all AI approaches overlap at some poi... (read more)

5JamesAndrix14y

Such a design would be harder to reason about. Let's say you've got a prototype you want to improve. How do you tell if a proposed change would make it smarter, break it, introduce a subtle cognitive bias, or make the AI want to kill you? In order to set on limits on the kinds of things an AI will do, you need to understand how it works. You can't be experimenting on a structure you partially understand, AND be certain that the experiments won't be fatal. This is easier when you've got a clearly defined structure to the AI, and know how the parts interact, and why.

0AndrewKemendo14y

How is that impossible with a replicated brain architecture? We can't make one if we don't know how it works. Of course. However, how you plan to structure AI what I am asking about. There are many theories about how to structure the AI - so why did the SIAI choose to only focus on a theoretical mathematical logic based approach rather than taking the most advanced, if still flawed, logic device known to man and replicating and improving that?

1Vladimir_Nesov14y

If you have the right tools, you can make a brain without understanding it. Reproductive system can make brains. Whole brain emulation doesn't require understanding of brain architecture, only the dynamics of its lowest-level components. You "know" how pi works, and how to set up a program that computes it, but you don't know what its quandrillionth digit is.

0AndrewKemendo14y

I fear the same philosophical reasoning may be applied to model neural architecture as is currently being used for econometric forecasting. Even the most complex economic models cannot account for significant exogenous variables. For the record I think we can get to WBE, however I think a premature launch would be terrible. Based on the lack of research into developmental AI (much work notably done by a friend - Dr. Frank Guerin at Aberdeen college) I think there is a long way to go. Granting that a brain model or WBE, would be as accurate as the biological version, why then would that not be the most efficient method? The problems with testing and implementation are the same as any other AI, if not easier because of familiarity, however it is grounded on specific biological benchmarks which at that point would be immediately identifiable. I could go on with my particular thoughts as to why biological simulation is in my estimation a better approach, however I am more interested in why the organization (people who have been thinking longer and with more effort than myself) decided otherwise. It would seem that their collective reasoning would give a sufficiently clear and precise answer such that there would be no ambiguity.

1Vladimir_Nesov14y

You have to instill the right preference, and just having a working improved brain doesn't give this capability. You are trying to make an overwhelmingly powerful ally; just making something overwhelmingly powerful is a suicide. As CannibalSmith said, brains are not particularly Friendly. Read the paper.

0AndrewKemendo14y

Of course - we have to BUILD IT RIGHT. I couldn't agree more. The cognitive model does not suggest a mere carbon copy of any particular brain at random, as you know it is not so limited in focus. The fantastic part about the method is at the point in which it is possible to do correctly (not simply an apparent approximation), the tools will likely be available (in the process of it being structured) to correct a large portion of what we identify as fatal cognitive errors. Any errors that are missed it stands to reason would be also missed given the same amount of time with any other developmental structure. I am familiar with the global risk paper you linked, AI: A modern approach which addresses the issue of cognitive modeling as well as Drescher's Good and Real and the problems associated with an FAI. The same existential risks and potential for human disasters are inherent in all AI systems - regardless of the structure, by virtue of it's "power." I think one of the draws to this type of development is the fantastic responsibility which comes with it's development, recognizing and accounting for the catastrophic results that are possible. That said, I have yet to read a decision theoretic explication as to which structure is an optimized method of development, weighing all known limiting factors. I think AI: A modern approach comes closest to doing this but falls short in that it specifically narrows it's focus without a thorough comparison of methods. So again, I ask, by what construct has it been determined that a logical symbolic programming approach is optimized?

0billswift14y

In other words they are doing it where the light's better, rather than where they dropped the keys. Given the track record of correctness proofs in comp sci, I don't think provably Friendly AI is even possible, hopefully I'm wrong there, but all they are doing is further crippling their likelihood of achieving AI before some military or business does.

-1timtyler14y

People control how companies operate - even though they don't understand all the components involved (in particular the brains). Any idea that you have to understand all the components of a system in order to exert a high level of control over it thus appears to have dubious foundations. There are alternative approaches to producing predictable systems which basically involve lots of unit tests. Testing is in vogue in the software engineering world. Few systems are simple enough to prove much about their behaviour. So: to make sure they behave as they are intended, they are intensively tested. It seems likely that machine intelligence will be no different.

5Vladimir_Nesov14y

When a failed test destroys the world, applicability of the normally very useful testing methodology should be reconsidered.

0AndrewKemendo14y

This would be true of any AI. Thus the AI box problem. It is unclear however, how a formal logic approach overcomes this problem and a replication approach does not. They both will need testing, but as you said the methodology should be reconsidered. The easiest way to change testing methodology for logic would be to improve on current logic methodology which has yielded arguably fantastic results - all done by faulty human brains.

-3timtyler14y

Normally testing is done in an offline "testing" mode - using a test harness or sandbox arrangement. Tests themselves are consequently harmless. Of course it is possible for the world to present eventualities that are not modelled by the test suite - but that's usually no big deal. I don't think it is realistic to confine machine intelligence to the domain of provably correct software. Anyone trying that approach would rather obviously be last to the marketplace with a product. I seriously doubt whether paranoid fantasies about DOOM will hinder progress towards machine intelligence significantly. I expect that the prophets of DOOM will be widely ignored. This isn't exactly the first time that people have claimed that the world is going to end.

1DonGeddis14y

Forget about whether your sandbox is a realistic enough test. There are even questions about how much safety you're getting from a sandbox. So, we follow your advice, and put the AI in a box in order to test it. And then it escapes anyway, during the test. That doesn't seem like a reliable plan.

-1timtyler14y

The idea that society is smart enough to build machine intelligence, but not smart enough to build a box to test it in does not seem credible to me: Humans build boxes to put other humans in - and have a high success rate of keeping them inside when they put their minds to it. The few rogue agents that do escape are typically hunted down and imprisoned again. Basically the builders of the box are much stronger and more powerful than what it will contain. Machine intelligence testing seems unlikely to be significantly different from that situation. The cited "box" scenario discusses the case of weak gatekeepers and powerful escapees. That scenario isn't very relevant in this case - since we will have smart machines on both sides when restraining intelligent machines in order to test them.

0thomblake14y

Either massive progress or DOOM will be wrought by those ignoring the DOOM-prophets; either the dynamists win or everyone loses, so the DOOM-prophets lose either way. It seems like a bad business to be in.

0timtyler14y

DOOM is actually big business. Check out all the disaster movies out there. DOOM sells. What could be more important than... THE END OF THE WORLD? What greater cause could there be than... SAVING THE WORLD? So, people buy the DOOM merchandise, contribute the DOOM dollars, and warn their friends about the impending DOOM - thus perpetuating the DOOM virus. That is part of why there have been so many DOOM prophets - DOOM pays.

0JamesAndrix14y

The more I think about this, the more it seems incorrect.

2CannibalSmith14y

Human brains are not particularly friendly.

3AndrewKemendo14y

I would disagree. The overwhelming majority of the average human's life is spent peacefully. It is actually fairly remarkable how rarely we have significant conflict, especially considering the relatively overcrowded places that humans live. Not to mention that it is only a small proportion of the human population that engages other humans destructively (not by proxy).

4Kaj_Sotala14y

The overwhelming majority of the average human's life is also spent in conditions where they are on relatively even grounds with everyone else. But once you start looking at what happens when people end up in situations where they are clearly more powerful than others? And can treat those others the way they like and without fear of retribution? Ugly.

1AndrewKemendo14y

I disagree. While there are some spectacular examples of what you describe, and they are indeed ugly, by and large there is wide distribution of hierarchical disparity even in daily life which is more often than not mutually beneficent. As an emperor I optimize my empire by ensuring that my subjects are philosophically and physically satisfied do I not? I think there is plenty of evidence to support this philosophy as the most sustainable (and positive) of hierarchical models; after all some of the most successful businesses are laterally organized.

0Kaj_Sotala14y

A certain philosophy being the most sustainable and positive isn't automatically the same as being the one people tend to adopt. Plus the answer to your question depends on what you're trying to optimize. Also, it sounds like you're still talking about a situation where people don't actually have ultimate power. If we're discussing a potential hard takeoff scenario, then considerations such as "which models have been the most successful for businesses before" don't really apply. Any entity genuinely undergoing a hard takeoff is one that isn't afterwards bound by what's successful for humans, any more than we are bound by the practices that work the best for ants.

0AndrewKemendo14y

I think there is more than ample evidence to suggest that those are significantly less likely to be adopted - however wouldn't a group of people who know that and can correct for it be the best test case of implementing an optimized strategy? I hold the view that it is unnecessary to hold ultimate power over FAI. I certainly wouldn't bind it to what has worked for humans thus far. Don't fear the AI, find a way to assimilate.

[-]Mitchell_Porter14y0

I have at least one other legacy identity here, dating from the old OB days, "mitchell_porter2". Is there some way to merge us?

[-]timtyler14y0

To quote from http://www.sens.org/files/sens/FHTI07-deGrey.pdf:

"But wait – who’s to say that progress will remain “only” exponential? Might not progress exceed this rate, following an inverse polynomial curve (like gravity) or even an inverse exponential curve? I, for one, don’t see why it shouldn’t. If we consider specifically the means whereby the Singularity is most widely expected to occur, namely the development of computers with the capacity for recursive improvement of their own workings,4 I can see no argument why the rate at which such a comp... (read more)

[-]kpreid14y0

Came across this: What We Can Learn About Pricing From Menu Engineers. Probably nothing new to you/us. Summary: Judgements of acceptable prices are strongly influenced by other seen prices.

[-]Wei Dai14y0

I have a couple of intuitions about the structure of human preferences over a large universe.

The first intuition is that your preferences over one part of the universe (or universe-history) should be independent of what happens in another part of the universe, if the "distance" between the two parts is great enough. In other words, if you prefer A happening to B happening in one part of the universe, this preference shouldn't be reversed no matter what you learn about a distant part of the universe. ("Distance" might be spatial, tempora... (read more)

[-]Wei Dai14y0

Several times recently I asked for simple clarifications about a comment that replied to something I wrote, and had my question ignored. (See here, here, and here.) And I don't know why. Did I violate some rule of etiquette, or what? How can I rephrase my questions to get a better response rate?

ETA: Here are the questions, in case people don't want to search through the comments to find them:

But I'm not sure what you mean by "metaethics, a solved problem". Can you give a link?
What prior work are you referring to, that hasn't been broadly disse

... (read more)

1Eliezer Yudkowsky14y

In my case, what tends to happen is that I either don't notice the question, or I notice that the question requires a bunch of work to respond to and then either get to it some time later or let it slide off entirely.

1CarlShulman14y

"or I notice that the question requires a bunch of work to respond to and then either get to it some time later or let it slide off entirely" Or don't recall it.

0anonym14y

I think in cases like these, you're more likely to get a response by adding another post as a reply to the person with just a single unanswered question (start with the one you care about most), so the person will see they have a new response in their inbox and realize they never answered an earlier question. If you post each of those 3 questions as a response to the person, in context, I'd be very surprised if you didn't get a response to at least 2 of the 3, as long as you include little to nothing else in each post so it's obvious what you're asking for and they can't respond to something else in the post. I've noticed that longish posts with multiple questions often get just one question answered and all the others ignored, intentionally or unintentionally. And posts that are longish with questions interspersed with non-questions tend to get responded too as if the non-questions were the substantive part, with the questions often ignored. (The other extremely common reason for not getting a response is identifying a flaw or asking a question that shows problems with the person's position, in which case most people seem to just ignore the post rather than admit they were wrong or can't answer a critique. I don't think that's the case here at all though.)

1Wei Dai14y

This seems like good advice. I did think about repeating the unanswered question, but was worried that I'd come off as obnoxious if the commenter was avoiding it deliberately for some reason. Given the multiple confirmations that that's probably not the case, I think I'll do so more often in the future. Thanks.

0Vladimir_Nesov14y

Not answering a question is Internet's way of walking away from a conversation. You don't usually say "excuse me, I'm late to a meeting", as interaction is asynchronous. In the current Internet culture, saying "I don't want to bother answering" sounds rude, and so the best solution signaling-wise is to just not answer.

4kpreid14y

In my experience (for myself and interacting with others), it's not uncommon at all for someone to * miss that there was a question at all * see a question, but put off thinking about it and then forget to get back to it * reply to some other part and forget about answering the question * figure replying to the question is uninteresting 'because the thread is dead' etc.

0wedrifid14y

The first two have responses as of the time that you posted this. (If there are remaining questions in a post they are far less likely to be answered after the first couple of replies.) The final example suffers somewhat from 'nobody knows what science doesn't know'. There are probably not too many people who can think of an example of a problem that UDT1 can not handle. For my part I probably wouldn't answer just because I don't like the name UDT1 and the language used to describe it irritates me. I'm not sure why Eleizer didn't answer but I probably wouldn't bother wasting thoughts wondering. Want an answer? Make a top level post about it. Include enough of a useful description of the theory and the problems it has already solved to make you not look bad. In particular, include links to said problems and resolutions. If you really want an answer then include an assertion that UDT1 has solved all the significant decision Problems that have been discussed on LessWrong.

1Wei Dai14y

Please expand on that. Why should I make that much effort to get a simple answer to a simple question? Eliezer obviously had something specific in mind when he wrote "Problem". Why didn't he just write a couple of sentences saying what it was when I indicated that I didn't get the reference? Same with the other two questions. I wasn't asking difficult questions, just simple clarifications.

0wedrifid14y

I'm not suggesting that you ought to have to. I don't think you violated any particular etiquette with your requests for clarification. If going meta and questioning whether the lack of reply is justifiable is your preferred use for the effort then by all means do that instead. I have no idea why Eliezer didn't answer you. Maybe he was busy. Maybe he was self absorbed. But I do make this observation in general: If someone presents a position along the lines of "it is right to believe there is are Problems's with X" then I usually don't expect them to answer me if I press them for an example. This is particularly the case if there are, in fact, no obvious examples. Even if an example could be given, successfully justifying themselves in response to what can be construed as a challenge does not necessarily benefit them. If you present your question in your own frame, however, the dynamics are entirely different.

[-][anonymous]14y0

Has Eliezer written anything good about the evolution of morality? It should probably go on a wiki page titled "Evolution of morality".

ETA: While I'm at it, how about reasons people are religious?

[-]dv82matt14y0

Concerning Newcomb’s Problem I understand that the dominant position among the regular posters of this site is that you should one-box. This is a position I question.

Suppose Charlie takes on the role of Omega and presents you with Newcomb’s Problem. So far as it is pertinent to the problem Charlie is identical to Omega with the notable exception that his prediction is only %55 likely to be accurate. Should you one-box or two-box in this case?

If you one-box then the expected utility is (.55 1,000,000) $550,000 and if you two-box then it is (.45 1,001,000) $450,450 so it seems you should still one-box even when the prediction is not particularly accurate. Thoughts?

1wedrifid14y

Good question. And with Charlie known to be operating exactly as defined then yes, I would one box. I wouldn't call him Charlie however as that leads to confusion. The significant problem with dealing with someone who is taking the role of Omega is in my ability to form a prediction about them that is sufficient to justify the 'cooperate' response. Once I have that prediction the rest, as you have shown, is just simple math.

1dv82matt14y

I don’t think Newcomb’s Problem can easily be stated as a real (as opposed to a simply logical) problem. Any instance of Newcomb’s problem that you can feasibly construct in the real world it is not a strict one shot problem. I would suggest that optimizing a rational agent for the strictly logical one shot problem one is optimizing for a reality that we don’t exist in. Even if I am wrong about Newcomb’s problem effectively being an iterated type of problem treating it as if it is seems to solve the dilemma. Consider this line of reasoning. Omega wants to make the correct prediction. I want Omega to put the million dollars in the box. If I one-box I will either reward Omega for putting the money in the box or punish Omega for not putting the money in the box. Since Omega has a very high success rate I can deduce that Omega puts a high value on making the correct prediction I will therefore put a correspondingly high value on the instrumental value of spending the thousand dollars to influence Omega’s decision. But here’s the thing, this reasoning occurs before Omega even presents you with the problem. It is worked out by Omega running your decision algorithm based on Omega’s scan of your brain. It is effectively the first iteration. You are then presented with the choice for what is effectively the second time and you deduce that any real Omega (as opposed to some platonic ideal of Omega) does something like the sequence described above in order to generate it’s prediction. In Charlie’s case you may reason that Charlie either doesn’t care or isn’t able to produce a very accurate prediction and so reason he probably isn’t running your decision algorithm so spending the thousand dollars to try to influence Charlie’s decision has very low instrumental value. In effect you are not just betting on the probability that the prediction is accurate you are also betting on whether your decision algorithm is affecting the outcome. I’m not sure how to calculate this but t

0[anonymous]14y

It can be stated as real in any and every universe that happens to have an omniscient benefactor who is known to be truthful and prone to presenting such scenarios. It's not real in any other situation. The benefit for optimising a decision making strategy to handle such things as the Newcomb problem is that it is a boundary case. If our decision making breaks down entirely at extreme cases then we can not trust it to be correct.

[-]ciantic14y0

Does this http://www.sciencedaily.com/releases/2009/08/090831130751.htm suffer same problem as:

12 healthy male volunteers were chosen to study what is "just right" amount of beer for driving car. These men consumed doses of beer at 2 bottles, 4 bottles, 8 bottles, and 16 bottles per day for two weeks for each dose amount, with beer being the only alcohol in their diet. Surely the 2 bottles would win, but it definitely ain't the "just right" amount.

Am I missing something in the sciencedaily news, or did they really end up to that conclusion, of 200mg from that test?

0MendelSchmiedekamp14y

Well, the article abstract isn't consistent with the description you linked to. One of the dangers of paraphrasing science.

0JGWeissman14y

From the abstract: "Twelve healthy male volunteers (aged 53–65 yr) were assigned to consume an intake of successively 200, 400, 800, and 1600 mg/d DHA, as the only {omega}-3 fatty acid, for 2 wk each dose." I don't know what inconsistency you noticed between the news article and the abstract, but it seems the abstract itself describes a study that is missing the control group that gets a dosage of 0.

0MendelSchmiedekamp14y

The following sounds like a control measurement was taken: "Blood and urine samples were collected before and after each dose of DHA and at 8 wk after arrest of supplementation." Also note, that the abstract doesn't say that 200mg is ideal as the science daily description does it says: "It is concluded that low consumption of DHA could be an effective and nonpharmacological way to protect healthy men from platelet-related cardiovascular events."

0JGWeissman14y

Taking measurements before and after the treatment is good, but that is not the same as having a separate control group, which could filter out effects of timing, taking the dose with food or water, etc. The abstract also claims "Therefore, supplementation with only 200 mg/d DHA for 2 wk induced an antioxidant effect." It is likely that there was a more complete conclusion in the full article.

0MendelSchmiedekamp14y

But the abstract does not make any "just right" claims, unlike the summary on science daily. Which is what you where complaining about. The abstract reads - we did an incremental test, and even at the lowest dosage we found an effect. This suggest that low dosages could be effective. I don't see anything wrong with that reasoning. The science daily summary is simply misrepresenting it. So, the original commenter isn't missing something in the science news, it is science daily who made the error.

0JGWeissman14y

The news article was not based on the abstract. It was based on the journal article (which is available with a subscription) that the abstract summarized. It is not reasonable to expect that every point in the news article be supported by the abstract.

1Douglas_Knight14y

extremely implausible, as a general rule. ETA:

0JGWeissman14y

So, perhaps the news article was based on press release that was based on the journal article. My point was that it was not produced solely from the abstract.

1Douglas_Knight14y

While the article is more reliable than the abstract, the abstract is more reliable than the press release and the news coverage, because there is better policing of its claims. And the abstract is more policed than the article, so though it may be less reliable because of compression, it is not biased towards sensationalism.

0MendelSchmiedekamp14y

I don't see why this is your point? In the very least it doesn't present counter evidence to my claim that the abstract contains information not present in the news article which mitigates or negates the concerns of the original comment.

0JGWeissman14y

So what? That point was in response to your other claim about what the abstract did not contain.

0MendelSchmiedekamp14y

It's just that with two distinctly different conclusions from the results mentioned from two different sources: the article authors (in the abstract) and Gerald Weissmann, M.D., Editor-in-Chief (in the news article), I place a much lower confidence in later being a reasonable reading of the research paper. But of course we could quite safely argue about readings and interpretations indefinitely. I'd point you to Derrida and Hermeneutics if you want to go that route. In any case, I'll update my estimates on the likelihood of the research paper having an errant conclusion based on Weismann's quote, and I suggest you do the same based on the evidence in the abstract - and then I suspect we have little more to discuss on the subject.

[-]listic14y-4

If noone was using computers today, how would you convince people to use them?

Moderation Log