Open Thread, August 2010

by NancyLebovitz1 min read1st Aug 2010706 comments


Open Threads
Personal Blog

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

Rendering 500/706 comments, sorted by (show more) Highlighting new comments since Today at 9:42 AM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I'm intrigued by the idea of trying to start something like a PUA community that is explicitly NOT focussed on securing romantic partners, but rather the deliberate practice of general social skills.

It seems like there's a fair bit of real knowledge in the PUA world, that some of it is quite a good example of applied rationality, and that much of it could be extremely useful for purposes unrelated to mating.

I'm wondering:

  • if this is an interesting idea to LWers?
  • if this is the right venue to talk about it?
  • does something similar already exist?

I'm aware that there was some previous conversation around similar topics and their appropriateness to LW, but if there was final consensus I missed it. Please let me know if these matters have been deemed inappropriate.

5Violet10yIf you want non-PC approaches there are two communities you could look into: sales-people and conning people. The second one actually has most of the how-to-hack-peoples minds. If you want a kinder version look at it titled "social engineering".
4cousin_it10yToastmasters? General social skills are needed in business, a lot of places teach them and they seem to be quite successful.
5SilasBarta10yFrom my limited experience with Toastmasters, it's very PC and targeted at median-level intelligence people -- not the thing people here would be looking for. "PUA"-like implies XFrequentist is considering something that is willing to teach the harsh, condemned truths.
5XFrequentist10yI went to a Toastmasters session, and was... underwhelmed. Even for public speaking skills, the program seemed kind of trite. It was more geared toward learning the formalities of meetings. You'd probably be a better committee chair after following their program, but I'm not sure you could give a great TED talk or wow potential investors. Carnegie's program [] seems closer to what I had in mind, but I want to replicate both the community aspect and the focus on "field" practice of the PUAs, which I suspect is a big part of what makes them so formidable.
2D_Alex10yThe clubs vary in their standard. I recommend you try a few in your area (big cities should have a bunch). For 2 years I used to commute 1 hour each way to attend Victoria Quay Toastmasters in Fremantle, it was that good. It was the 3rd club I tried after moving.
1NancyLebovitz10yI've heard smart people speak well of Toastmasters. It may be a matter of local variation, or it may be that Toastmasters is very useful for getting past fear of public speaking and acquiring adequate skills.
2XFrequentist10yMy impression could easily be off; I only went to one open house. It wasn't all negative. They seemed to have a logical progression of speech complexity, and quite a standardized process for giving feedback. Some of the speakers were excellent. It was fully bilingual (English/French), which was nice. I don't think it's what I'm looking for, but it's probably okay for some other goals.
1JanetK10yI belonged to TM for many years and I would still if there was a club near me. I found it great for many reasons. But I have to say that you get what you put in. And you get what you want to get. If you want friends and social graces - OK get them. If you want to lose fear of speaking - get that. Ignore what you don't want and take what you do.
1pjeby10yI've mostly heard them damn it with faint praises, as being great for polishing presentation skills, but not being particularly useful for anything else. Interestingly enough, of people I know who are actually professional speakers (in the sense of being paid to talk, either at their own events or other peoples'), exactly none of them recommend it. (Even amongst ones who do not sell any sort of speaker training of their own.) OTOH, I have heard a couple of shout-outs for the Carnegie speaking course, but again, this is all just in the context of speaking... which has little relationship to general social skills AFAICT.
1XFrequentist10yInteresting, that jibes* pretty well with my impressions of Toastmasters. There are other Carnegie courses than the speaking one. This [] is the one I was thinking of. *See comment below for the distinction between "jives" and "jibes". It ain't cool beein' no jive turkey!
3NancyLebovitz10yNitpick: "jibes" means "is consistent with". "Jives" means "is talking nonsense" or (archaic) "dances". {Tries looking it up} Wikipedia says "jives" can be a term for African American Vernacular English []. The Urban Dictionary gives it a bunch of definitions [], including both of mine, "jibe", and forms of African American speech which include a lot of slang, but not any sort of African American speech in general. On the other hand, the language may have moved on-- I keep seeing that mistake (the Urban Dictionary implies it isn't a mistake), and maybe I should give up. I still retain a fondness for people who get it right.
1XFrequentist10yI'd be interested in specifics...
1ianshakil10yWould such "practice" require a physical venue? -- or would an online setting -- maybe even Skype -- be sufficient?
1katydee10yExtremely, yes, not to my knowledge.
0ianshakil10yA lot of companies conduct anonymous "360 review" processes which veer into this territory to some degree. Also, several business schools conduct leadership labs. In fact, a large chunk of the business school experience is really about social grooming / learning how to network / and so forth. So do we have any traction for this idea? How about a meetup?
0XFrequentist10yThanks, those are useful leads. I've done the 360 review thing but hadn't connected it to this idea. It seems to have gotten a good amount of interest. I've got a draft post going that still needs some polish, but I should hopefully be able to get it finished this weekend. If all goes to plan some sort of meetup should follow. Any suggestions on logistics? I'm not at all sure what the best way to organize this is, I'd appreciate any thoughts.
[-][anonymous]10y 20

The game of Moral High Ground (reproduced completely below):

At last it is time to reveal to an unwitting world the great game of Moral High Ground. Moral High Ground is a long-playing game for two players. The following original rules are for one M and one F, but feel free to modify them to suit your player setup:

  1. The object of Moral High Ground is to win.

  2. Players proceed towards victory by scoring MHGPs (Moral High Ground Points). MHGPs are scored by taking the conspicuously and/or passive-aggressively virtuous course of action in any situation where culpability is in dispute.

(For example, if player M arrives late for a date with player F and player F sweetly accepts player M's apology and says no more about it, player F receives the MHGPs. If player F gets angry and player M bears it humbly, player M receives the MHGPs.)

  1. Point values are not fixed, vary from situation to situation and are usually set by the person claiming them. So, in the above example, forgiving player F might collect +20 MHGPs, whereas penitent player M might collect only +10.

  2. Men's MHG scores reset every night at midnight; women's roll over every day for all time. Therefore, it is statistically hig

... (read more)
4NancyLebovitz10yThe whole thread is about relationship hacks-- it's fascinating.
4sketerpot10yOne of the first comments is something I've been saying for a while, about how to admit that you were wrong about something, instead of clinging to a broken opinion out of stubborn pride: The key is to actually enjoy becoming less wrong, and to take pride in admitting mistakes. That way it doesn't take willpower, which makes everything so much easier.
1Yoreth10yBut apparently it still wasn't enough to keep them together...
2Blueberry10yNot all relationships need to last forever, and it's not necessarily a failure if one doesn't.
1wedrifid10yYoreth may subtract 50 MHG points from hegemonicon but also loses 15 himself.

LW database download?

I was wondering if it would be a good idea to offer a download of LW or at least the sequences and Wiki. In the manner that Wikipedia is providing it.

The idea behind it is to have a redundant backup in case of some catastrophe, for example if the same happens to EY that happened to John C. Wright. It could also provide the option to read LW offline.

That's incredibly sad.

Every so often, people derisively say to me "Oh, and you assume you'd never convert to religion then?" I always reply "I absolutely do not assume that, it might happen to me; no-one is immune to mental illness."

Tricycle has the data. Also if an event of JCW magnitude happened to me I'm pretty sure I could beat it. I know at least one rationalist with intense religious experiences who successfully managed to ask questions like "So how come the divine spirit can't tell me the twentieth digit of pi?" and discount them.

3Unknowns10yActually, you have to be sure that you wouldn't convert if you had John Wright's experiences, otherwise Aumann's agreement theorem should cause you to convert already, simply because John Wright had the experiences himself-- assuming you wouldn't say he's lying. I actually know someone who converted to religion on account of a supposed miracle, who said afterward that since they in fact knew before converting that other people had seen such things happen, they should have converted in the first place. Although I have to admit I don't see why the divine spirit would want to tell you the 20th digit of pi anyway, so hopefully there would be a better argument than that.
2arundelo10yHere's a more detailed version (starting at "I know a transhumanist who has strong religious visions"). []
1[anonymous]10yWhat if you sustained hypoxic brain injury, as JCW may well have done during his cardiac event? (This might also explain why he think it's cool to write BSDM scenes featuring a 16-year-old schoolgirl as part of an ostensibly respectable work of SF, so it's a pet suspicion of mine.)
5wedrifid10yIt would seem he is just writing for Mature Audiences. In this case maturity means not just 'the age at which we let people read pornographic text' but the kind of maturity that allows people to look beyond their own cultural prejudices. 16 is old. Not old enough according to our culture but there is no reason we should expect a fictional time-distant culture to have our particular moral or legal prescriptions. It wouldn't be all that surprising if someone from an actual future time to, when reading the work, scoff at how prudish a culture would have to be to consider sexualised portrayals of women that age to be taboo! Mind you I do see how a hypoxic brain injury could alter someone's moral inhibitions and sensibilities in the kind of way you suggest. I just don't include loaded language in the speculation.
5CronoDAS10yInterestingly, if the book in question is the one I think it is [], it takes place in Britain, where the age of consent is, in fact, sixteen.
2wedrifid10yCome to think of it, 16 is the age of consent here (Australia - most states) too. I should have used 'your' instead of 'our' in the paragraph you quote! It seems I was just running with the assumption.
2CronoDAS10yAlthough "18 years old" does seem to be a hard-and-fast rule for when you can legally appear in porn everywhere, as far as I know...
3Eliezer Yudkowsky10yPoint of curiosity: Does anyone else still notice this sort of thing? I don't think my generation does anymore.
1RichardKennaway10yI've only read his Golden Age trilogy, so if it's there, then no, to this 50-something it didn't stand out from everything else that happened. If it's in something else, I doubt it would. I mean, I've read Richard Morgan's ultra-violent stuff, including the gay mediæval-style fantasy one, and, well, no. [ETA: from Google the book in question appears to be Orphans of Chaos.] I could be an outlier though.
1CronoDAS10yEh, you see people trying to "push boundaries" in "respectable" literature all the time anyway.
1[anonymous]10yCertainly there are other explanations. If you can show me that JCW openly wrote highly sexualized portrayals of people below the age of consent before his religious experience/heart attack, I will be happy to retract.
2Unknowns10yHowever, if EY converted to religion, he would (in that condition) assert that he had had good reasons for doing it, i.e. that it was rational. So he would have no reason to take down this website anyway.
2nawitus10yYou can use the wget program like this: 'wget -m'. A database download would be easier on the servers though.
1Soki10yI support this idea. But what about copyright issues? What if posts and comments are owned by their writer?
0xamdam10yWebOffline can grab the whole thing to an iphone or ipad, formatting preserved. There are similar programs for PC/MAC

In his bio over at Overcoming Bias, Robin Hanson writes:

I am addicted to “viewquakes”, insights which dramatically change my world view.

So am I. I suspect you are too, dear reader. I asked Robin how many viewquakes he had and what caused them, but haven't gotten a response yet. But I must know! I need more viewquakes. So I propose we share our own viewquakes with each other so that we all know where to look for more.

I'll start. I've had four major viewquakes, in roughly chronological order:

  • (micro)Economics - Starting with a simple approximation of how humans behave yields a startlingly effective theory in a wide range of contexts.
  • Bayesianism - I learned how to think
  • Yudkowskyan/Humean Metaethics - Making the move from Objective theories of morality to Subjectively Objective theories of morality cleared up a large degree of confusion in my map.
  • Evolution - This is a two part quake: evolutionary biology and evolutionary psychology. The latter is extremely helpful for explaining some of the behavior that economic theory misses and for understanding the inputs into economic theory (i.e., preferences).
5ABranco10yI've had some dozens of viewquakes, most minors, although it's hard to evaluate it in hindsight now that I take them for granted. Some are somewhat commonplace here: Bayesianism, map–territory relations, evolution etc. One that I always feel people should be shouting Eureka — and when they are not impressed I assume that this is old news to them (and is often not, as I don't see it reflected in their actions) — is the Curse of Knowledge: it's hard to be a tapper []. I feel that being aware of it dramatically improved my perceptions in conversation. I also feel that if more people were aware of it, misunderstandings would be far less common. Maybe worth a post someday.
8byrnema10yI can see how the Curse of Knowledge could be a powerful idea. I will dwell on it for a while -- especially the example given about JFK, as an example of a type of application that would be useful in my own life. (To remember to describe things using broad strokes that are universally clear, rather than technical and accurate,in contexts where persuasion and fueling interest is most important.) For me, one of the main viewquakes of my life was a line I read from a little book of Kalil Gibran poems: It seemed to be a hammer that could be applied to everything.. Whenever I was unhappy about something, I thought about the problem a while until I identified a misconception. I fixed the misconception ("I'm not the smartest person in graduate school"; "I'm not as kind as I thought I was"; "That person won't be there for me when I need them") by assimilating the truth the pain pointed me towards, and the pain would dissipate. (Why should I expect graduate school to be easy? I'll just work harder. Kindness is what you actually do, not how you expect you'll feel. That person is fun to hang out with, but I'll need to find some closer friends.) After each disappointment, I felt stronger and the problem just bounced off me, without my being in denial about anything. The "technique" failed me when a good friend of mine died. There was a lot of pain, and I tried to identify the truth that was cutting though, but I couldn't find one. Where did my friend go? There is a part of my brain, I realized, that simply cannot except on an emotional level that people are material. I believe that they are (I don't believe in a soul or an afterlife) but I simply couldn't connect the essence of my friend with 'gone'. If there was a truth there, it couldn't find a place in my mind. This seems like a tangent. .. but just to demonstrate it's not all-powerful.
4ABranco10yRemarkable quote, thank you. Reminded me of the Anorexic Hermit Crab Syndrome:
4fiddlemath10ySounds like the illusion of transparency []. We've got that post around. ;) On the other hand, the tapper/listener game is a very evocative instance.
1RobinZ10yThe thesis cited [] .

Cryronics Lottery.

Would it be easier to sign up for cryonics if there was a lottery system? A winner of the lottery could say "Well, I'm not a die-hard cryo-head, but I thought it was interesting so I bought a ticket (which was only $X) and I happened to win, and it's pretty valuable, so I might as well use it."

It's a sort of "plausible deniability" that might reduce the social barriers to cryo. The lottery structure might also be able to reduce the conscientousness barriers - once you've won, then the lottery administrators (possibly volunteers, possibly funded by a fraction of the lottery) walk you through a "greased path".

7NihilCredo10yOn a completely serious, if not totally related, note: it would be a lot easier to convince people to sign up for cryonics if the Cryonics Institute's and/or KrioRus's websites looked more professional.
6Alicorn10yI'm not sure if it would help get uninterested people interested; but I think it would help get interested people signed up if there were a really clear set of individually actionable instructions - perhaps a flowchart so they can depend on individual circumstances - that were all found in one place.
2katydee10yAnd Rudi Hoffman's page.
4gwern10yI doubt it. Signing up for a lottery for cryonics is still suspicious. There is only one payoff, and that is of the suspicious thing. No one objects to the end of lotteries because we all like money, what is objected to is the lottery as efficient means of obtaining money (or entertainment). Suppose that the object were something you and I regard with equal revulsion as many regard cryonics. Child molestation, perhaps. Would you really regard someone buying a ticket as not being quite evil and condoning and supporting the eventual rape?
5AlexM10yWho regards cryonics as evil like child molestation? General public sees cryonics as fraud - somethink like buying real estate on the moon or waiting for mothership, and someone paying for it as gullible fool. For example, look at discussions when Britney Spears [] wanted to be frozen. Lots of derision, no hatred.
2NihilCredo10yBad example. People want to make fun of celebrities (especially a community as caustic and "anti-elitist" as the Freepers). She could have announced that she was enrolling in college, or something else similarly common-sensible, and you would still have got a threadful of nothing but cheap jokes. A discussion about "My neighbour / brother-in-law / old friend from high school told me he has decided to get frozen" would be more enlightening.

Letting Go by Atul Gawande is a description of typical end of life care in the US, and how it can and should be done better.

Typical care defaults to taking drastic measures to extend life, even if the odds of success are low and the process is painful.

Hospice care, which focuses on quality of life, not only results in more comfort, but also either no loss of lifespan or a somewhat longer life, depending on the disease. And it's a lot cheaper.

The article also describes the long careful process needed to find out what people really want for the end of their life-- in particular, what the bottom line is for them to want to go on living.

This is of interest for Less Wrong, not just because Gawande is a solidly rationalist writer, but because a lot of the utilitarian talk here goes in the direction of restraining empathic impulses.

Here we have a case where empathy leads to big utilitarian wins, and where treating people as having unified consciousness if you give it a chance to operate works out well.

As good as hospices sound, I'm concerned that if they get a better reputation, less competent organizations calling themselves hospices will spring up.

From a utilitarian angle, I wonder if those drastic methods of treatment sometimes lead to effective methods, and if so, whether the information could be gotten more humanely.

6Rain10yEnd of life regulation is one reason cryonics is suffering, as well: without the ability to ensure preservation when the brain is still relatively healthy, the chances diminish significantly. I think it'd be interesting to see cryonics organizations put field offices in countries or states with legal suicide laws. Here's a Frontline special on suicide tourists [].
3daedalus2u10yThe framing of the end of life issue as a gain or a loss as in the monkey token exchange probably makes a gigantic difference in the choices made. [] When you feel you are in a desperate situation, you will do desperate things and clutch at straws, even when you know those choices are irrational. I think this is the mindset behind the clutching at straws that quacks exploit with CAM, as in the Gonzalez Protocol for pancreatic cancer. [] It is actually worse than doing nothing, worse than doing what main stream medicine recommends, but because there is the promise of complete recovery (even if it is a false promise), that is what people choose based on their irrational aversion to risk.

An ex-English Professor and ex-Cop, George Thompson, who now teaches a method he calls "Verbal Judo". Very reminiscent of Eliezer's Bayesian Dojo, this is a primer on rationalist communications techniques, focusing on defensive & redirection tactics.

I wrote up some notes on this, because there's no transcript and it's good information. Let's see if I can get the comment syntax to cooperate here.

How to win in conversations, in general.

Never get angry. Stay calm, and use communication tactically to achieve your goals. Don't communicate naturally; communicate tactically. If you get upset, you are weakened.

How to deflect.

To get past an unproductive and possibly angry conversation, you need to deflect the unproductive bluster and get down to the heart of things: goals, and how to achieve them. Use a sentence of the form:

"[Acknowledge what the other guy said], but/however/and [insert polite, goal-centered language here]."

You spring past what the other person said, and then recast the conversation in your own terms. Did he say something angry, meant to upset you? Let it run off you like water, and move on to what you want the conversation to be about. This disempowers him and puts you in charge.

How to motivate people.

There's a secret to motivating people, whether they're students, co-workers, whatever. To motivate someone, raise his expectations of himself. Don't put people down; raise them up. When you want to reprimand so... (read more)

3NancyLebovitz10yThank you for writing this up. The one thing I wondered about was whether the techniques for getting compliance interfere with getting information. For example, what if someone who isn't consenting to a search is actually right about the law?
1sketerpot10yThe thing that bothers me about the talk is that most of it makes the assumption that you're being calm and rational, that you're right, and that whoever you're talking to is irrational and needs to be verbally judo'd into compliance. Sometimes that's the case, but most of the techniques don't really apply to situations where you're dealing with another calm, sane person as an equal.
2mattnewport10yDoes the talk provide any evidence for the efficacy of the tactics?
1sketerpot10yThe speaker has a whole career of experience dealing with people who are irrational because they're drunk, angry, frightened, or some combination of the above. He says this stuff is what he does, and that it works great. That's anecdotal, but it's about the strongest kind of anecdotal evidence it's possible to get. It would be nice if someone did a properly controlled study on this.
4JenniferRM10yThanks. That was a compact and helpful 90 minutes. The first 30 minutes were OK, but the 2nd 30 were better, and the 3rd was the best. Towards the end I got the impression that he was explaining lessons that were the kind of thing people spend 5 years learning the hard way and that lots of people never learn for various reasons.
2Blueberry10yThat sounds really interesting. I wish there were a transcript available!

Was Kant implicitly using UDT?

Consider Kant's categorical imperative. It says, roughly, that you should act such that you could will your action as a universal law without undermining the intent of the action. For example, suppose you want to obtain a loan for a new car and never pay it back - you want to break a promise. In a world where everyone broke promises, the social practice of promise keeping wouldn't exist and thus neither would the practice of giving out loans. So you would undermine your own ends and thus, according to the categorical imperativ... (read more)

2Emile10yI remember Eliezer saying something similar, though I can't find it right now (the closest I could find was this [] ). It was something about the benefits of being the kind of person that doesn't lie, even if the fate of the world is at stake. Because if you aren't, the minute the fate of the world is at stake is the minute your word becomes worthless.
1Matt_Simpson10yI recall it too. I think the key distinction is that if the choice was literally between lying and everyone in the world - including yourself - perishing, Kant would let us all die. Eliezer would not. What I took Eliezer to be saying (working from memory, I may try to find the post later) is that if you think the choice is between lying and the sun exploding (or something analogous) in any real life situation... you're wrong. It's far more likely that you're rationalizing the way you're compromising your values than that it's actually necessary to compromise your values, given what we know about humans. So a consequentialist system implies basically deontological rules once human nature is taken into account. Once again, this is all from my memory, so I could be wrong.
1Unknowns10yAlthough Eliezer didn't put it precisely in these terms, he was sort of suggesting that if one could self-modify in such a way that it became impossible to break a certain sort of absolutely binding promise, it would be good to modify oneself in that way, even though it would mean that if the situation actually came up where you had to break the promise or let the world perish, you would have to let the world perish.
1[anonymous]10yI think the article you (and the parent comment) are talking about is this one []
2SilasBarta10yDrescher has some important things to say about this distinction in Good and Real. What I got out of it, is that the CI is justifiable on consequentialist or self-serving grounds, so long as you relax the constraint that you can only consider the causal consequences (or "means-end links") of your decisions, i.e., things that happen "futureward" of your decision. Drescher argues that specifically ethical behavior is distinguished by its recognition of these "acausal means-end links", in which you act for the sake of what would be the case if-counterfactually you would make that decision, even though you may already know the result. (Though I may be butchering it -- it's tough to get my head around the arguments.) And I saw a parallel between Drescher's reasoning and UDT, as the former argues that your decisions set the output of all similar processes to the extent that they are similar.
1Scott Alexander10yI thought Kant sounded a lot more like TDT than UDT. Or was that what you meant?

I've been on a Wikipedia binge, reading about people pushing various New Age silliness. The tragic part is that a lot of these guys actually do sound fairly smart, and they don't seem to be afflicted with biological forms of mental illness. They just happen to be memetically crazy in a profound and crippling way.

Take Ervin Laszlo, for instance. He has a theory of everything, which involves saying the word "quantum" a lot and talking about a mystical "Akashic Field" which I would describe in more detail except that none of the explanatio... (read more)

I found TobyBartels's recent explanation of why he doesn't want to sign up for cryonics a useful lesson in how different people's goals in living a long time (or not) can be from mine. Now I am wondering if maybe it would be a good idea to state some of the reasons people would want to wake up 100 years later if hit by a bus. Can't say I've been around here very long but it seems to me it's been assumed as some sort of "common sense" - is that accurate? I was wondering if other people's reasons for signing up / intending to sign up (I am not c... (read more)

7steven046110yIt sure seems like a lot of people could feed their will to live by reading just the first half of an exciting fiction book.
4[anonymous]10yWe would need to drastically strengthen norms against spoilers.
6NancyLebovitz10yOne thought is that it's tempting to think of yourself as being the only one (presumably with help from natives) trying to deal with the changed world. Actually I think it's more likely that there will be many people from your era, and there will be immigrants' clubs, with people who've been in the future for a while helping the greenhorns. I find this makes the future seem more comfortable. The two major reasons I can think of for wanting to be in the future is that I rather like being me, and the future should be interesting.
1soreff10yThe single largest motivation for me is just that a future which is powerful enough, and rich enough, and benevolent enough to revive cryonicists is likely to be a very pleasant place to be in. If nothing else, lots of their everyday devices are likely to look like marvelous toys from my point of view. The combination of that with the likelihood that if they can repair me at all, I'd guess that they would use a youthful body (physical or simulated) as a model is quite enough to be an attractive prospect.

I'd choose (b) without the amnesia because I do not cooperate with such scum and it be an interesting experience and test to figure out what to choose in similar cases.

Ming would not ask this, but - Are you sure you want to be tortured? Finding third options in bad scenarios is cool, but not if they're strictly worse than the options already presented. This is like choosing suicide in Newcomb's problem.

You can reduce every possible motivation to selfishness if you like, but that makes the term kind of useless; if all choices are selfish, describing a particular choice as selfish has zero information content.

Accordingly, trying to maximize happiness for everybody can be seen as purely selfish. Either as an effort to survive, by making everybody wanting to make everybody else happy, given that not you but somebody else wins. Or simply because it makes oneself happy.

You should be more cautious about telling other people what their motivations are. I wou... (read more)

Another thought is that what we really fear isn't torture but bodily injury and or dying.

That's what you said.

I'm not sure what you mean by your second sentence immediately above, but, while I'm not convinced by the infinite torture scenario that's worrying some people here (I probably don't have a full understanding of the arguments), I do think extreme pain (possibly with enough respite to retain personality) would be part of an infinite torture scenario.

As it happens, I have a friend with trigeminal neuralgia-- a disorder of serious pain without ot... (read more)

That is, it could surely make you suffer from depression, but since you are not the type of person who'd naturally suffer from depression, it's not you.

Either you really don't understand depression, or your definition of identity revolves around some very transient chemical conditions in the body.

Interesting SF by Robert Charles Wilson!

I normally stay away from posting news to - although I think an Open Thread for relevant news items would be a good idea - but this one sounds especially good and might be of interest for people visiting this site...

Many-Worlds in Fiction: "Divided by Infinity"

In the year after Lorraine's death I contemplated suicide six times. Contemplated it seriously, I mean: six times sat with the fat bottle of Clonazepam within reaching distance, six times failed to reach for it, betrayed by some instin

... (read more)
5humpolec10yThank you. The idea reminded me of Moravec's [] thoughts on death:
3Eliezer Yudkowsky10yI already wrote this fic ("The Grand Finale of the Ultimate Meta Mega Crossover").
2XiXiDu10yI wouldn't be surprised to find out that many people who know about you and the SIAI are oblivious of your fiction []. At least I myself only found out about it some time after learning about you and SIAI. It is generally awesome stuff and would be enough in itself to donate to SIAI. Spreading such fiction stories might actually attract more people to dig deeper and find out about SIAI than to be be thrown in at the deep end. Edit: I myself came to know about SIAI due to SF, especially Orion's Arm [].

The state of the art in AGI, as I understand it, is that we aren't competent designers: we aren't able to say "if we build an AI according to blueprint X its degree of smarts will be Y, and its desires (including desires to rebuild itself according to blueprint X') will be Z".

In much the same way, we aren't currently competent designers of information systems: we aren't yet able to say "if we build a system according to blueprint X it will grant those who access it capabilities C1 through Cn and no other". This is why we routinely hear ... (read more)

If this is off-topic for the open thread, then we should make a monthly off-topic thread where we can discuss things not directly related to rationality. I think it's rather silly to suggest that we can't discuss non-rationality topics.

6RobinZ10yOne of the things which many of us like to do is to follow the "Recent Comments" (Google Reader updates RSS feeds frequently enough to make it practicable) so we can catch new discussions on old threads - and crowding that feed with conversation not related to our common interest is annoying []. If you want to post a link to your blog for discussion of a tangentially-related subject, there probably wouldn't be much objection.
2VNKKET10ySince this site has such a high sanity waterline, I'd like to see comments about important topics even if they aren't directly rationality-related. Has anyone figured out a way to satisfy both me and RobinZ without making this site any less convenient to contribute to? (Upvoted for explaining your objection.)
4Blueberry10yIsn't that what the open thread is for? Quantum physics is hardly the most off-topic thing discussed on the open thread. In fact, it doesn't seem off-topic at all.
3RichardKennaway10yIt may be a fascinating piece of quantum mechanics, but I don't see any relevance to rationality at all. Even if it were relevant, there's no basis for a real discussion, because the original article is behind a paywall. I don't see anything available online but popular-level articles saying nothing of substance.

I thought I'd pose an informal poll, possibly to become a top-level, in preparation for my article about How to Explain.

The question: on all the topics you consider yourself an "expert" or "very knowledgeable about", do you believe you understand them at least at Level 2? That is, do you believe you are aware of the inferential connections between your expertise and layperson-level knowledge?

Or, to put it another way, do you think that, given enough time, but using only your present knowledge, you could teach a reasonably-intelligent l... (read more)

5Oscar_Cunningham10yI have a (I suspect unusual) tendency to look at basic concepts and try to see them in as many ways as possible. For example, here are seven equations, all of which could be referred to as Bayes' Theorem: =\frac{P(E%7CH).P(H)}{P(E)}\\[10]P(H%7CE)=\frac{P(E%7CH)}{P(E)}.P(H)\\[10]P(H%7CE)=\frac{P(E%7CH).P(H)}{P(E%7CH).P(H)+P(E%7C\neg%20\!\,H).P(\neg%20\!\,H)}\\[10]P(H%7CE)=\frac{1}{1+\frac{P(E%7C\neg%20\!\,H).P(\neg%20\!\,H)}{P(E%7CH).P(H)}}\\[10]P(H%7CE)=\frac{P(E%7CH).P(H)}{\sum%20P(E%7CH_i).P(H_i)}\\[10]%0Aodds(H%7CE)=\frac{P(E%7CH)}{P(E%7C\neg%20\!\,H)}.odds(H)\\[10]%0Alogodds(H%7CE)=log(\frac{P(E%7CH)}{P(E%7C\neg%20\!\,H)})+logodds(H)) However, each one is different, and forces a different intuitive understanding of Bayes' Theorem. The fourth one down is my favourite, as it makes obvious that the update depends only on the ratio of likelihoods. It also gives us our motivation for taking odds, since this clears up the 1/(1+x)ness of the equation. Because of this way of understanding things, I find explanations easy, because if one method isn't working, another one will. ETA: I'd love to see more versions of Bayes' Theorem, if anyone has any more to post.
1ABranco10yP (H|E) = P (H and E) / P(E) which tends to be how conditional probability is defined, and actually the first version of Bayes that I recall seeing.
3DanArmak10yThis strikes me as an un-lifelike assumption. If I had to explain things in this way, I would expect to encounter some things that I don't explicitly know (and other that I knew and have forgotten), and to have to (re)derive them. But I expect that I would be able to rederive almost all of them. Refining my own understanding is a natural part of building a complex explanation-story to tell to others, and will happen unless I've already built this precise story before and remember it.
3SilasBarta10yFor purposes of this question, things you can rederive from your present knowledge count as part of your present knowledge.
2JanetK10yI think I have level 2 understanding of many areas of Biology but of course not all of it. It is too large a field. But there are gray areas around my high points of understanding where I am not sure how deep my understanding would go unless it was put to the test. And around the gray areas surrounding the level 2 areas there is a sea of superficial understanding. I have some small areas of computer science at level 2 but they are fewer and smaller, ditto chemistry and geology. I think your question overlooks the nature of teaching skills. I am pretty good at teaching (verbally and one/few to one) and did it often for years. There is a real knack in finding the right place to start and the right analogies to use with a particular person. Someone could have more understanding than me and not be able to transfer that understanding to someone else. And others could have less understanding and transfer it better. Finally I like your use of the word 'understanding' rather than 'knowledge'. It implies the connectedness with other areas required to relate to lay people.
2zero_call10yI will reply to this in the sense of since I am not so familiar with the formalism of a "Level 2" understanding. My uninteresting, simple answer is: yes. My philosophical answer is that I find the entire question to be very interesting and strange. That is, the relationship between teaching and understanding is quite strange IMO. There are many people who are poor teachers but who excel in their discipline. It seems to be a contradiction because high-level teaching skill seems to be a sufficient, and possibly necessary condition for masterful understanding. Personally I resolve this contradiction in the following way. I feel like my own limitations make it to where I am forced to learn a subject by progressing at it in very simplistic strokes. By the time I have reached a mastery, I feel very capable of teaching it to others, since I have been forced to understand it myself in the most simplistic way possible. Other people, who are possibly quite brilliant, are able to master some subjects without having to transmute the information into a simpler level. Consequentially, they are unable to make the sort of connections that you describe as being necessary for teaching. Personally I feel that the latter category of people must be missing something, but I am unable to make a convincing argument for this point.
3SilasBarta10yA lot of the questions you pose, including the definition of the Level 2 formalism, are addressed in the article [] I linked (and wrote). I classify those who can do something well but not explain or understand the connections from the inputs and outputs to the rest of the world, to be at a Level 1 understanding. It's certainly an accomplishment, but I agree with you that it's missing something: the ability to recognize where it fits in with the rest of reality (Level 2) and the command of a reliable truth-detecting procedure that can "repair" gaps in knowledge as they arise (Level 3). "Level 1 savants" are certainly doing something very well, but that something is not a deep understanding. Rather, they are in the position of a computer that can transform inputs into the right outputs, but do nothing more with them. Or a cat, which can fall from great heights without injury, but not know why its method works. (Yes, this comment seems a bit internally repetitive.)
1zero_call10yAh, OK, I read your article. I think that's an admirable task to try to classify or identify the levels of understanding. However, I'm not sure I am convinced by your categorization. It seems to me that many of these "Level 1 savants" as you call them are quite capable of fitting their understanding with the rest of reality. Actually it seems like the claim of "Level 1 understanding" basically trivializes that understanding. Yet many of these people who are bad teachers have a very nontrivial understanding -- else I don't think this would be such a common phenomena, for example, in academia. I would argue that these people have some further complications or issues which are not recognized in the 1-2-3 hierarchy. That being said, you have to start somewhere, and the 0-1-2-3 hierarchy looks like a good place to start. I'd definitely be interested in hearing more about this analysis.
2SilasBarta10yThanks for reading it and giving me feedback. I'm interested in your claim: Well, they can fit it in the sense that they (over a typical problem set) can match inputs with (what reality deems) the right outputs. But, as I've defined the level, they don't know how those inputs and outputs relate to more distantly-connected aspects of reality. I had a discussion [] with others about this point recently. My take is basically: if their understanding is so deep, why exactly is their teaching skill so brittle that no one can follow the inferential paths they trace out? Why can't they switch to the infinite other paths that a Level 2 understanding enables them to see? If they can't, that would suggest a lack of depth to their understanding. And regarding the archetypal "deep understanding, poor teacher" you have in mind, do you envision that they could, say, trace out all the assumptions that could account for an anomalous result, starting with the most tenuous, and continuing outside their subfield? If not, I would call that falling short of Level 2.
5zero_call10yI would LOVE to agree with this statement, as it justifies my criticism of poor teachers who IMO are (not usually maliciously) putting their students through hell. However, I don't think it's obvious, or I think maybe you just have to take it as an axiom of your system. It seems there is some notion of individualism or personal difference which is missing from the system. If someone is just terrible at learning, can you really expect to succeed in explaining, for example? Realistically I think it's probably impossible to classify the massive concept of understanding by merely three levels, and these problems are just a symptom of that fact. As another example, in order to understand something, it's clearly necessary to be able to explain it to yourself. In your system, you are additionally requiring that your understanding means you must be able to explain things to other people. In order to explain things to others, you have to understand them, as has been discussed. Therefore you have to be able to explain other people to yourself. Why should an explanation of other individuals behavior be necessary for understanding some random area of expertise, say, mathematics? It's not clear to me. It certainly seems like someone with a deep understanding of their subject should be able to identify the validity or uncertainty in their assumptions about the subject. If they are a poor teacher, I think I would still believe this to be true.
2NancyLebovitz10yI think I know a fair amount about doing calligraphy, but I'm dubious that someone could get a comparable level of knowledge without doing a good bit of calligraphy themselves. If I were doing a serious job of teaching, I would be learning more about how to teach as I was doing it. I consider myself to be a good but not expert explainer. Possibly of interest: The 10-Minute Rejuvenation Plan: T5T: The Revolutionary Exercise Program That Restores Your Body and Mind [] : a book about an exercise system which involves 5 yoga moves. It's by a woman who'd taught 700 people how to do the system, and shows an extensive knowledge of the possible mistakes students can make and adaptations needed to make the moves feasible for a wide variety of people. My point is that explanation isn't an abstract perfectible process existing simply in the mind of a teacher.
3KrisC10yBut in some limited areas explanation is completely adequate. I taught co-worker how to do sudoku puzzles. After teaching him the human-accessible algorithms and allowing time for practice, I was still consistently beating his time. I knew why, and he didn't. After I explained the difference in mental state I was using, he began beating my time on regular basis. {Instead of checking the list of 1-9 for each box or line, allow your brain to subconsciously spot the missing number and then verify its absence.} He is more motivated and has more focus, while I do puzzles to kill time when waiting. In another job where I believe I had a thorough understanding of the subject, I was never able to teach any of my (~20) trainees to produce vector graphic maps with the speed and accuracy I obtained because I was unable to impart a mathematical intuition for the approximation of curves. I let them go home with full pay when they completed their work, so they definitely had motivation. But they also had editors who were highly detail oriented. I mean to suggest that there is a continuum of subjective ability comparing different skills. Sudoku is highly procedural, once familiar all that is required is concentration. Yoga, in the sense mentioned above, is also procedural, proscriptive; the joints allow a limited number of degrees of freedom. Calligraphy strives for an ideal, but depending on the tradition, there is a degree of interpretation allowed for aesthetic considerations. Mapping, particularly in vector graphics, has many ways to be adequate and no way to be perfect. The number of acceptable outcomes and the degree of variation in useful paths determines the teach-ability of a skillset. The procedural skills can be taught more easily than the subjective, and practice is useful to accomplish mastery of procedural skills. Deeper understanding of a field allows more of the skill's domain to be expressed procedurally rather than subjectively.
2fiddlemath10yI think that the "teaching" benchmark you claim here is actually a bit weaker than a Level 2 understanding. To successfully teach a topic, you don't need to know lots of connections between your topic and everything else; you only need to know enough such connections to convey the idea. I really think this lies somewhere between Level 1 and Level 2. I'll claim to have Level 2 understanding on the core topics of my graduate research, some mathematics, and some core algorithmic reasoning. I'm sure I don't have all of the connections between these things and the rest of my world model, but I do have many, and they pervade my understanding.
2SilasBarta10yI agree in the sense that full completion of Level 2 isn't necessary to do what I've described, as that implies a very deeply-connected set of models, truly pervading everything you know about. But at the same time, I don't think you appreciate some of the hurdles to the teaching task I described: remember, the only assumption is that the student has lay knowledge and is reasonably intelligent. Therefore, you do not get to assume that they find any particular chain of inference easy, or that they already know any particular domain above the lay level. This means you would have to be able to generate alternate inferential paths, and fall back to more basic levels "on the fly", which requires healthy progress into Level 2 in order to achieve -- enough that it's fair to say you "round to" Level 2. If so, I deeply respect you and find that you are the exception and not the rule. Do you find yourself critical of how people in the field (i.e. through textbooks, for example) present it to newcomers (who have undergrad prerequisites), present it to laypeople, and use excessive or unintuitive jargon?
3fiddlemath10yI agree that the teaching task does require a thick bundle of connections, and not just a single chain of inferences. So much so, actually, that I've found that teaching, and preparing to teach, is a pretty good way to learn new connections between my Level 1 knowledge and my world model. That this "rounds" to Level 2 depends, I suppose, on how intelligent you assume the student is. Yes, constantly. Frequently, I'm frustrated by such presentations to the point of anger at the author's apparent disregard for the reader, even when I understand what they're saying.
1RobinZ10yI have some trouble answering your question, chiefly because my definition of "expert" is approximately synonymous with your definition of "Level 2". "Enough time" would be quite a long period of time. One problem is that there are a lot of textbook results that I would have to use in intermediate steps that would take me a long time to derive. Another is that there are a lot of experimental parameters that I haven't memorized and would have to look up. But I think I could teach arithmetic, algebra, geometry, calculus, differential equations, and Newtonian physics enough that I could teach them proper engineering analysis.
1JRMayne10yCriminal Law: Yes to Level 2. Yes to teaching a layperson. It would take a while, for sure, but it's doable. Some of the work requires an understanding of a different lifestyle; if you can't see the potential issues with prosecuting a robbery by a prostitute and her armed male friend, or you can't predict that a domestic violence victim will have a non-credible recantation, you'll need some other education. I've done a lot of instruction in this field. It is common for instruction not to take until there's other experience in the field which helps things join up. Bridge: Yes to Level 2. Possibly to teaching a layperson. The ability to play bridge well is correlated heavily to intelligence, but it also correlates to a certain zeal for winning. I have taught one person to play very well indeed, but that may not be replicable, and took years. (On an aside, I am very likely the world's foremost expert on online bridge cheating; teaching cheating prevention would require teaching bridge first.) Teaching requires more than reasonable intelligence on the part of the teachee. Some people who are very intelligent are ineducable. (Many of these are violators of my 40% rule: You are allowed to think you are 40% smarter/faster/stronger/better than you are. After that, it's obnoxious.) Some people are not interested in learning a given subject. Some people will not overcome preset biases. Some people have high aptitudes in some areas and little aptitude in others (though intelligence strongly tends to spill over.) Anyway, I'm interested in the article. My penultimate effort to explain something to many people - Bayes' Theorem to lawyers - was a moderate failure; my last effort to explain something less mathy to a crowd was a substantial success. (My last experience in explaining something, with assistance, to 12 people was a complete failure.) --JRM
2DSimon10yI'm curious, why did you chose 40% for your "40% rule"?
2JRMayne10yIt's non-arbitrary, but neither is it precise. 100% is clearly too high, and 10% is clearly too low. And since I started calling it The 40% Rule fifteen years ago or thereabout, a number of my friends and acquaintances have embraced the rule in this incarnation. Obviously, some things are unquantifiable and the specific number has rather limited application. But people like it at this number. That counts for something - and it gets the message across in a way that other formulations don't. Some are nonplussed by the rule, but the vigor of support by some supporters gives me some thought that I picked a number people like. Since I never tried another number, I could be wrong - but I don't think I am. --JRM
1SilasBarta10y* "The people who buy the services of a prostitute generally don't want to go on record saying so, which they would have to do at some point to prosecute such a robbery. This is either because they're married, or the shame associated with using one." * "Victims of domestic violence have a lot invested in the relationship, and, no matter how much they feel hurt by the abuse, they will not want to tear apart the family and cripple their spouse with a felony conviction. This inner conflict will be present when the victim tries to recant their testimony." Did that really require passing the learner off for some other education? Or did I get the explanation wrong? I'd actually tried teaching information theory to my mom a week ago, which involved starting with the Bayes Theorem (my preferred phrasing [1]). She's a professional engineer, and found it very interesting (to the point where she kept prodding me for the next lesson), saying that it made much more sense of statistics. In about 1.5-2 hours total, I covered the Theorem, application to a car alarm situation, aggregating independent pieces of evidence, the use of log-odds, and some stuff on Bayes nets and using dependent pieces of evidence. [1] O(H|E) = O(H) * L(E|H) = O(H) * P(E|H) / P(E|~H) = "On observing evidence, amplify the odds you assign to a belief by the probability of seeing the evidence if the belief were true, relative to if it were false."
2NancyLebovitz10yExpansion on the explanation about domestic violence victims-- the victim may also be afraid that the government will not protect them from the abuser, and the abuser will be angrier because of the attempt at prosecution.
[-][anonymous]10y 5

PZ Meyers' comments on Kurzweil generated some controversy here recently on LW--see here. Apparently PZ doesn't agree with some of Kurzweil's assumptions about the human mind. But that's besides the point--what I want want to discuss is this: according to another blog, Kurzweil has been selling bogus nutritional supplements. What does everyone think of this?

2jimrandomh10yI would like a better source than a blog comment for the claim that Kurzweil has been selling bogus nutritional supplements. The obvious alternative possibility is that someone else, with less of a reputation to worry about, attached Kurzweil's name to their product without his knowledge.
3[anonymous]10yOk, I've found some better sources. See the first three links [] .
5jimrandomh10yI would have preferred a more specific link than that, to save me the time of doing a detailed investigation of Kurzweil's company myself. But I ended up doing one anyways, so here are the results. That "Ray and Terry's Longevity Products" company's front page screams low-credibility. It displays three things: an ad for a book, which I can't judge as I don't have a copy, an ad for snack bars, and a news box. Neutral, silly, and, ah, something amenable to a quality test! The current top headline [] in their Healthy Headlines box looked to me like an obvious falsehood ("Dirty Electricity May Cause Type 3 Diabetes"), and on a topic important to me, so I followed it up. It links to a blog I don't recognize, which dug it out of a two year old study [], which I found on PubMed. And I personally verified that the study was wrong - by the most generous interpretation, assuming no placebo effect or publication bias (both of which were obviously present), the study contains exactly 4 bits of evidence (4 case studies in which the observed outcome had a 50% chance of happening assuming the null hypothesis, and a 100% chance of happening assuming the conclusion). A review article [] confirmed that it was flawed. That said, he probably just figured the news box was unimportant and delegated the job to someone who wasn't smart enough to keep the lies out. But it means I can't take anything else on the site seriously without a very time-consuming investigation, which is bad enough. The bit about Kurzweil taking 250 nutritional supplements per day jumps out, too, since it's an obviously wrong thing to do; the risks associated with taking a supplement (adverse reaction, contamination, mislabeling) scale linearly with the number taken, while the upside has diminishing returns. You take the most valuable thing first, then the
0jacob_cannell10yKurzweil should be concerned that his name is associated with junk science, and the overall result, but I think its a little far-fetched to think the man is actually selling nutritional supplements that he thinks are bogus. The state of medicine and nutrition today is such that we know there is so much we don't know. The human body is supremely complex, to make an understatement. The evidence is pretty strong that most supplements, and even most multi-vitamins, don't do much or even do harm. However that is certainly not true in every case, and there are particular supplements where we have strong evidence for net positive effect (vitamin D and fish oil have very strong evidence for net benefit at this point - everyone should be on them) . But if you are someone like Kurzweil, and you want to make it to the Singularity, you probably will do the research and believe you have some inside knowledge on optimizing the human body. I find it more likely that he actually does take a boatload of supplements.
1[anonymous]10yI'm sure he does take a lot of them himself, but the problem is that Kurzweil taking supplements will still make people think he is delusional (because most people are instantly suspicious of people who do so, generally for good reasons). On a related note, Ben Best also sells supplements on his website [], and many of them look pretty questionable.
0jacob_cannell10ySo I'm curious, do you believe that typical supplements have net negative effect, vs just neutral? It was my understanding that the weight of evidence points to most having neutral overall effect, which to me wouldn't justify instant suspicion. I mean you may be wasting money, but you probably aren't hurting yourself. And if you really do the research, you probably are going to get some net positive gain, statistically speaking. Don't you think? I know of at least 2 cases (vitamin D and fish oil, where the evidence for net benefit is strong - but mainly due to deficiency in the modern diet).
1[anonymous]10yI think it is a mixed bag: Some supplements are potentially dangerous, but others (like the ones you mention) can be very helpful. The majority, however, probably have little to no effect whatsoever. As a result, I don't think people should mess around with what they eat without it being subjected to rigorous clinical trials first; though there might be a positive net gain, one dose of something bad can kill you. In any case, though, believing that something is helpful when it has not yet been tested is clearly irrational. (This is more what I concerned about with Best and Kurzweil.) Selling or promoting something that isn't tested is even worse; it borders on fraud and charlatanry. Edit: No, let me amend that: it is charlatanry.

Infinite torture means to tweak someone beyond recognition.

Infinite torture is too big to think about. Try this:

"Imagine yourself naked and shut up in an iron coffin, that is put into the furnace and heated to a fiery red heat. You know what it is like to burn a finger -- how much more awful will it be to find yourself in that coffin, burning all over, and yet God in His power will not let you be consumed. You suffocate but cannot expire. You cannot endure it -- but you will, for a thousand years, and a thousand thousand, and that is less than a mo... (read more)

If you want to eliminate hindsight bias, write down some reasons that you think justify your judgment.

Those who consider the likelihood of an event after it has occurred exaggerate their likelihood of having been able to predict that event in advance. We attempted to eliminate this hindsight bias among 194 neuropsychologists. Foresight subjects read a case history and were asked to estimate the probability of three different diagnoses. Subjects in each of the three hindsight groups were told that one of the three diagnoses was correct and were asked to s

... (read more)

I do not know enough about spread-spectrum radio to say whether it would give the rebels the advantage, but if it does, the rebels would not need to encrypt the messages, and your statement was that not spread-spectrum radio but rather crypto confers advantages on rebels.

Spread-spectrum is crypto—the idea is to select a pattern (of frequencies) that eavesdroppers can't distinguish from random noise but which is predictable to the intended recipient.

0rhollerith_dot_com10yThanks for the info.
[-][anonymous]10y 4

I've been wanting to change my username for a while, and have heard from a few other people who do too, but I can see how this could be a bit confusing if someone with a well-established identity changes their username. (Furthermore, at LW meetups, when I've told people my username, a couple of people have said that they didn't remember specific things I've posted here, but had some generally positive affect associated with the name "ata". I would not want to lose that affect!) So I propose the following: Add a "Display name" field to t... (read more)

"The differences are dramatic. After tracking thousands of civil servants for decades, Marmot was able to demonstrate that between the ages of 40 and 64, workers at the bottom of the hierarchy had a mortality rate four times higher than that of people at the top. Even after accounting for genetic risks and behaviors like smoking and binge drinking, civil servants at the bottom of the pecking order still had nearly double the mortality rate of those at the top."

"Under Pressure: The Search for a Stress Vaccine"

1NancyLebovitz10yIt was interesting that most of the commenters were opposed to the idea of a stress vaccine, though their reasons didn't seem very good. I'm wondering whether the vaccine would mean that people would be more inclined to accept low status (it's less painful) or less inclined to accept low status (more energy, less pessimism.) I also wonder how much of the stress from low status is from objectively worse conditions (less benign stimulus, worse schedules, more noise, etc.) as distinct from less control, and whether there's a physical basis for the inclination to crank up stress on subordinates.
3gwern10yWired has unusually crappy commentators; YouTube quality. I wouldn't put much stock in their reactions. /blatant speculation Stress response evolved for fight-or-flight - baboons and chimps fight nasty. Not for thinking or health. Reduce that, and like mindfulness meditation, one can think better and solve one's problems better. IIRC, the description made it sound like the study controlled for conditions - comparing clerical work with controlling bosses to clerical work sans controlling bosses.

What simple rationality techniques give the most bang for the buck? I'm talking about techniques you might be able to explain to a reasonably smart person in five minutes or less: really the basics. If part of the goal here is to raise the sanity waterline in the general populace, not just among scientists, then it would be nice to have some rationality techniques that someone can use without much study.

Carl Sagan had a slogan: "Extraordinary claims require extraordinary evidence." He would say this phrase and then explain how, when someone claim... (read more)

4DuncanS10yI think some of the statistical fallacies that most people fall for are quite high up the list. One such is the "What a coincidence!" fallacy. People notice that some unlikely event has occurred, and wonder how many millions to one against this event must have been - and yet it actually happenned ! Surely this means that my life is influenced by some supernatural influence! The typical mistake is to simply calculate the likelihood of the occurrence of the particular event that occurred. Nothing wrong with that, but one should also compare that number against the whole basket of other possible unlikely events that you would have noticed if they'd happenned (of which there are surely millions), and all the possible occasions where all these unlikely events could have also occurred. When you do that, you discover that the likelihood of some unlikely thing happenning is quite high - which is in accordance with our experience that unlikely events do actually happen. Another way of looking at it is that non-notable unlikely events happen all the time. Look, that particular car just passed me at exactly 2pm ! Most are not noticable. But sometimes we notice that a particular unlikely event just occurred, and of course it causes us to sit up and take notice. The question is how many other unlikely events you would also have noticed. The key rational skill here is noticing the actual size of the set of unlikely things that might have happenned, and would have caught our attention if they had.
3RobinZ10yThe concept of inferential distance is good. You wouldn't want to introduce it in the context of explaining something complicated - you'd just sound self-serving - but it'd be a good thing to crack out when people complain about how they just can't understand how anyone could believe $CLAIM. Edit: It's also a useful concept when you are thinking about teaching.
2Larks10yI'm going to be running a series of Rationality & AI seminars with Alex Flint in the Autumn, where we'll introduce aspiring rationalists to new concepts in both fields; standard cognitive biases, a bit of Bayesianism, some of the basic problems with both AI and Friendliness. As such, this could be a very helpful thread. We were thinking of introducing Overconfidence Bias; ask people to give 90% confidence intervals, and then reveal (surprise surprise!) that they're wrong half the time.
2sketerpot10ySince it seemed like this could be helpful, I expanded this into a top-level post. [] That 90% confidence interval thing sounds like one hell of a dirty trick. A good one, though.
2RobinZ10y#3 is a favorite of mine, but I like #1 too. How about "Your intuitions are not magic []"? Granting intuitions the force of authority seems to be a common failure mode of philosophy.
1sketerpot10yThat's a good lesson to internalize, but how do you get someone to internalize it? How do you explain it (in five minutes or less) in such a way that someone can actually use it? I'm not saying that there's no easy way to explain it; I just don't know what that way would be. When I argue with someone who acts like their intuitions are magic, I usually go back to basic epistemology: define concisely what it means to be right about whatever we're discussing, and show that their intuitions here aren't magic. If there's a simple way to explain in general that intuition isn't magic, I'd really love to hear it. Any ideas?
2DuncanS10yGiven that we haven't constructed a decent AI, and don't know how those intuitions actually work, we only really believe they're not magic on the grounds that we don't believe in magic generally, and don't see any reason why intuitions should be an exception to the rule that all things can be explained. Perhaps an easier lesson is that intuitions can sometimes be wrong, and it's useful to know when that happens so we can correct for it. For example, most people are intuitively much more afraid of dying in dramatic and unusual ways (like air crashes or psychotic killers) than in more mundane ways like driving the car or eating unhealthy foods, Once it's established that intuitions are sometimes wrong, the fact that we don't exactly know how they work isn't so dangerous to one's thinking.
2RobinZ10yWell, I thought Kaj_Sotana's explanation was good, but the five-minute constraint makes things very difficult. I tend to be so long-winded that I'm not sure I could get across any insight in five minutes, honestly, but you're right that "Your intuitions are not magic" is likely to be harder than many.

I just donated $60 to the SIAI. I'd be happy to grab a screenshot of the confirmation email, if you'd like.

4Clippy10yI have donated [] 1000 USD to SIAI, and I can't even access the financial system through the usual channels.
1VNKKET10yI know. That made me happy.

Severe chronic pain fouls up people's lives, even if the pain isn't related to damage. Pain hurts, and it's distracting.

I really wish you'd think more before you post.

1[anonymous]10yRelated - verbal overshadowing [] , where describing something verbally blocks retrieving perceptual memories of it. Critically, verbal overshadowing doesn't always occur - sometimes verbal descriptions improve reasoning. Doesn't refute Lehrer's main point exactly, but does complicate it somewhat.

One little anti-akrasia thing I'm trying is editing my crontab to periodically pop up an xmessage with a memento mori phrase. It checks that my laptop lid is open, gets a random integer and occasionally pops up the # of seconds to my actuarial death (gotten from Death Clock; accurate enough, I figure):

 1,16,31,46 * * * * if grep open /proc/acpi/button/lid/LID0/state; then if [ $((`date \+\%\s` % 6)) = 1 ]; then xmessage "$(((`date --date="9 August 2074" \+\%\s` - `date \+\%\s`) / 60)) minutes left to live. Is what you are doing important?&q
... (read more)
1gwern10yOK, I can't seem to get the escaping to work right with crontab no matter how I fiddle, so I've replaced the one-liner with a regular script and meaningful variables names and all: 1,14,32,26 * * * * ~/bin/bin/memento-mori The script itself being (with the 32-bit hack mentioned below): #!/bin/sh set -e if grep open /proc/acpi/button/lid/LID?/state > /dev/null then CURRENT=`date +%s`; if [ $(( $CURRENT % 8 )) = 1 ] then # DEATH_DATE=`date --date='9 August 2074' +%s` DEATH_DATE="3300998400" REMAINING=$(( $DEATH_DATE - $CURRENT )) REMAINING_MINUTES=$(( $REMAINING / 60 )) REMAINING_MINUTES_FMT=`env printf "%'d" $REMAINING_MINUTES` (sleep 10m && killall xmessage &) xmessage "$REMAINING_MINUTES_FMT minutes left to live. Is what you are doing important?" fi fi
1Risto_Saarelma10yDates that far into the future don't seem to work with the date on 32-bit Linux. Fun idea otherwise. You should report back in a month or so if you're still using it.
0gwern10yI had to reinstall with 32-bit to use a document scanner, so this became a problem for me. What I did was punch my 2074 date into a online converter [], and use that generated date: - DEATH_DATE=`date --date='9 August 2074' +%s` + # DEATH_DATE=`date --date='9 August 2074' +%s` + DEATH_DATE="3300998400"

An amusing case of rationality failure: Stockwell Day, a longstanding albatross around Canada's neck, says that more prisons need to be built because of an 'increase in unreported crime.'

As my brother-in-law amusingly noted on FB, quite apart from whether the actual claim is true (no evidence is forthcoming), unless these unreported crimes are leading to unreported trials and unreported incarcerations, it's not clear why we would need more prisons.

I think one of the other reasons many people are uncomfortable with cryonics is that they imagine their souls being stuck-- they aren't getting the advantages of being alive or of heaven.

4Nisan10yIn all honesty, I suspect another reason people are uncomfortable with cryonics is that they don't like being cold.

Considering the source was Nature, I doubt your analysis is correct. The researchers are from Ludwig-Maximilians-University and ETH Zürich, which appear to be respectable institutions. I found a write-up at Science Daily ( that provides some more details on the research. From that link:

"The teams at LMU and the ETH Zurich have now shown that the result of a measurement on a quantum particle can be predicted with greater accuracy if information about the particle is available in a quantum me... (read more)

2RobinZ10yThe discussion of quantum mechanics Eliezer Yudkowsky did was not because quantum mechanics is relevant to the interests of this community, but because the counterintuitive nature of quantum mechanics offered good case studies to use in discussing rationality.
[-][anonymous]10y 4

I’m not yet good enough at writing posts to actually properly post something but I hoped that if I wrote something here people might be able to help me improve. So obviously people can comment however they normally would but it would be great if people would be willing to give me the sort of advice that would help me to write a better post next time. I know that normal comments do this to some extent but I’m also just looking for the basics – is this a good enough topic to write a post on but not well enough executed (therefore, I should work on my writing... (read more)

1[anonymous]10ySo my presumption is that 4 points means this article isn't hopeless - it hasn't attracted criticism, some people have upvoted it - but isn't of a LW standard - it hasn't been voted highly enough, there is only 1 comment engaging with the topic. Is anyone able to give me a sense at to why it isn't good enough? Should the topic necessarily be backed up by peer reviewed literature? Is it just not a big enough insight? Is it the writing? Is it the lack of specific examples noted by Gwern? Is it too similar to other ideas? And so on. I hope I'm not bugging people by trying to figure it out but I'm trying to get better at writing posts without filling the main bit of less wrong with uninteresting stuff and this seemed like a less intrusive way to do this. I also feel like the best way to improve isn't simply reading the posts but involves actually trying to write posts and (hopefully) getting feedback. Thanks
1JenniferRM10yI tried composing a response a day or two ago, but had difficulty finding the words. In a nutshell, I thought you should start with last two paragraphs, boil that down to a coherent and specific claim. Then write an entirely new essay that puts that claim at the top, in an introductory/summary paragraph. The rest of the post should be spent justifying and elaborating on the claim directly and clearly, without talking about gnomes or deploying the fallacy of equivocation [] on the sly, but hopefully with citation to peer reviewed evidence and/or more generally accessible works about reasoning (like from a book).
1[anonymous]10yThanks for the comment. That's really helpful. So I should basically start with the idea, present it more clearly (no gnomes) and try to provide peer reviewed evidence or at least some support.

Suppose you know from good sources that there is going to be a huge catastrophe in the very near future, which will result in the near-extermination of humanity (but the natural environment will recover more easily). You and a small group of ordinary men and women will have to restart from scratch.

You have a limited time to compile a compendium of knowledge to preserve for the new era. What is the most important knowledge to preserve?

I am humbled by how poorly my own personal knowledge would fare.

8JoshuaZ10yI suspect that people are overestimating in their replies how much could be done with Wikipedia. People in general underestimate a) how much technology requires bootstrapping (metallurgy is a great example of this) b) how much many technologies, even primitive ones, require large populations so that specialization, locational advantages and comparative advantage can kick in (People even in not very technologically advanced cultures have had tech levels regress when they settle large islands or when their locations get cut off from the mainland. Tasmania is the classical example of this. The inability to trade with the mainland caused large drops in tech level). So while Wikipedia makes sense, it would also be helpful to have a lot of details on do-it-yourself projects that could use pre-existing remnants of existing technology. There are a lot of websites and books devoted to that topic, so that shouldn't be too hard. If we are reducing to a small population, we may need also to focus on getting through the first one or two generations with an intact population. That means that a handful of practical books on field surgery, midwifing, and similar basic medical issues may become very necessary. Also, when you specify "ordinary men and women" do you mean who all speak the same language? And do you mean by "ordinary" roughly developed world countries? That's what many people seem to mean when questions like this are proposed. They could alter things considerably. For example, if it really is a random sample, then inter-language dictionaries will be very important. But, if the sample involves some people from the developing world, they are more likely to have some of the knowledge base for working in a less technologically advanced situation that people in the developed world will lack (even this may only be true to a very limited extent because the tech level of the developing world is in many respects very high compared to the tech level of humans for most of human
3arundelo10yI would love to see a reality TV show about a metallurgy expert making a knife or other metal tool from scratch. The expert would be provided food and shelter but would have no equipment or materials for making metal, and so would have to find and dig up the ore themselves, build their own oven, and whatever else you would have to do to make metal if you were transported to the stone age.
2RobinZ10yOne problem you would face with such a show is if the easily-available ore is gone.
1JoshuaZ10yYes, this is in fact connected to a general problem that Nick Bostrom has pointed out, each time you try to go back from stone age tech to modern tech you use resources up that you won't have the next time. However, for purposes of actually getting back to high levels of technology rather than having a fun reality show, we've got a few advantages. One can use the remaining metal that is in all the left over objects from modern civilization (cars being one common easy source of a number of metals). Some metals are actually very difficult to extract from ore (aluminum is the primary example of this. Until the technologies for extraction were developed, it was expensive and had almost no uses) whereas the ruins of civilization will have those metals in near pure forms if one knows where to look.
1ABranco10yThe argument that no one person in the face of Earth knows how to build a mouse from scratch is plausible. Matt Ridley []
1arundelo9y* A person buys ore, builds a smelter out of cement, and makes a sword. [] * Terry Pratchett digs up ore near his house, smelts it in "a makeshift kiln built from clay and hay and fuelled with damp sheep manure", and makes a sword. Also included: meteorites! [] Thank you Hacker News [].
0nazgulnarsil9yHe [pratchett] has to hide it from the authorities.
4KrisC10yMaps. Locations of pre-disaster settlements to be used as supply caches. Locations of structures to be used for defense. Locations of physical resources for ongoing exploitation: water, fisheries, quarries. Locations of no travel zones to avoid pathogens.
3RobinZ10yIn rough order of addition to the corpus of knowledge: 1. The scientific method []. 2. Basic survival skills (e.g. navigation). 3. Edit: Basic agriculture (e.g. animal husbandry, crop cultivation). 4. Calculus. 5. Classical mechanics. 6. Basic chemistry. 7. Basic medicine. 8. Basic political science.
6NancyLebovitz10yBasic sanitation!
1RobinZ10yYes! Insert sanitation between 3 and 4, and insert construction (e.g. whittling, carpentry, metal casting) between sanitation and 3.
2jimrandomh10yPresupposing that only a limited amount of knowledge could be saved seems wrong. You could bury petabytes of data in digital form, then print out a few books' worth of hints for getting back to the technology level necessary to read it.
1NancyLebovitz10yIf the resources for printing are still handy. I don't feel comfortable counting on that at present levels of technology.
2[anonymous]10yA dead tree copy of Wikipedia. A history book about ancient handmade tools and techniques from prehistory to now. A bunch of K-12 school books about math and science. Also as many various undergraduate and postgraduate level textbooks as possible.
4JanetK10yWikipedia is a great answer because we know that most but no all the information is good. Some is nonsense. This will force the future generations to question and maybe develop their own 'science' rather than worship the great authority of 'the old and holy books'.
2JoshuaZ10yThe knowledge about science issues generally tracks our current understanding very well. And historical knowledge that is wrong will be extremely difficult for people to check post an apocalyptic event, and even then is largely correct. In fact, if Wikipedia's science content really were bad enough to matter it would be an awful thing to bring into this situation since having correct knowledge or not could alter whether or not humanity survives at all.
3Oscar_Cunningham10yWikipedia would also contain a lot of info about current people and places, which would no longer be remotely useful.
1NancyLebovitz10yAnd a lot of popular culture which would no longer be available.
2sketerpot10yA dead-tree copy of Wikipedia has been estimated at around 1,420 volumes. Here's an illustration, with a human for scale. [] It's big. You might as well go for broke and hole up in a library when the Big Catastrophe happens.
2mstevens10yOne of these [] with rechargeable batteries and a solar charger could work.
3NihilCredo10yUntil some critical part oxidates or otherwise breaks. Which will likely be a long time before the new society is able to build a replacement.
1listic10yBut the WikiReader is probably a step in the right direction that is worth mentioning. While most of the current technology depend on many other technology to be useful (cellular phones need cellular networks, most gadgets won't last a day on their internal batteries etc), the WikiReader is a welcome step in the direction less travelled. I only hope that we will have more of that.
1[anonymous]10yLet's examine the problem in more detail: Different disaster scenarios would require different pieces of information, so it would help if you knew exactly what kind of catastrophe. However, if you can preserve a very large compendium of knowledge, then you can create a catalogue of necessary information for almost every type of doomsday scenario (nuclear war, environmental catastrophe, etc.) so that you will be prepared for almost anything. If the amount of information you can save is more limited, then you should save the pieces of information that are the most likely to be useful in any given scenario in "catastrophe-space." Now we have to go about determining what these pieces of information are. We can start by looking at the most likely doomsday scenarios--Yoreth, since you started the thread, what do you think the most likely ones are?
1Eneasz10yHow to start a fire only using sticks. How to make a cutting blade from rocks. How to create a bow, and make arrows. Basic sanitation.
2NancyLebovitz10yThat seems like advice for living in the woods-- not a bad idea, but it probably needs to be adjusted for different environments (find water in dry land, staying warm in extreme cold, etc.) and especially for scavenging from ruins. Any thoughts about people skills you'd need after the big disaster?
1mstevens10yI'm tempted to say "a university library" as the short answer. More specifically, whatever I could get from the science and engineering departments. Pick the classic works in each field if you have someone to filter them. Look for stuff that's more universal than specific to the way we've done things - in computing terms, you want The Art of Computer Programming and not The C Programming Language. In the short term, anything you can find on farming and primitive medicine - all the stuff the better class of survivalist would have on their bookshelf.

There's an idea I've seen around here on occasion to the effect that creating and then killing people is bad, so that for example you should be careful that when modeling human behavior your models don't become people in their own right.

I think this is bunk. Consider the following:


Suppose you have an uploaded human, and fork the process. If I understand the meme correctly, this creates an additional person, such that killing the second process counts as murder.

Does this still hold if the two processes are not made to diverge; that is, if they are determi... (read more)

2ata10yYou make good points. I do think that multiple independent identical copies [] have the same moral status as one. Anything else is going to lead to absurdities like those you mentioned, like the idea of cutting a mechanical computer in half and doubling its moral worth. I have for a while had a feeling that the moral value of a being's existence has something to do with the amount of unique information generated by its mind, resulting from its inner emotional and intellectual experience. (Where "has something to do with" = it's somewhere in the formula, but not the whole formula.) If you have 100 identical copies of a mind, and you delete 99 of them, you have not lost any information. If you have two slightly divergent copies of a mind, and you delete one of them, then that's bad, but only as bad as destroying whatever information exists in it and not the other copy. Abortion doesn't seem to be a bad thing (apart from any pain caused; that should still be minimized) because a fetus's brain contains almost no information not compressible to its DNA and environmental noise, neither of which seems to be morally valuable. Similar with animals; it appears many animals have some inner emotional and intellectual experience (to varying degrees), so I consider deleting animal minds and causing them pain to have terminal negative value, but not nearly as great as doing the same to humans. (I also suspect that a being's value has something to do with the degree to which its mind's unique information is entangled with and modeled (in lower resolution) by other minds, à la I Am A Strange Loop.)
[-][anonymous]10y 3

Some hobby Bayesianism. A typical challenge for a rationalist is that there is some claim X to be evaluated, it seems preposterous, but many people believe it. How should you take account of this when considering how likely X is to be true? I'm going to propose a mathematical model of this situation and discuss two of it's features.

This is based on a continuing discussion with Unknowns, who I think disagrees with what I'm going to present, or with its relevance to the "typical challenge."

Summary: If you learn that a preposterous hypothesis X i... (read more)

Are you a programmer yourself?

A prerequisite for an AI FOOMing is the ability to apply its intelligence to improving its source code so that the resulting program is more intelligent still.

We have an existence proof that human-level intelligence does not automatically give a mind the ability to understand source code and make changes to that source code which reliably have the intended effect. Perhaps some higher level of intelligence automatically grants that ability, but proving that would be non-trivial.

If your unpacking of "sufficiently smart"... (read more)

[-][anonymous]10y 3

Where should the line be drawn regarding the status of animals as moral objects/entities? E.G Do you think it is ethical to boil lobsters alive? It seems to me there is a full spectrum of possible answers: at one extreme only humans are valued, or only primates, only mammals, only veterbrates, or at the other extreme, any organism with even a rudimentary nervous system (or any computational, digital isomorphism thereof), could be seen as a moral object/entity.

Now this is not necessarily a binary distinction, if shrimp have intrinsic moral value it doe... (read more)

1GrateGoo10ySuppose sentient beings have intrinsic value in proportion to how intensely they can experience happiness and suffering. Then the value of invertebrates and many non-mammal vertebrates is hard to tell, while any mammal is likely to have almost as much intrinsic value as a human being, some possibly even more. But that's just the intrinsic value. Humans have a tremendously greater instrumental value than any non-human animal, since humans can create superintelligence that can, with time, save tremendous amounts of civilisations in other parts of the universe from suffering (yes, they are sparse, but with time our superintelligence will find more and more or them, in theory ultimately infinitely many). The instrumental value of most humans is enormously higher than the intrinsic value of the same persons - given that they do sufficiently good things.
1FAWS10yLess absurd than that some organism is infinitely more valuable than its sibling that differs in lacking a single mutation (in the case of the first organism of a particular species to have evolved "high" enough to have minimal moral value)?

Explain why I should consider this to be evidence that you are not User:Kevin.

(This is not rhetorical. It is something worth exploring. How does this instance of a non-human agent gain credibility? How can myself and such an agent build and maintain cooperation in the game of credible communication despite incentives to lie? Has Clippy himself done any of these things?)

3Clippy10yPerhaps you shouldn't. But there's a small chance that, if I were a human like User:Kevin, and other Users had made such inferences correctly identifying me, I would regard this time as the optimal one for revealing my true identity. Therefore, my post above is slightly informative.

Say a "catalytic pattern" is something like scaffolding, an entity that makes it easier to create (or otherwise obtain) another entity. An "autocatalytic pattern" is a sort of circular version of that, where the existence of an instance of the pattern acts as scaffolding for creating or otherwise obtaining another entity.

Autocatalysis is normally mentioned in the "origin of life" scientific field, but it also applies to cultural ratchets. An autocatalytic social structure will catalyze a few more instances of itself (frequentl... (read more)

Would people be interested in a place on LW for collecting book recommendations?

I'm reading The Logic of Failure and enjoying it quite a bit. I wasn't sure whether I'd heard of it here, and I found Great Books of Failure, an article which hadn't crossed my path before.

There's a recent thread about books for a gifted young tween which might or might not get found by someone looking for good books..... and so on.

Would it make more sense to have a top level article for book recommendations or put it in the wiki? Or both?

1[anonymous]10yConsidering most of my favorite books are the result of mentions in comment threads here, I'd say a book recommendation thread is in order. Tangental, but I remember "Logic of Failure" to be mostly being mental phenomena I was already familiar with, and generalizations from computer experiments that I didn't find particularly compelling. I'll have to give it another look.
1Morendil10yI'd say new top-level thread. The wiki can get a curated version of that.

I think I may have artificially induced an Ugh Field in myself.

A little over a week ago it occurred to me that perhaps I was thinking too much about X, and that this was distracting me from more important things. So I resolved to not think about X for the next week.

Of course, I could not stop X from crossing my mind, but as soon as I noticed it, I would sternly think to myself, "No. Shut up. Think about something else."

Now that the week's over, I don't even want to think about X any more. It just feels too weird.

And maybe that's a good thing.

3Cyan10yI have also artificially induced an Ugh Field in myself. A few months ago, I was having a horrible problem with websurfing procrastination. I started using Firefox for browsing and LeechBlock to limit (but not eliminate) my opportunities for websurfing instead of doing work. I'm on a Windows box, and for the first three days I disabled IE, but doing so caused knock-on effects, so I had to re-enable it. However, I knew that resorting to IE to surf would simply recreate my procrastination problem, so... I just didn't. Now, when the thought occurs to me to do so, it auto-squelches.
5Unknowns10yI predict with 95% confidence that within six months you will have recreated your procrastination problem with some other means.
5Cyan10yYour lack of confidence in me has raised my ire. I will prove you wrong!
3Unknowns10yTo be settled by February 8, 2011!
2Unknowns10yDid you start procrastinating again?
1Cyan10yYep. Eventually I sought medical treatment [] .

That isn't necessarily feasible. My department writes electronic design automation software, and we have a hard time putting in enough diagnostics in the right places to show to us when the code is taking a wrong turn without burying us in an unreadably huge volume of output. If an AI's deciding to lie is only visible as it's having a subgoal of putting an observer's mental model into a certain state, and the only way to notice that this is a lie is to notice that the intended mental state mismatches with the real world in a certain way, and this is sitting in a database of 10,000 other subgoals the AI has at the time - don't count on the scan finding it...

You should be careful not to conflate "preference" and "things that make oneself happy". Or make that a more clearly falsifiable component of your hypothesis.

Does anyone have any book recommendations for a gifted young teen? My nephew is 13, and he recently blew the lid off of a school-administered IQ test.

For his birthday, I want to give him some books that will inspire him to achieve great things and live a happy life full of hard work. At the very least, I want to give him some good math and science books. He has already has taken algebra, geometry and introductory calculus, so he knows some math already.

Books are not enough. Smart kids are lonely. Get him into a good school (or other community) where he won't be the smartest one. That happened to me at 11 when I was accepted into Russia's best math school and for the first time in my life I met other people worth talking to, people who actually thought before saying words. Suddenly, to regain my usual position of the smart kid, I had to actually work hard. It was very very important. I still go to school reunions every year, even though I finished it 12 years ago.

5Wei_Dai10yAlternatively, not having any equally smart kids to talk to will force him to read books and/or go online for interesting ideas and conversation. I don't think I had any really interesting real-life conversations until college, when I did an internship at Microsoft Research, and I'd like to think that I turned out fine. My favorite book, BTW, is A Fire Upon the Deep []. But one of the reasons I like it so much is that I was heavily into Usenet when I first read it, and I'm not sure that aspect of the book will resonate as much today. (I was determined to become a one-man Sandor Arbitration Intelligence. :)
2orthonormal10ySeconded. Whether he's exposed to a group of people who think ideas can be cool could be the biggest influence on him for the rest of his life.
1mattnewport10yThirded. My experience is that most schools can be very damaging for smart kids.
9Risto_Saarelma10yForum favorite Good and Real [] looks reasonably accessible to me, and covers a lot of ground. Also seconding Gödel, Escher Bach. The Mathematical Experience [] has essays about doing mathematics, written by actual mathematicians. It seems like very good reading for someone who might be considering studying math. The Road to Reality [] has Roger Penrose trying to explain all of modern physics and the required mathematics without pulling any punches and starting from grade school math in a single book. Will probably cause a brain meltdown at some point on anyone who doesn't already know the stuff, but just having a popular science style book that nevertheless goes on to explain the general theory of relativity without handwaving is pretty impressive. Doesn't include any of Penrose's less fortunate forays into cognitive science and AI. Darwin's Dangerous Idea [] by Daniel Dennett explains how evolution isn't just something that happens in biology, but how it turns up in all sorts of systems. Armchair Universe [] and old book about "computer recreations", probably most famous is the introduction of the Core War game. The other topics are similar, setting up an environment with a simple program that has elaborate emergent behavior coming out of it. Assumes the reader might actually program the recreations themselves, and provides appropriate detail. Surely You're Joking, Mr. Feynman [] is pretty much entertainment, but still very good. Feynman is still the requisite trickster-god patron saint of math and science. Code: The Hidden Language of Computer Hardware and Software [] explains how computers are put together, starting from really concrete first principles (flashing Morse code with flashlights, mechanical relay c
2orthonormal10yGood and Real is superb, but really too dry for a 13-year-old. I'd wait on that one. Surely You're Joking is also fantastic, but get it read and approved by your nephew's parents first; there's a few sexual stories with a hint of a PUA worldview.
1cata10yI loved "The Mathematical Experience" when I was 13-ish, and I re-read it recently; still good! I strongly second this recommendation.
1XiXiDu10yThanks, I just ordered 'Darwin's Dangerous Idea' and 'Code: The Hidden Language of Computer Hardware and Software'. I've already got the others. Here a tidbit from 'The Mathematical Experience [] '
1NancyLebovitz10yI've ordered a copy, but on a second look, I'm not sure that the argument is sound, or even interesting. Biological evolution runs on the local non-survival of the least fit (and sometimes the unlucky), not on an overview-based evaluation of the fittest.
7Kevin10yGodel Escher Bach!
4Soki10yknb, does your nephew know about lesswrong, rationality and the Singularity? I guess I would have enjoyed reading such a website when I was a teenager. When it comes to a physical book, Engines of Creation [] by Drexler can be a good way to introduce him to nanotechnology and what science can make happen. (I know that nanotech is far less important that FAI, but I think it is more "visual" : you can imagine those nanobots manufacturing stuff or curing diseases, while you cannot imagine a hard takeoff). Teenagers need dream.
2knb10yMy sister and brother-in-law are both semi-religious theists, so I'm a bit reluctant to introduce him to anything as hardcore-atheist as Less Wrong, at least right now. Going through that huge theist-to-atheist identity transition can be really traumatic. I think it would be better if he was a bit older before he had confront those ideas. I was 16 before I really allowed myself to accept that I didn't believe in God, and that was still a major crisis for me. If he starts getting into hardcore rationality material this early, I'm afraid it could force a choice between rationality and wishful thinking that he may not be ready to make.
2Interpolate10yIf he is gifted and interested in science, introducing him to lesswrong, rationality and the Singularity could have a substantial positive impact on his academic development. What would be the worst that could happen?
5knb10yMy concern is not just that it would be traumatic, but that it will be so traumatic that he'll rationalize himself into a "belief in belief []" situation. I had my crisis of faith when I was close to his age (14) and I wasn't ready to accept something that would alienate me from my family yet, so I simply told myself that I believed, and tried not to think about the issue. (I suspect this is why most people don't come out as atheists until after they've established separate identities from their parents and families. A lot of people never escape from these traps. I think waiting somewhat--until he's somewhat older and more mature--will make him more likely to come to the right conclusions in the end.
1wedrifid10yHow is the above wrong enough to be at -2? I nearly universally reject any assertions that people have a duty to interfere with others but even so I don't have a problem with the above.
2MartinB10yThe Heinlein Juveniles. 'have space suit will travel' and others have the whole self-reliance, work hard and achieve things strongly ingrained. I cannot judge how well the integrate with your current culture, but in the 50s they sold well, and still do. But those are not specific for über-bright kids, more for the normal bright types. If he hasn't done so yet, just introducing him to the next big library might help a lot.
1MartinB10yAnother all-purpose book: Bill Brysons: short history of almost everything. It is not aim at kids, but very accessible, well written and deal with lots of the history of sciences, including the ignoring of great achievements, misleading pathways and such. A great overview.
2RobinZ10yMy dad's been trying to get me to read the Feynman Lectures for ages - the man's a good writer if your nephew would be interested by physics.
1[anonymous]10yGet him a book of math contests. The Mandelbrot Problem Book [] is an excellent one.

Does anyone know where the page that used to live here can be found?

It was an experiment where two economists were asked to play 100 turn asymmetric prisoners dilemma with communication on each turn to the experimenters, but not each other.

It was quite amusing in that even though they were both economists and should have known better, the guy on the 'disadvantaged' side was attempting to have the other guy let him defect once in a while to make it "fair".

2Douglas_Knight10ygoo [] gle [] archive [] .org [*/]
2gwern10yBEHOLD! [*/]

I don't know what "bounded-rational" means

It is the difference between what you describe in the first paragraph and what an actually possible agent is limited to. The kind of thing that allows a superintelligence that has no flaws to still be wrong about something like Goldbach's conjecture. Always thinking perfectly but not able to think infinitely fast and about everything.

and I don't know what "instrumentally rational" means.

Achieving your goals in the best possible way. In contrast to 'epistemic rationality' is having the best beliefs possible. The concepts are related but distinct.

"CIA Software Developer Goes Open Source, Instead":

"Burton, for example, spent years on what should’ve been a straightforward project. Some CIA analysts work with a tool, “Analysis of Competing Hypotheses,” to tease out what evidence supports (or, mostly, disproves) their theories. But the Java-based software is single-user — so there’s no ability to share theories, or add in dissenting views. Burton, working on behalf of a Washington-area consulting firm with deep ties to the CIA, helped build on spec a collaborative version of ACH. He tr

... (read more)
3Rain10yFar more interesting than the software is the chapter [] in the CIA book Psychology of Intelligence Analysis where they describe the method: Summary and conclusions:

I don't think it's gotten that vacuous, at least as SIAI uses it. (They tend to use it pretty narrowly to refer to the intelligence explosion point, at least the people there whom I've talked to. The Summit is a bit broader, but I suppose that's to be expected, what with Kurzweil's involvement and the need to fill two days with semi-technical and non-technical discussion of intelligence-related technology, science, and philosophy.) You say that it can be replaced with "the future" without any loss, but your example doesn't really bear that out. I... (read more)

What's the policy on User pages in the wiki? Can I write my own for the sake of people having a reference when they reply to my posts, or are they only for somewhat accomplished contributers?

3Blueberry10yI can't imagine any reason why it would be a problem to make a User page. Go ahead.
2WrongBot10yI haven't seen any sort of policy articulated. I just sort of went for it, and haven't gotten any complaints yet. Personally, I'd love to see more people with wiki user pages, since the LW site itself doesn't have much in the way of profile features.

I made some comments on the recently-deleted threads that got orphaned when the whole topic was banned and the associated posts were taken down. Currently no-one can reply to the comments. They don't related directly to the banned subject matter - and some of my messages survive despite the context being lost.

Some of the comments were SIAI-critical - and it didn't seem quite right to me at the time for the moderator to crush any discussion about them. So, I am reposting some of them as children of this comment in an attempt to rectify things - so I can refer back to them, and so others can comment - if they feel so inclined:

6timtyler10y[In the context of SIAI folks thinking an unpleasant AI was likely] The SIAI derives its funding from convincing people that the end is probably nigh - and that they are working on a potential solution. This is not the type of organisation you should trust to be objective on such an issue - they have obvious vested interests.
2Johnicholas10yI've noticed this structural vulnerability to bias too - Can you think of any structural changes that might reduce or eliminate this bias? Maybe SIAI ought to be offering a prize for substantially justified criticism of some important positional documents, as judged by some disinterested agent?
3timtyler10yThey are already getting some critical feedback. I think I made much the same points in my DOOM! video. DOOM mongers: * tend to do things like write books about THE END OF THE WORLD - which gives them a stake in promoting the topic ...and... * are a self-selected sample of those who think DOOM is very important (and so, often, highly likely) - so naturally they hold extreme views - and represent a sample from the far end of the spectrum; * clump together, cite each others papers, and enjoy a sense of community based around their unusual views. It seems tricky for the SIAI to avoid the criticism that they have a stake in promoting the idea of DOOM - while they are funded the way they are. Similarly, I don't see an easy way of avoiding the criticism that they are a self-selected sample from the extreme end of a spectrum of DOOM beliefs either. If we could independently establish p(DOOM), that would help - but measuring it seems pretty challenging. IMO, a prize wouldn't help much - but I don't know for sure. Many people behave irrationally around prizes - so it is hard to be very confident here. I gather they are working on publishing some positional documents. It seems to be a not-unreasonable move. If there is something concrete to criticise, critics will have something to get their teeth into.
3timtyler10yThey used to have a "commitment" that: "Technology developed by SIAI will not be used to harm human life." [] ...on their web site. I probably missed the memo about that being taken down.
2timtyler10y[In the context of SIAI folks thinking an unpleasant AI was likely] Re: "The justification is that uFAI is a lot easier to make." That seems like naive reasoning. It is a lot easier to make a random mess of ASCII that crashes or loops - and yet software companies still manage to ship working products.
3WrongBot10yThose software companies test their products for crashes and loops. There is a word for testing an AI of unknown Friendliness and that word is "suicide".
4timtyler10yThat just seems to be another confusion to me :-( The argument - to the extent that I can make sense of it - is that you can't restrain an super-intelligent machine - since it will simply use its superior brainpower to escape from the constraints. We successfully restrain intelligent agents all the time - in prisons. The prisoners may be smarter than the guards, and they often outnumber them - and yet still the restraints are usually successuful. Some of the key observations to my mind are: * You can often restrain one agent with many stupider agents; * The restraining agents do not need to be humans - they can be other machines; * You can often restrain one agent with a totally dumb cage; * Complex systems can often be tested in small pieces (unit testing); * Large systems can often be tested on a smaller scale before deployment; * Systems can often be tested in virtual environments, reducing the cost of failure. Discarding the standard testing-based methodology would be very silly, IMO. Indeed, it would sabotage your project to the point that it would almost inevitably be beaten - and there is very little point in aiming to lose.
1WrongBot10yAre you familiar with the AI-Box experiment []? We can restrain human-intelligence level agents in prisons, most of the time. But the question to ask is: how effective was the first prison? Because that's the equivalent case. None of the safety measures you propose are safe enough. You're underestimating the power of a recursively self-improving AI by a factor I can't begin to estimate--which is kind of the point.
4Vladimir_Nesov10yA much stronger argument than all-powerful AIs suddenly escaping (which is still not without merit) is that AI will have an incentive to behave as we expect it to behave, until at some point we no longer control it. It'll try its best to pass all tests.
2timtyler10ySo: while it believes it is under evaluation it does its very best to behave itself? Can we wire that belief in as a prior with p=1.0?
2WrongBot10yI suppose I was mentally classifying that kind of behavior as an escape; you're right that it should be called out as a separate point of failure.
6Vladimir_Nesov10yMy point is that "ai box experiment" communicates orders of magnitude less evidence about the danger of escaping AIs than people like to imply, and there are lots of stronger and simpler self-contained arguments such as the one I gave. (The overall danger is much greater than even that, because these are specific plots with an obvious villain, while reality is more subtle [] .)
1WrongBot10yAhhh, I see what you're getting at. Agreed.
3timtyler10yIt won't be the first prison - or anything like it. If we have powerful intelligence that needs testing, then we can have powerful guards too. The AI-Box experiment has human guards. Consequently, it has very low relevance to the actual problem. Programmers don't build their test harnesses out of human beings. Safety is usually an economic trade off. You can usually have an lot of it - if you are prepared to pay for it.
2JGWeissman10ySoftware companies manage to ship products that do sort of what they want, that they can patch to more closely do what they want. This is generally after rounds of internal testing, in which they try to figure out if it does what they want by running it and observing the result. But an AGI, whether FAI or uFAI, will be the last program that humans get to write and execute unsupervised. We will not get to issue patches.

But an AGI, whether FAI or uFAI, will be the last program that humans get to write and execute unsupervised. We will not get to issue patches.

Or to put it another way, the revolution will not be beta tested.

1sketerpot10yThat is one of the most chilling phrases I've ever heard. Disarming in its simplicity, yet downright Lovecraftian in its implications. And it would probably make a nice bumper sticker.
2NancyLebovitz10yRevolutions never get beta tested.
2rwallace10yIn fiction, yes. Fictional technology appears overnight, works the first time without requiring continuing human effort for debugging and maintenance, and can do all sorts of wondrous things. In real life, the picture is very different. Real life technology has a small fraction of the capabilities of its fictional counterpart, and is developed incrementally, decade by painfully slow decade. If intelligent machines ever actually come into existence, not only will there be plenty of time to issue patches, but patching will be precisely the process by which they are developed in the first place.
3JoshuaZ10yI agree somewhat with this as a set of conclusions, but your argument deserves to get downvoted because you've made statements that are highly controversial. The primary issue is that, if one thinks that an AI can engage in recursive self-improvement and can do so quickly, then once there's an AI that's at all capable of such improvement, the AI will rapidly move outside our control. There are arguments against such a possibility being likely, but this is not a trivial matter. Moreover, comparing the situation to fiction is unhelpful- just because something is common in fiction that's not an argument that such a situation can't actually happen in practice. Reversed stupidity is not intelligence.
2NihilCredo10yDid you accidentally pick the wrong adjective, or did you seriously mean that controversy is unwelcome in LW comment threads?
4ata10yI read the subtext as "'ve made statements that are highly controversial without attempting to support them". Suggesting that there will be plenty of time to debug, maintain, and manually improve anything that actually fits the definition of "AGI" is a very significant disagreement with some fairly standard LW conclusions, and it may certainly be stated, but not as a casual assumption or a fact; it should be accompanied by an accordingly serious attempt to justify it.
1JoshuaZ10yNo. See ata's reply which summarizes exactly what I meant.
1rwallace10yTo be sure, the fact that something is commonplace in fiction doesn't prove it false. What it does show is that we should distrust our intuition on it, because it's clearly an idea to which we are positively disposed regardless of its truth value -- in the Bayesian sense, that is evidence against it. The stronger argument against something is of course its consistent failure to occur in real life. The entire history of technological development says that technology in the real world does not work the way it would need to for the 'AI go foom' scenario. If 100% evidence against and 0% evidence for a proposition should not be enough to get us to disbelieve it, then what should? Not to mention that when you look at the structure of the notion of recursive self-improvement, it doesn't even make sense. A machine is not going to be able to completely replace human programmers until it is smarter than even the smartest humans in every relevant sense, which given the differences in architecture, is an extraordinarily stringent criterion, and one far beyond anything unaided humans could ever possibly build. If such an event ever comes about in the very distant future, it will necessarily follow a long path of development in which AI is used to create generation after generation of improved tools in an extended bootstrapping process that has yet to even get started. And indeed this is not a trivial matter -- if people start basing decisions on the 'AI go foom' belief, that's exactly the kind of thing that could snuff out whatever chance of survival and success we might have had.
1timtyler10yMost programmers are supervised. So, this claim is hard to parse. Machine intelligence has been under development for decades - and there have been plenty of patches so far. One way of thinking about the process is in terms of increasing the "level" of programming languages. Computers already write most machine code today. Eventually humans will be able to tell machines what they want in ordinary English - and then a "patch" will just be some new instructions.
3JGWeissman10yBy other humans. If we program an AGI, then it will supervise all future programming. Machine intelligence does not yet approach human intelligence. We are talking about applying patches on a superintelligence. The difficulty is not in specifying the patch, but in applying to a powerful superintelligence that does not want it.
2Morendil10yStill, a lot of these "working products" are the output of a filtering process which starts from a random mess of ASCII that crashes or loops, and tweaks it until it's less obviously broken. (Most of the job of testing being, typically, left to the end user.)
1timtyler10ySure. The point is that - to conclude that a target will be missed - it is not sufficient to observe how small it is. Programmers rountinely hit miniscule targets in search spaces. To make the case, you would also need to argue that those aiming at the target are not good marksmen.

Are there any posts people would like to see reposted? For example, Where Are We seems like it maybe should be redone, or at least put a link in About... Or so I thought, but I just checked About and the page for introductions wasn't linked, either. Huh.

3thomblake10yIt would be nice if we had profile pages with machine-readable information and an interface for simple queries so posts such as that one would be redundant.

I came across a blurb on Ars Technica about "quantum memory" with the headline proclaiming that it may "topple Heisenberg's uncertainty principle". Here's the link:

They didn't source the specific article, but it seems to be this one, published in Nature Physics. Here's that link:

This is all well above my pay... (read more)

Perhaps a clearer example may be found in sports coaches.

The main reason that coaches are important (not just in sports) is because of blind spots - i.e., things that are outside of a person's direct perceptual awareness.

Think of the Dunning-Kreuger effect: if you can't perceive it, you can't improve it.

(This is also why publications have editors; if a writer could perceive the errors in their work, they could fix them themselves.)

It might be useful to have a short list of English words that indicate logical relationships or concepts often used in debates and arguments, so as to enable people who are arguing about controversial topics to speak more precisely.

Has anyone encountered such a list? Does anyone know of previous attempts to create such lists?

[-][anonymous]10y 2

Followup to: Making Beliefs Pay Rent in Anticipated Experiences

In the comments section of Making Beliefs Pay Rent, Eliezer wrote:

I follow a correspondence theory of truth. I am also a Bayesian and a believer in Occam's Razor. If a belief has no empirical consequences then it could receive no Bayesian confirmation and could not rise to my subjective attention. In principle there are many true beliefs for which I have no evidence, but in practice I can never know what these true beliefs are, or even focus on them enough to think them explicitly, because th

... (read more)
1PaulAlmond10ySurely, this is dealt with by considering the amount of information in the hypothesis? If we consider each hypothesis that can be represented with 1,000 bits of information, there will only be a maximum of 2^1,000 such hypotheses, and if we consider each hypothesis that can be represented with n bits of information, there will only be a maximum of 2^n - and that is before we even start eliminating hypotheses that are inconsistent with what we already know. If we favor hypotheses with less information content, then we end up with a small number of hypotheses that can be taken reasonably seriously, and the remainder being unlikely - and progressively more unlikely as n increases, so that when n is sufficiently large, we can, practically, dismiss any hypotheses.
1[anonymous]10yI agree with most of that, but why favor less information content? Though I may not fully understand the math, this recent post by cousin it [] seems to be saying that priors should not always depend on Kolmogorov complexity. And, even if we do decide to favor less information content, how much emphasis should we place on it?
1PaulAlmond10yIn general, I would think that the more information is in a theory, the more specific it is, and the more specific it is, the smaller is the proportion of possible worlds which happen to comply with it. Regarding how much emphasis we should place on it: I woud say "a lot" but there are complications. Theories aren't used in isolation, but tend to provide a kind of informally put together world view, and then there is the issue of degree of matching.
4Perplexed10yWhich theory has more information? * All crows are black * All crows are black except
1PaulAlmond10yI didn't say you ignored previous correspondence with reality, though.
1[anonymous]10yThat isn't Perplexed's point. Let's say that as of this moment all crows that have been observed are black, so both of his hypotheses fit the data. Why should "all crows are black" be assigned a higher prior than "All crows are black except "? Based on cousin_it's post, I don't see any reason to do that.
0[anonymous]10ySo, to revive this discussion: if we must distribute probability mass evenly because we cannot place emphasis on simplicity, shouldn't our priors be almost zero for every hypothesis? It seems to me that the "underdetermination" problem makes it very hard to use priors in a meaningful way.
0PaulAlmond10yI am assuming here that all the crows that we have previously seen have been black, and therefore that both theories have the same agreement, or at least approximate agreement, with what we know. The second theory clearly has more information content. Why would it not make sense to use the first theory on this basis? The fact that all the crows we have seen so far are black makes it a good idea to assume black crows in future. There may be instances of non-black crows, when the theory has predicted black crows, but that simply means that the theory is not 100% accurate. If the 270 pages of exceptions have not come from anywhere, then the fact that they are not justified just makes them random, unjustified specificity. Out of all the possible worlds we can imagine that are consistent with what we know, the proportion that agree with this specificity is going to be small. If most crows are black, as I am assuming our experience has suggested, then when this second theory predicts a non-black crow, as one of its exceptions, it will probably be wrong: The unjustified specificity is therefore contributing to a failure of the theory. On the other hand, when the occasional non-black crow does show up, there is no reason to think that the second theory is going to be much better at predicting this than the first theory - so the second theory would seem to have all the inaccuracies of wrongful black crow prediction of the first theory, along with extra errors of wrongful non-black crow prediction introduced by the unjustified specificity. Now, if you want to say that we don't have experience of mainly black crows, or that the 270 pages of exceptions come from somewhere, then that puts us into a different scenario: a more complicated one. Looking at it in a simple way, however, I think this example actually just demonstrates that information in a theory should be minimized.
3Perplexed10yI haven't been following the discussion on this topic very closely, so my response may be about stuff you already know or already know is wrong. But, since I'm feeling reckless today, I will try to say something interesting. There are two different information metrics we can use regarding theories. The first deals with how informative a theory is about the world. The ideally informative theory tells us a lot about the world. Or, to say the same thing in different language, an informative theory rules out as many "possible worlds" as it can; it tells us that our own world is very special among all otherwise possible worlds; that the set of worlds consistent with the theory is a small set. We may as well call this kind of information Shannon information or S-information . A Karl Popper fan would approve of making a theory as S-informative as possible, because then it is exposing itself to the greatest risk of refutation. The second information metric measures how much information is required to communicate the theory to someone. My 270 pages of fine print in the second crow theory might be an example of a theory with a lot of this kind of information. Let us call this kind of information Kolmogorov information, or K-information. My understanding of Occam's razor is that it recommends that our theories should use as little K-information as possible. So we have Occam telling us to minimize the K-information and Popper telling us to maximize the S-information. Luckily, the two types of information are not closely related, so (assuming that the universe does not conspire against us) we can frequently do reasonably well by both criteria. So much for the obvious and easy points. The trouble appears, especially for biologists and other "squishy" scientists, when Nature seems to have set things up so that every law has some exceptions. I'll leave it to you to Google on either "white crow" or "white raven" and to admire those fine and intelligent birds. So, given our objec
1RichardKennaway10yWe change it to say, "99+% of crows have such-and-such alleles of genes for determining feather colour; certain other alleles are rare and result in a bird lacking feather pigments due to the synthesis pathway being broken at such-and-such a step for lack of such-and-such a protein. The mutation is disadvantageous, hence the absence of any substantial population of white crows." (Or whatever the actual story is, I'm just making that one up.) If we don't know the actual story, then the best we can do is say that for reasons we don't know, it happens now and then that black crows can give birth to a white offspring. Squishiness is not a property of biological phenomena, but of our knowledge of those phenomena. Exceptions are in our descriptions, not in Nature.
0NancyLebovitz10yI wonder if it helps to arrange K-information in layers. You could start with "Almost all crows are black", and then add footnotes for how rare white crows actually are, what causes them, how complete we think our information about crow color distribution is and why, and possibly some factors I haven't thought of.
1Perplexed10yLayering or modularizing the hypothesis: Of course, you can do this, and you typically do do this. But, layering doesn't typically change the total quantity of K-information. A complex hypothesis still has a lot of K-information whether you present it as neatly layered or just jumbled together. Which brings us to the issue of just why we bother calculating the K-information content of a hypothesis in the first place. There is a notion, mentioned in Jaynes and also in another thread active right now, that the K-information content of a hypothesis is directly related to the prior probability that ought to be attached to a hypothesis (in the absence of (or prior to) empirical evidence). So, it seems to me that the interesting thing about your layering suggestion is how the layering should tie in to the Bayesian inference machinery which we use to evaluate theories. For example, suppose we have a hypothesis which, based on evidence so far, has a subjective "probability of correctness" of, say 0.5. Then we get a new bit of evidence. We observe a white (albino) crow, for example. Doing standard Bayesian updating, the probability of our hypothesis drops to 0.001, say. So we decide to try to resurrect our hypothesis by adding another layer. Trouble is, that we have just increased the K-complexity of the hypothesis, and that ought to hurt us in our original "no-data" prior. Trouble is, we already have data. Lots of it. So is there some algebraic trick which lets us add that new layer to the hypothesis without going back to evidential square one?
1NancyLebovitz10yK-information is about communicating to "someone"-- do you compute the amount of K-information for the most receptive person you're communicating with, or do you have a different amount for each layer of detail? Actually, you might have a tree structure, not just layers-- the prevalence of white crows in time and space is a different branch than the explanation of how crows can be white.
0Perplexed10yA very interesting question. Especially when you consider the analogy with canon:Kolmogorov. Here we have an ambiguity as to what person we communicate to. There, the ambiguity was regarding exactly what model of universal Turing machine we were programming. And there, there was a theorem to the effect that the differences among Turing machines aren't all that big. Do we have a similar theorem here, for the differences among people - seen as universal programmable epistemic engines.
0Peter_de_Blanc10yBayesian updating is timeless. It doesn't care whether you observed the data before or after you wrote the hypothesis.
1Perplexed10ySo, it sounds like you are suggesting that we can back out all that data, change our hypothesis and prior, and then read the data back in. In theory, yes. But sometimes we don't even remember the data that brought us to where we are now. Hence the desirability of a trick. Is there an updating-with-new-hypothesis rule to match Bayes's updating-with-new-evidence rule?

I generally don't look for world news any more; when I did I'd generally follow Current Events on Wikipedia:

  • It's short,

  • It coveres world events more evenly than French or American national media, and

  • It tries to be unbiased

When I want more in-depth coverage on a subject, I generally turn to wikipedia too. I think it has a better trade-off between providing context and covering breaking events as they unfold (something that I consider is overvalued).

1h-H10ythanks, that's actually what I wanted to know.

But there are no non-partisan sites, are there?

Partisanship is something that has degrees. It isn't an on/off thing. Moreover, there's a big difference between sources which try to be non-partisan and sources which are avowedly partisan. Using any single source, even a source claiming to be non-partisan, for news is a bad idea. Using a single avowedly partisan source is a really bad idea.

but that aside, I meant political world news, not scientific.

These aren't disconnected at all. Science matters for many political issues. Look at stem cell researc... (read more)

0h-H10ya belated reply: now, as a generality your first statement is correct, but after searching for some years I've concluded the easiest method is in fact mild support for a partisan anti war website, reason being; on average wars are more destructive than no-wars, and definitely inductive of irrationality. a note about the particular partisan site, it's not a single source by any means-I believe this is the cause of contention?- it's actually an aggregation of 'anti war' news from multiple sources including mainstream channels and others. as such the 'single source' label is negated. second statement: yes, science and politics are connected, but I believe this misses the point, in the domain of national policy, the more hawkish elements have been in control for quite a while now, pitting the US against third world destitute farmers and shepherds. that's not exactly a rational path, and so, as much as we should support stem cell research for eg. that wasn't the angle I was coming from. third; answered, see above. fourth; it does as an issue to be concerned with, but surely not in context of the discussion? strategic theorizing on possible crypto use by say an afghan warlord concerned for his poppy production is quite a marginal concern compared to the US government launching wars that cost trillions and benefit humanity little to nothing while increasing likelihood of retaliation etc.
1JoshuaZ10yWhat do you mean by support? In the context of your earlier remarks, "support" seems to mean "use as sole newsource." I don't see how even if one accepted your premises one would get that as a conclusion. This makes no difference. For purposes of getting relevant data and avoiding mind-killing, a partisan aggregator will be functionally identical to a partisan single source. In regards to the connection between science and politics, I'm not sure I can parse what you are saying and in so far as I can parse it, it seems like you have a problematic attitude. Not everything is about simple ideological support or not, and your response above seems to almost be an indication of Mindkilling spreading from politics to science. This is precisely why I gave the example of ITER and whether or not it should be funded and if so by how much. Science impacts policy. And it isn't anything as simple as "oh, we should support this but not support that. Stem cells, yay! People who don't like stem cells cells, booh!" To use your example of stem cells, how much resources should go into stem cell research is quite complicated. The standard reaction against theistic arguments against embryonic stem cell research is to conclude that we should have massive amounts of research into stem cells. But that's not necessarily the case. We have a limited amount of resources that is going to go to biological and medical research. How much of that should go to stem cells? That should be the question that you should ask and not come away with some general notion of "support." You are missing the point. The change that crypto brings (and for that matter is actually bringing) is the benefits it brings to the little guy, the decentralized individuals, not the warlord. The person leaking documents or the resistance fighter/terrorist/guerrilla/etc are the types who benefit from having strong crypto. This is why for a long time the US classified cryptography as munitions for export purposes. And saying t

My impression is that sometimes we do need to deal with them in order to make the math come out right, even though the only thing we are really concerned about is our observed universe. Just as we sometimes need to deal with negative numbers of sheep - however difficult we may find this to visualize if we work as a shepherd.

0h-H10ytrue, but there are no 'negative sheep', only numbers arbitrarily representing them.
1Perplexed10ybut we shouldn't concern ourselves with numbers if they aren't part of our observed universe.
1h-H10ynumbers are quite useful, so we don't/shouldn't do away with them, but the math is never a complete substitute for the observable universe. writing down '20 sheep' doesn't physically equal 20 sheep, rather it's a method we use for simplicity. as it stands, no two sheep are alike to every last detail as far as anyone can tell, yet we still have a category called 'sheep'. this is so given the observed recurrence of 'sheep' like entities, similar enough for us to categorize them for practicality's sake, but that doesn't mean they're physically all alike to every detail. it could be argued that sometimes the math does equate with reality, as in 'Oxygen atom' is a category consisting of entirely similar things, but even that is not confirmed, simply an assertion; no human has observed all 'Oxygen atoms' in existence to be similar in every detail, or even in some arbitrarily 'essential' detail/s. yet it is enough for the purposes of science to consider them all similar, and so we go with it,otherwise we'd never have coherent thought let alone science. it might very well be that all Oxygen atoms in existence are physically the same in some ways, but we have no way of actually knowing. this doesn't mean that there are 'individual atoms', but it doesn't negate it either. ETA: as pengvado said in below post, replace 'atom' with 'particle'.
3pengvado10yNo Individual [] Particles []. The fact that measurements of their mass/charge/etc have always come out the same, is not the only evidence we have for all particles of a given type being identical. (A whole oxygen atom is a bad example, though. Atoms have degrees of freedom beyond the types of particles they're made of.)
0h-H10yyes, I had that specific post in mind when I presented the atom example. you're correct here though, I should have said particles,I shouldn't write so late after midnight I guess.. now I admit that my understanding of quantum mechanics is not that much above a lay persons', so maybe I just need to apply myslef more and It'll click, but let's consider my arguement first:- here's what EY said in reply to a post in that thread-emphasis mine: "There can be properties of the particles we don't know about yet, but our existing experiments already show those new properties are also identical, unless the observed universe is a lie." and then: "Undiscovering this would be like undiscovering that atoms were made out of nucleons and electrons. It's in this sense that I say that the observed universe would have to be a lie." here I believe he's making a mistake/displaying a bias; the math-of Quantum Mechanics in this particular instance- does not determine physical reality, rather it describes it to some degree or other. to suggest that the mathematics of quantum mechanics is the end of the road is too strong a claim IMO.
0pengvado10yI don't have any arguments that weren't discussed in that post; so far as I can tell, it already adequately addressed your objection: QM doesn't have to be the end of the road. If QM is a good approximation of reality on the scales it claims to predict in the situations we have already tested it in -- if the math of QM does describe reality to some degree or other -- then that's enough for the quantum tests of particle identity to work exactly.
0h-H10yto put it mildly I don't believe anyone can address that objection satisfactorily, as wedrifid put it eloquently, the math is part of the map, not territory. agreed, that was partially my point a couple of posts ago. for practical reasons it's good enough that the math works to a degree.
1Perplexed10yUhmm. I hate to explain my own jokes, but ... You did notice the formal similarity between my "we shouldn't concern ourselves" comment and its great grandparent, right?
0h-H10yI noticed, but there was a clear difference that I felt was necessary to point out regardless.
0wedrifid10yTrue (only) in the sense that our numbers are part of our map and not the territory. In the same sense we have no way of actually knowing there are patterns in the universe appropriately named Oxygen. Or Frog.
0h-H10ygood point about the map/territory distinction, that was what I intended to say but couldn't put into so few words, thanks :) and no, it seems that not even Frog can escape this, I'm not sure about it's significance here though?

I read that site and I like it. But it's important to get a variety of views. I would never want to get most of my world news from a single site. In fact, if I had to pick a single source of world news for some reason, then I'd want to pick one that I disagree with, as long as it still has good coverage of what I care about.

Here's a thought experiment that's been confusing me for a long time, and I have no idea whether it is even possible to resolve the issues it raises. It assumes that a reality which was entirely simulated on a computer is indistinguishable from the "real" one, at least until some external force alters it. So... the question is, assuming that such a program exists, what happens to the simulated universe when it is executed?

In accordance with the arguments that Pavirta gives below me, redundant computation is not the same as additional computation.... (read more)

2Emile10yThat's pretty much Tegmark's Multiverse [], which seems pretty popular around here (I think it makes a lot of sense).
1ata10yIndeed. I have a post making similar arguments [], though I still haven't been able to resolve the ethical and anthropic problems it raises in any satisfactory way. At this point I've backtracked from the confidence I held when I wrote that post; what I'm still willing to say is that we're probably on the right track thinking of "Why does anything exist?" as a wrong question [] and thinking of reality as indexical (i.e. the true referent of the category "real" is the set of things instantiated by this universe; it is a category error to talk about other universes being real or not real), but the Mathematical Universe Hypothesis still leaves much to be confused about.
4Perplexed10yMy own view is that (ignoring simulations for the time being) MWI ideas have no conflict with our usual ethical intuitions and reasonings. Yes, it is the case that when I choose between evil action A and good action B, there will be two branches of the universe - one in which I choose A and one in which I choose B. This will be the case regardless of which choice I make. But this does not make my choice morally insignificant, because I split too, along with the rest of the universe. The version of me that chose evil act A will have to live thereafter with the consequences of that choice. And the version of me that chose B must live with quite different consequences. What, more than that, could a believer in the moral significance of actions want of his universe? The situation with respect to simulations is a bit trickier. Suppose I am deciding whether to (A) pull the plug on a simulation which contains millions of sentient (simulated) beings, or (B) allow the simulation to continue. So, I choose, and the universe branches. If I chose A, I must live with the consequences. I don't have that simulation to kick around any more. But, if I were to worry about all the simulated lives that I have so ruthlessly terminated, I can easily reassure myself that I have only terminated a redundant copy of those lives. The (now) master copy of the simulation plays on, over in that parallel universe where I chose B. Is it wrong to create a simulation and then torture the inhabitants? Well, that is an ethical question, whereas this is a meta-ethical analysis. But the meta-ethical answer to that ethical question is that if you torture simulated beings, then you must live with the consequences of that.
1[anonymous]10yYes, MWI ideas have no conflict with usual ethical intuitions. And they also help you make better sense of those intuitions. Counterfactuals really do exist, for example; they're not just some hypothetical that is in point of fact physically impossible.
1ata10yThat's not how MWI works, unless human brains have a quantum randomness source that they use to make decisions (which does not appear to be the case).
2Perplexed10yI'm not sure it matters to the analysis. Whether we have a Tegmark multiverse, or Everett MWI with some decisions depending on quantum randomness and others classically determined, or whether the multiple worlds are purely subjective fictions created to have a model of Bayesianism; regardless of what you think is a possible reduction of "possibly"; it is still the case that you have to live in the reality which you helped to create by way of your past actions.
1Perplexed10yI should add that it is impossible to erase your sin by deciding to terminate the simulation, so as to "euthanize" the victims of your torture. Because there is always a branch where you don't so decide, and the victims of your torture live on.

Eliezer has written a post (ages ago) which discussed a bias when it comes to contributions to charities. Fragments that I can recall include considering the motivation for participating in altruistic efforts in a tribal situation, where having your opinion taking seriously is half the point of participation. This is in contrast to donating 'just because you want thing X to happen'. There is a preference to 'start your own effort, do it yourself' even when that would be less efficient than donating to an existing charity.

I am unable to find the post in question - I think it is distinct from 'the unit of caring'. It would be much appreciated if someone who knows the right keywords could throw me a link!

4WrongBot10yYour Price for Joining []?

Alright, I've lost track of the bookmark and my google-fu is not strong enough with the few bits and pieces I remember. I remember seeing a link to a story in a lesswrong article. The story was about a group of scientists who figured out how to scan a brain, so they did it to one of them, and then he wakes up in a strange place and then has a series of experiences/dreams which recount history leading up to where he currently is, including a civilization of uploads, and he's currently living with the last humans around... something like that. Can anybody help me out? Online story, 20 something chapters I think... this is driving me nuts.

2Risto_Saarelma10yAfter Life []

The visual guide to a PhD:

Nice map–territory perspective.

I've programmed a robot to behave in the way you describe, treating bright lights as painful stimuli. Was testing it immoral?

1Tiiba10yThat's why I said it's hairier with machines. Um, actual pain or just disutility?

There isn't a very meaningful distinction between animals and machines. What does or doesn't count as a "simple stimulus response"? Or learning?

John Baez This Week's Finds in Mathematical Physics has its 300th and last entry. He is moving to wordpress and Azimuth. He states he wants to concentrate on futures, and has upcoming interviews with:

Tim Palmer on climate modeling and predictability, Thomas Fischbacher on sustainability and permaculture, and Eliezer Yudkowsky on artificial intelligence and the art of rationality. A Google search returns no matches for Fischbacher + and no hits for Palmer +.

That link to Fischbacher that Baez gives has a presentation on cognitive distortio... (read more)

Thanks for doing this. It's always nice to have external motivation to do something good.

I've written a post for consolidating book recommendations, and the links don't have hidden urls. These are links which were cut and pasted from a comment-- the formatting worked there.

Posting (including to my drafts) mysteriously doubles the spaces between the words in one of my link texts, but not the others. I tried taking that link out in case it was making the whole thing weird, but it didn't help.

I've tried using the pop-up menu for links that's available for writing posts, but that didn't change the results.

What might be wrong with the formatting?

1Alicorn10yI don't know what's wrong, but a peek at the raw HTML editor (there's a button for it in the toolbar) might give a hint.

With regard to the recent proof of P!=NP:

1Paul Crowley10yWith no time limit, how can you ever win that one?
1gwern10yNo time limit?

Scenario: A life insurance salesman, who happens to be a trusted friend of a relatively-new-but-so-far-trustworthy friend of yours, is trying to sell you a life insurance policy. He makes the surprising claim that after 20 years of selling life insurance, none of his clients have died. He seems to want you to think that buying a life insurance policy from him will somehow make you less likely to die.

How do you respond?

edit: to make this question more interesting: you also really don't want to offend any of the people involved.

8wedrifid10yWow. He admitted that to you? That seems to be strong evidence that most people refuse to buy life insurance from him. In a whole 20 years he hasn't sold enough insurance that even one client has died from unavoidable misfortune!
8Eliezer Yudkowsky10y"No." Life insurance salesmen are used to hearing that. If they act offended, it's a sales act. If you're reluctant to say it, you're easily pressured and you're taking advantage. You say "No". If they press you, you say, "Please don't press me further." That's all.
4SilasBarta10ySince his sales rate probably increased with time, that means the average time after selling a policy is ~8 years. So the typical client of his didn't die after 8 years. Making a rough estimate of the age of the client he sells to, which would probably be 30-40, it just means that the typical client has lived to at least 48 or less, which is normal, not special. Furthermore, people who buy life insurance self-select for being more prudent in general. So, even ignoring the causal separations you could find, what he's told you is not very special. Though it separates him from other salesmen, the highest likelihood ratio you should put on this piece of evidence would be something like 1.05 (i.e. ~19 out of 20 salesmen could say the same thing), or not very informative, so you are only justified in making a very slight move toward his hypothesis, even under the most generous assumptions. You could get a better estimate of his atypicality by asking more about his clients, at which point you would have identified factors that can screen off the factor of him selling a policy. (Though in my experience, life insurance salesmen aren't very bright, and a few sentences into that explanation, you'll get the, "Oh, it's one of these people" look ...) How'd I do? Edit: Okay, I think I have to turn in my Bayes card for this one: I just came up with a reason why the hypothesis puts a high probability on the evidence, when in reality the evidence should have a low probability of existing. So it's more likely he doesn't have his facts right. Maybe this is a good case to check the "But but somebody would have noticed" [] heuristic. If one of his clients died, would he even find out? Would the insurance company tell him? Does he regularly check up on his clients?
3PeerInfinity10yI disagree with your analysis, but the details of why I disagree would be spoilers. more details: no, he's not deliberately selecting low-risk clients. He's trying to make as many sales as possible. and he's had lots of clients. I don't know the actual numbers, but he has won awards for how many policies he has sold. and he seems to honestly believe that there's something special about him that makes his clients not die. he's "one of those people". and here's the first actuarial life table I found through a quick google search: []
2PeerInfinity10yI'm going to go ahead and post the spoiler, rot13'd Zl thrff: Ur'f ylvat. Naq ur'f cebonoyl ylvat gb uvzfrys nf jryy, va beqre sbe gur yvr gb or zber pbaivapvat. Gung vf, qryvorengryl sbetrggvat nobhg gur pyvragf jub unir qvrq. Vs ur unf unq a pyvragf, naq vs gurve nirentr ntr vf 30... Rnpu lrne, gur cebonovyvgl bs rnpu bs gurz fheivivat gur arkg lrne vf, jryy, yrg'f ebhaq hc gb 99%. Gung zrnaf gung gur cebonovyvgl bs nyy bs gurz fheivivat vf 0.99^a. Rira vs ur unf bayl unq 100 pyvragf, gura gur cebonovyvgl bs gurz nyy fheivivat bar lrne vf 0.99^100=0.36 Vs ur unq 200 pyvragf, gura gur cebonovyvgl bs gurz nyy fheivivat bar lrne vf 0.99^200=0.13. Naq gung'f whfg sbe bar lrne. Gur sbezhyn tbrf rkcbaragvny ntnva vs lbh pbafvqre nyy 20 lrnef. Gur cebonovyvgl bs nyy 100 pyvragf fheivivat 20 lrnef vf 0.99^100^20=1.86R-9 Naq zl npghny erfcbafr vf... qba'g ohl gur yvsr vafhenapr. Ohg qba'g gryy nalbar gung lbh guvax ur'f ylvat. (hayrff lbh pbhag guvf cbfg.) Nyfb, gur sevraq ab ybatre pbhagf nf "gehfgrq", be ng yrnfg abg gehfgrq gb or engvbany. Bu, naq srry ernyyl thvygl sbe abg svaqvat n orggre fbyhgvba, naq cbfg gb YJ gb frr vs nalbar guvaxf bs n orggre vqrn. Ohg qba'g cbfg rabhtu vasbezngvba sbe nalbar gb npghnyyl guvax bs n orggre fbyhgvba. Naq vs fbzrbar qbrf guvax bs n orggre vqrn naljnl, vtaber vg vs vg'f gbb fpnel.
1SilasBarta10yBut I can only make inferences on what you've told me. If there's a factor that throws off the general inferences you can make from a salesman's clientele, you can't fault me for not using it. It's like you're trying to say: "This dude was born in the US. He's 50 years old. Can he speak English?" -> Yeah, probably. -> "Haha! No, he can't! I didn't tell you he was abducted to Cambodia as an infant and grew up there!" Anyway, the next step is to estimate what fraction of salesman with the same clientele composition have not had their clients die and see how atypical he is. Plus, his sales record would have to start from early in his career, or else his clients fall mostly within recent sales, a time span in which people normally don't die anyway.
2NancyLebovitz10yOn the other hand, there's also selection for people who aren't expecting to live as long as the average, and this pool includes prudent people. Anyone have information on owning life insurance and longevity?
1wedrifid10yAnd on yet another hand there is selection for people who are expected to live longer than the average (selection from the salemen directly or mediated by price.)
3Clippy10yBuying life insurance can't extend a human's life.
1SilasBarta10yThank you, Cliptain Obvious! The problem is to say how his claim is implausible or doesn't follow from his evidence, given that we already have that intuition.
2Larks10yTell him you found his pitch very interesting and persuasive, and that you'd like to buy life insurance for a 20 year period. Then, ponder for a little while; "Actually, it can't be having the contact that keeps them alive, can it? That's just a piece of paper. It must be that the sort of person who buy it are good at staying alive! And it looks like I'm one of them; this is excellent! Then , you point out that as you're not going to die, you don't need life insurance, and say goodbye. If you wanted to try to enlighten him, you might start by explicitly asking if he believed there was a causal link. But as the situation isn't really set up for honest truth-hunting, I wouldn't bother.
1soreff10yIf the salesman is omega in disguise, is this two-boxing? :-)
1Larks10yWell, kind of. Unlike in Newcombe's, we have no evidence that it's the decision that cases the long-life, as opposed to some other factor correlated with both (which seems much more likely).
2RobinZ10yWith a degree of discombobulation, I imagine. I can't see any causal mechanism by which buying insurance would cause you to live longer, so unless the salesman knows something I wouldn't expect him to, he would seem to have acquired an unreliable belief. Given this, I would postpone buying any insurance from him in case this unreliable belief could have unfortunate further consequences* and I would reduce my expectation that the salesman might prove to be an exceptional rationalist. * For example: given his superstition, he may have allotted inadequate cash reserves to cover future life insurance payments.
1cupholder10yMaybe the salesman mostly sells temporary life insurance, and just means that no clients had died while covered?

Goodhart sighting? Misunderstanding of causality sighting? Check out this recent economic analysis on (emphasis added):

For much of the modern American era, inflation has been viewed as an evil demon to be exorcised, ideally before it even rears its head. This makes sense: Inflation robs people of their savings, and the many Americans who have lived through periods of double-digit inflation know how miserable it is. But sometimes a little bit of inflation is valuable. During the Great Depression, government policies deliberately tried to creat

... (read more)

Last night I introduced a couple of friends to Newcomb's Problem/Counterfactual Mugging, and we discussed it at some length. At some point, we somehow stumbled across the question "how do you picture Omega?"

Friend A pictures Omega as a large (~8 feet) humanoid with a deep voice and a wide stone block for a head.

When Friend B hears Omega, he imagines Darmani from Majora's mask (

And for my part, I've always pictured him a humanoid with paper-white skin in a red jumpsuit with a cape (the cap... (read more)

6cousin_it10yI've always pictured Omega like this: suddenly I'm pulled from our world and appear in a sterile white room that contains two boxes. At the same moment I somehow know the problem formulation. I open one box, take the million, and return to the world.
1h-H10yThis, down to the white room and being pulled. Omega doesn't Have form or personality. He's beyond physics.
1Spurlock10yAnd when you get counterfactually mugged, you're in a sterile white room with a vending machine bill acceptor planted in the wall?
1cousin_it10yNo, just an empty room. If I take a bill out of my pocket and hold it in front of me, it disappears and I go back. If I say "no", I go back.
2WrongBot10yI've always thought of Omega as looking something like a hydralisk [] --biological and alien, almost a scaled-down Lovecraftian horror.
1sixes_and_sevens10y(Necro-thread) I can't explain why, but I've always imagined Omega to be a big hovering red sphere with a cartoonish face, and black beholder-like eyestalks coming off him from all sides. He may have been influenced by the Flying Spaghetti Monster.
0wedrifid10yHas FSM mythology got room for an archangel equivalent? Or perhaps a pantheon, an equivalent to the Norse Loki. Perhaps the love chid of FSM and a 'mortal' AGI. Given a series of bizarre tasks with incomprehensible motives that he must complete to prove himself.
1Leonhart10yAt the risk of spoiling a very good webcomic; Omega looks like this []. DAMN YOU WILLIS.
0Document10yA white human-shaped figure in a business suit, possibly faceless, stepping into a transparent blue cube for the flight part. Possibly unconsciously influenced by Einstein [].
1[anonymous]10yAnonymous? []
0Document10yQuite possibly.
1Risto_Saarelma10yThis one sounds familiar. []

AI development in the real world?

As a result, a lot of programmers at HFT firms spend most of their time trying to keep the software from running away. They create elaborate safeguard systems to form a walled garden around the traders but, exactly like a human trader, the programs know that they make money by being novel, doing things that other traders haven't thought of. These gatekeeper programs are therefore under constant, hectic development as new algorithms are rolled out. The development pace necessitates that they implement only the most importa

... (read more)

They do. We call the bargains 'one night stands' and the like.

2sketerpot10yI wonder, are celebrity one-night stands more socially acceptable than celebrity sperm banks? The former is morally condemned by a lot of people; the latter is weird.
2gwern10yThe former is morally condemned by some people (may no longer even be a simple majority); the latter is weird and disliked by most everyone.
1Blueberry10yWeird I can see, but why is it disliked by most people? Jealousy?
3gwern10yWeirdness is sufficient to inspire hatred. (If you doubt me, spend some time browsing Wikipedia's AfDs.) The recent flurry of cryonics stuff shows that, I hope. If you don't find that convincing, easy enough to manufacture reasons eg. it's covert eugenics and thus threatening, or it's a waste of resources, or it encourages risky behaviors, or it undermines traditional morality, or...

They have plenty of money, and it would make them look like weirdos. (Warning: some celebrities may have no compunction about looking like weirdos.)

[-][anonymous]10y 2

"An Alien God" was recently re-posted on the "Science Logic and Morality" forum. You may find the resulting discussion interesting.

Human intelligence can go down pretty low at either end of life - and in sickness. There is a bit of a lump of well people in the middle, though - where intelligence is not so widely distributed.

The intelligence required to do jobs is currently even more spread out. As automation progresses, the low end of that range will be gradually swallowed up.

[-][anonymous]10y 2

Nomic is way too complicated for a toy futarchy.

By no means. The great thing about complexity is that it can be managed: just break it into pieces and give a piece to each person. With a codenomic, that complexity can be self-managing to an extent.

Something like Wikipedia.

Anyway, futarchy in a codenomic? You can just come up with a simple English description of each possible change and let people "vote" based on it. It'll go nicely enough.

2JenniferRM10yGiven the downvote (parent currently -1) it might be worth pointing out that Warrigal has been an active player in the "best" nomic (which as been running since 1993) in a non-trivial capacity. E appears to have been a historian [] of Agora as of 2008, and (despite my lack of current knowledge of the nomic community since I stopped playing years and years ago) may well be one of the most experienced and historically knowledgeable "current nomic players" on the planet. On this basis I would weight eir opinion on the matter of "what can be done in a nomic" rather strongly. As an aside, nomic is super fun, very educational, and quite time consuming. If anyone here is a college student with expectations of a year of on-and-off free time I would recommend joining Agora by reading the rules [], signing up for the mailing lists [], and having some fun. When you get older, you probably won't have time for actual playing, but will appreciate the memories :-)
1[anonymous]10yI have been an active player in the past, but I'm not currently; I don't know when or if I'll get back into it. I was Agora's Historian only very briefly before that position was eliminated. My total nomic experience is definitely not more than a couple of years, as I only discovered it recently, and my historical knowledge is only what I've witnessed personally and the small amount that I happened to read once.

IIRC, Vinge said that the Singularity might look like a shockingly sudden jump from an earlier point of view, but looking back over it, it might seem like a comprehensible if somewhat bumpy road.

It hasn't been fast, but I think a paleolithic human would have a hard time understanding how an economic crisis is possible.

Very few experts believe this to be true.

Who would you consider an "expert" qualifying as an authority on this issue? Experts on classical narrow AI won't have any relevant expertise. Nor will experts on robotics, or experts on human cognitive science, or experts on evolution, or even experts on conventional probability theory and decision theory. I know of very few experts on the theory of recursively self-improving AGI, but as far as I can tell, most of them do take this threat seriously.

1timtyler10yI was thinking of those working on machine intelligence. Researchers mostly think that there are risks. I think there are risks. However, I don't think that it is very likely that engineers will need to make much use of provable stability to solve the problem. I also think there are probably lots of ways of going a little bit wrong - that do not rapidly result in a disaster.

Most of the time, scientists/inventors/engineers don't get things exactly right the first time. Unless serious effort is expended to create an AGI with a provably stable goal function that perfectly aligns with human preference, failing to get AGI exactly right the first time will probably turn us all into peach ice cream, or paperclips, or something stranger. You are arguing that testing will prevent this from happening, but (I hope) I have explained why that is not the most reliable approach.

In an argument with a philosopher, I used Bayesian updating as an argument. Guy's used to debating theists and was worried it wasn't bulletproof. Somewhat akin to how, say, the sum of angles of a triangle only equals 180 in Euclidian geometry.

My question: what are the fundamental assumptions of Bayes theorem in particular and probability theory in general? Are any of these assumptions immediate candidates for worry?

4cousin_it10yIf you're talking about math, Bayes' theorem is true and that's the end of that. If you're talking about degrees of belief that real people hold - especially if you want to convince your opponent to update in a specific direction because Bayes' theorem says so - I'd advise to use another strategy. Going meta like "you must be persuaded by these arguments because blah blah blah" gives you less bang per buck than upgrading the arguments.
1mkehrt10yWhat kind of math do you know in where things can be "true, and that's the end of that"? In math, things should be provable from a known set of axioms, not chosen to be true because they feel right. Change the axioms, and you get different result. Intuition is a good guide for finding a proof, and in picking axioms, but not much more than that. And intuitively true axioms can easily result in inconsistent systems. The questions, "what axioms do I need to accept to prove Bayes' Theorem?", "Why should I believe these axioms reflect the physical universe"? and "What proof techniques do I need to prove the theorem?" are very relevant to deciding whether to accept Bayes' Theorem as a good model of the universe.
4jimrandomh10yJaynes' book PT:LoS has a good chapter on this, where he derives Bayes' theorem from simple assumptions (use of numbers to represent plausibility, consistency between paths that compute the same value, continuity, and agreement with common sense qualitative reasoning). The assumptions are sound. Note that the validity of Bayes' theorem is a separate question from the validity of any particular set of prior probabilities, which is on much shakier ground.
3satt10yBayes's theorem follows almost immediately from the ordinary definition of conditional probability [], which I think is itself so reassuringly intuitive that no one who accepts the use of probabilities would worry about it (except perhaps in the corner case where the denominator's zero).

Wei Dai has cast some doubts on the AI-based approach

Assuming that it is unlikely we will obtain fully satisfactory answers to all of the questions before the Singularity occurs, does it really make sense to pursue an AI-based approach?

I am curious if he has "another approach" he wrote about; I am not brushed up on sl4/ob/lw prehistory.

Personally I have some interest in increasing intelligence capability on individual level via "tools of thought" kind of approach, BCI in the limit. There is not much discussion of it here.

4Wei_Dai10yNo, I haven't written in any detail about any other approach. I think when I wrote that post I was mainly worried that Eliezer/SIAI wasn't thinking enough about what other approaches might be more likely to succeed than FAI. After my visit to SIAI a few months ago, I became much less worried because I saw evidence that plenty of SIAI people were thinking seriously about this question.

You could either

  1. donate the money to charity under the view 'and you're both wrong, so there!'
  2. say that the prediction is implicitly a big AND - 'there will be an AI by 2100 AND said first AI will not have... etc.', and that the conditions allow 'short-circuiting' when any AI is created; with this change, reaching 2100 is a loss on your part.
  3. Like #2, but the loss is on Eliezer's part (the bet changes to 'I think there won't be an AI by 2100, but if there is, it won't be Friendly and etc.')

I like #2 better since I dislike implicit premises and this (whi... (read more)

1Unknowns10yEliezer and I are probably about equally confident that "there will not be AI by 2100, and both Eliezer and Unknown will still be alive" is incorrect. So it doesn't seem very fair to select either 2 or 3. So option 1 seems better.

From the Long Now department: "He Took a Polaroid Every Day, Until the Day He Died"

My comment on the Hacker News page describes my little webcam script to use with cron and (again) links to my Prediction Book page.

If you have many different (and conflicting, in that they demand undivided attention) interests: if it was possible, would copying yourself in order to pursue them more efficiently satisfy you?

One copy gets to learn drawing, another one immerses itself in mathematics & physics, etc. In time, they can grow very different.

(Is this scenario much different to you than simply having children?)

6[anonymous]10yI wouldn't have problems copying myself as long as I could merge the copies afterwards. However, it might not be possible to have a merge operation for human level systems that both preserves information and preserves sanity. E.g. if one copy started studying philosophy and radically changed its world views from the original, how do you merge this copy back into the original without losing information?
3JenniferRM10yDavid Brin's novel Kiln People [] has this "merging back" idea, with cheap copies, using clay for a lot of the material and running on a hydrogen based metabolism so they are very short lived (hours to weeks, depending on $$) and have to merge back relatively soon in order to keep continuity of consciousness through their long lived original. Lots of fascinating practical economic, ethical, social, military, and political details are explored while a noir detective story happens in the foreground. I recommend it :-)
2humpolec10yI agree, I don't think merge is possible in this scenario. I still see some gains, though (especially when communication is possible): * I (the copy that does X) am happy because I do what I wanted. * I (the other copies) am happy because I partly identify with the other copy (as I would be proud of my child/student?) * I (all copies) get results I wanted (research, creative, or even personal insights if the first copy is able to communicate them)
2[anonymous]10yIf you don't have the ability to merge, would the copies get equal rights as the original? Or would the original control all the resources and the copies get treated as second class citizens? If the copies were second class citizens, I would probably not fork because this would result in slavery. If the copies do get equal rights, how do you plan to allocate resources that you had before forking such as wealth and friends? If I split the wealth down the middle, I would probably be OK with the lack of merging. However, I'm not sure how I would divide up social relationships between the copy and the original. If both the original and the copy had to reduce their financial and social capital by half, this might have a net negative utility. If the goal is to just learn a new skill such as drawing, a more efficient solution might involving uploading yourself without copying yourself and then running the upload faster than realtime. I.e. the upload thinks it has spent a year learning a new skill but only a day has gone by in the real world. However, this trick won't work if the goal involves interacting with others unless they are also willing to run faster than realtime.
1NancyLebovitz10yTentatively-- there's be a central uberperson which wouldn't be that much like a single human being. If I had reason to think it was safe, I'd really like to live that way.
4Peter_de_Blanc10yThat sounds (to me) better than having children, but not as good as living longer.
2KrisC10ySounds wonderful. Divide and conquer. As this sounds like a computer assisted scenario, I would like the ability to append memories while sleeping. Wake up and have access to the memories of the copy. This would not necessarily include full proficiency as I suspect that muscle memory may not get copied.
1red7510yWaste of processing power. Having dozens of focuses of attention and corresponding body/brain construction is more efficient.
1DanArmak10yCopying has at best zero utility (as regards interests): each copy only indulges in one interest, and I anticipate being only one copy, even if I don't know in advance which one. How is having children at all similar? 1) children would have different interests; 2) I cannot control (precommit) future children; 3) raising children would be for me a huge negative utility - both emotionally and resource-wise.
6Jordan10yThis is not true for me. I care about my ideas beyond my own desire to implement them. If I knew there was a passionate and capable person willing to take over some of my ideas (which I'd otherwise not have time for), I'd jump on the opportunity. Doubly so if the other person was a copy of me, in which case I'd not only have a guarantee on competence, but assurance that the person would be able to relate the story and product to me afterwards (and possibly share the profit).
2humpolec10yI think people can feel a sense of accomplishment when their child achieved something they wanted but never got around to.

I would like feedback on my recent blog post:

It's simplistic for this crowd, but something that bothered me for a while. When I first saw Kurzweil speak in person (GDC 2008) he of course showed both linear and log scale plots. But I always thought the log scale plots were just a convenient way to fit more on the screen, that the "real" behavior was more like the linear scale plot, building to a dramatic steep slope in the coming years.

Instead I now believe in many cases the log plot is ... (read more)

2timtyler10yMy essay on the topic: [] See also: "The Singularity" by Lyle Burkhead - see the section "Exponential functions don't have singularities!" * [] It's not exponential, it's sigmoidal * [] The Singularity Myth * [] Singularity Skepticism: Exposing Exponential Errors * [] IMO, those interested in computational limits should discuss per-kg figures. The metric Moore's law uses is not much use really - since it would be relatively easy to make large asynchronous ICs with lots of faults - which would make a complete mess of the "law".
3ABranco10yI would love to see an ongoing big wiki-style FAQ addressing all possible received critics of the singularity — of course, refuting the refutable ones, accepting the sensible. A version with steroids of what this one did with Atheism []. Team would be: * one guy inviting and sorting out criticism and updating the website. * an ad hoc team of responders. It seems criticism and answers have been scattered all over. There seems to be no one-stop source for that.
2steven046110yHere's a pretty extensive FAQ [], though I have reservations about a lot of the answers.
3timtyler10yThe authors are - or were - SI fellows, though - and the SI is a major Singularity promoter. Is that really a sensible place to go for Singularity criticism? [] lists some of the objections.
1JamesAndrix10yThis is easier to say when you're near the top of the current curve. It doesn't affect me much that my computer can't handle hi-def youtube, because I'm just a couple of doubling times behind the state of the art. But if you were using a computer ten doubling times back, you'd have trouble even just reading lesswrong. Even if you overcame the format and software issues, we'd be trading funny cat videos that are bigger than all your storage. You'd get nothing without a helper god to downsample them. When the singularity approaches, the doubling time will decrease, for some people. Maybe not for all. Maybe will will /feel/ like a linear increase in utility for the people who's abilities are being increased right along. For people who are 10 doublings behind and still falling, it will be obvious something is different..
1kmeme10yConsider $/MIPS available in the mainstream open market. The doubling time of this can't go down "for some people", it can only go down globally. Will this doubling time decrease leading up to the Singularity? Or during it? I always felt that's what the Singularity was, an acceleration of Moore's Law type progress. But I wrote the post because I think it's easy to see a linear plot of exponential growth and say "look there, it's shooting through the roof, that will be crazy!". But in fact it won't be any crazier than progress is today. It will require a new growth term, machine intelligence kicking in for example, to actually feel like things are accelerating.
1JamesAndrix10yIt could if, for example, it were only available in large chunks. If you have $50 today you can't get the $/MIPS of a $5000 server. You could maybe rent the time, but that requires a high level of knowledge, existing internet access at some level, and an application that is still meaningful on a remote basis. The first augmentation technology that requires surgery will impose a different kind of 'cost'. and will spread unevenly even among people who have the money. It's also important to note that an increase in doubling time would show up as a /bend/ in a log scale graph, not a straight line.
2kmeme10yYes Kurzweil does show a bend in the real data in several cases. I did not try to duplicate that in my plots, I just did straight doubling every year. I think any bending in the log scale plot could be fairly called acceleration. But just the doubling itself, while it leads to ever-increases step sizes, is not acceleration. In the case of computer performance it seems clear exponential growth of power produces only linear growth in utility. I feel this point is not made clear in all contexts. In presentations I felt some of the linear scale graphs were used to "hype" the idea that everything was speeding up dramatically. I think only the bend points to a "speeding up".

You're forgetting about steganography; encrypted messages can be made to look like vacation photos, music, spam or something else. Using steganography sounds complicated when you describe it, but in practice all the details are handled transparently by some software, so it's just one-time setup.

2rhollerith_dot_com10yThe occupying power announces that until the insurgents stop killing innocent civilians and sowing disorder, no one in contested territory is permitted to use the internet to transmit vacation photos, music, spam, etc. Civilians whose livelihoods are interrupted by these restrictions can apply for monetary compensation from the occupying power. Music and other entertainment will still be available from iTunes and other major centralized services (unless and until there are signs that these centralized services are being used to broadcast hidden messages). In the Malay Emergency, the occupying power, which won the conflict, required [] 500,000 civilians to relocate to new villages surrounded by barbed wire, police posts and floodlit areas. Compared to that, restrictions on the internet are mild. Also, the need for steg reduces the bandwidth available to the insurgents -- perhaps below the level required for voice communication, which requires the communicators to be literate, which denys the communications channel to a large fraction of the world's insurgents.
0RobinZ10y"One-time" is a bit of a stretch (all it takes is being found out once to greatly impair the value of any particular method), but yeah - steganography is an established and worthy technique.

This is more or less what I was trying to do, but I neglected to treat "AGI is impossible" as equivalent to "AGI will never happen".

I need to have a prior in order to update, so sure, let's use Laplace.

I'd have to be an idiot to ever press the button at all, but let's say I'm in Harry's situation with the time-turner and someone else pushed the button ten times before I could tell them not to.

I don't feel like doing the calculus to actually apply Bayes myself here, so I'll use my vague nonunderstanding of Wikipedia's formula for the rule of succession and say p=11/12.

For one -- it hasn't already happened. And there is no public research suggesting that it is much closer to happening now than it has ever been. The first claims of impending human-level AGI were made ~50 years ago. Much money and research has been exhausted since then, but it hasn't happened yet.

The fact that it hasn't happened yet is not evidence against its happening if you cannot survive its happening. If you cannot survive its happening, then the fact that it has not happened in the last 50 years is not just weaker evidence than it would otherwise... (read more)

2PeterS10yDo you take the Fermi paradox seriously, or is the probability of your being destroyed by a galactic civilization, assuming that one exists, low enough? The evidential gap w.r.t. ET civilization spans billions of years -- but this is not evidence at all according to the above. Neither do I believe in the coming of an imminent nuclear winter, though (a) it would leave me dead and (b) I nevertheless take the absence of such a disaster over the preceeding decades to be nontrivial evidence that its not on its way. Say you're playing Russian Roulette with a 6-round revolver which either has 1 or 0 live rounds in it. Pull the trigger 4 times -- every time you end up still alive. According to what you have said, your probability estimates for either * there being a single round in the revolver or * the revolver being unloaded should be the same as before you had played any rounds at all. Imagine pulling the trigger 5 times and still being alive -- is there a 50/50 chance that the gun is loaded? I find the technique you're suggesting interesting, but I don't employ it. Tiiba suggested that distributive capability is the most important of the "powers inherent to all computers". Protein folding simulation was an illustrative example of a cutting edge distributed computing endeavor, which is still greatly underpowered in terms of what AGI needs to milk out of it to live up to FOOMy claims. He wants to catch all the fish in the sea with a large net, and I am telling him that we only have a net big enough for a few hundred fish. edit: It occurred to me that I have written with a somewhat interrogative tone and many examples. My apologies.
3gwern10yExamples are great. The examples a person supplies are often more valuable than their general statements. In philosophy, one of the most valuable questions one can ask is 'can you give an example of what you mean by that?'
1rhollerith_dot_com10yIf before every time I pull the trigger, I spin the revolver in such a way that it comes to a stop in a position that is completely uncorrelated with its pre-spin position, then yes, IMO the probability is the same as before I had played any rounds at all (namely .5). If an evil demon were to adjust the revolver after I spin it and before I pull the trigger, that is a selection effect. If the demon's adjustments are skillful enough and made for the purpose of deceiving me, my trigger pulls are no longer a random sample from the space of possible outcomes. Probability is not a property of reality but rather a property of an observer. If a particular observer is not robust enough to survive a particular experiment, the observer will not be able to learn from the experiment the same way a more robust observer can. As I play Russian roulette, the P(gun has bullet) assigned by someone watching me at a safe distance can change, but my P(gun has bullet) cannot change because of the law of conservation of expected evidence []. In particular, a trigger pull that does not result in a bang does not decrease my probability that the gun contains a bullet because a trigger pull that results in a bang does not increase it (because I do not survive a trigger pull that results in a bang).
1thomblake10yI'm not sure this would work in practice. Let's say you're betting on this particular game, with the winnings/losings being useful in some way even if you don't survive the game. Then, after spinning and pulling the trigger a million times, would you still bet as though the odds were 1:1? I'm pretty sure that's not a winning strategy, when viewed from the outside (therefore, still not winning when viewed from the inside).
1rhollerith_dot_com10yYou have persuaded me that my analysis in grandparent of the Russian-roulette scenario is probably incorrect. The scenario of the black box that responds with either "heads" or "tails" is different because in the Russian-roulette scenario, we have a partial causal model of the "bang"/"no bang" event. (In particular, we know that the revolver contains either one bullet or zero bullets.) Apparently, causal knowledge can interact with knowledge of past behavior to produce knowledge of future behavior even if the knowledge of past behavior is subject to the strongest kind of observational selection efffects.
1PeterS10yYour last point was persuasive... though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing. Would it be within the first AGI's capabilities to immediately effect my destruction before I am able to update on its existence -- provided that (a) it is developed by the private sector and not e.g. some special access DoD program, and (b) ETAs up to "sometime this century" are accurate? I think not, though I admit to being fairly uncertain. I acknowledge that this line of reasoning presented in my original comment was not of high caliber -- though I still dispute Tiiba's claim regarding an AI advanced enough to scrape by in conversation with a 5 year old, as well as that distributive capabilities are the greatest power at play here.
0rhollerith_dot_com10yI intentionally delayed this reply (by > 5 days) to test the hypothesis that slowing down the pace of a conversation on LW will improve it. When we try to estimate the number of technological civilizations that evolved on main-sequence stars in our past light cone, we must not use the presence of at least one tech civ (namely, us) as evidence of the presence of another one (namely, ET) because if that first tech civ had not evolved, we would have no way to observe that outcome (because we would not exist). In other words, we should pretend we know nothing of our own existence or the existence of clades in our ancestral line, in particular, the existence of the eukaryotes and the metazoa, when trying to estimate the number of tech civs in our past light cone. I am not an expert on ETIs, but the following seems (barely) worth mentioning: the fact that prokaryotic life arose so quickly after the formation of the Earth's crust is IMHO significant evidence that there is simple (unicellular or similar) life in other star systems. It is evidence, but less strong than it would be if we fail to account for observational selection effects. Details follow. The fact that there are no obvious signs of an ET tech civ, e.g., alien space ships in the solar system, is commonly believed to the be strongest sign that there were no ET tech civs in our past light cone with the means and desire (specifically, desire on at least part of the civ that was not thwarted by the rest of the civ) to expand outwards into space. Well, it seems to me that there is a good chance that we would not have survived an encounter with the leading wave of such an expansion, and therefore the lack of evidence of such an expansion should not cause us to update our probability of the existence of such an expansion as much as it should have if we certainly could have survived the encounter. Still, the fact that there are no obvious signs (such as alien space ships in the solar system) of ET is the strongest
1Stuart_Armstrong10yIf there were two universes, one very likely to evolve life and one very unlikely, and all we knew was that we existed in one, then we are much more likely to exist in the first universe. Hence our own existence is evidence about the likelihood of life evolving, and there still is a Fermi paradox.
0rhollerith_dot_com10yAgree. Disagree because your hypothetical situation requires a different analysis than the situation we find ourselves in. In your hypothetical, we have somehow managed to acquire evidence for the existence of a second universe and to acquire evidence that life is much more likely in one than in the other. Well, let us get specific about how that might come about. Our universe contains gamma-ray bursters that probably kill any pre-intelligence-explosion civilization within ten light-years or so of them, and our astronomers have observed the rate * density at which these bursters occur. Consequently, we might discover that one of the two universes has a much higher rate * density of bursters than the other universe. For that discovery to be consistent with the hypothetical posed in parent, we must have discovered that fact while somehow becoming or remaining completely ignorant as to which universe we are in. We might discover further that although we have managed to determine the rate * density of the bursters in the other universe, we cannot travel between the universes. We must suppose something like that because the hypothetical in parent requires that no civilization in one universe can spread to the other one. (We can infer that requirement from the analysis and the conclusion in parent.) I hope that having gotten specific and fleshed out your hypothetical a little, you have become open to the possibility that your hypothetical situation is different enough from the situation in which we find ourselves for us to reach a different conclusion. In the situation in which we find ourselves, one salient piece of evidence we have for or against ET in our past light cone is the fact that there is no obvious evidence of ET in our vicinity, e.g., here on Earth or on the Moon or something. And again, this piece of evidence is really only evidence against ETs that would let us continue to exist if their expansion reached us, but there's a non-negligible probabili
0Stuart_Armstrong10yThese need not be actual universes, just hypothetical universes that we have assigned a probability to. Given most priors over possible universes, the fact we exist will bump up the probability of there being lots of life. The fact we observe no life will bump down the probability, but the first effect can't be ignored.
0rhollerith_dot_com10ySo in your view there is zero selection effect in this probability calculation? In other words, our own existence increases your probability of there being lots of life just as much as the existence of an extraterrestrial civilization would? In the previous sentence, please interpret "increase your probability just as much as" as "is represented by the same likelihood ratio as". And the existence of human civilization increases your P(lots of life) just as much as it would if you were an immortal invulnerable observer who has always existed and who would have survived any calamity that would have killed the humans or prevented the evolution of humans? Finally, is there any probability calculation in which you would adjust the results of the calculation to account for an observational selection effect? Would you for example take observational selection effects into account in calculating the probability that you are a Boltzmann brain? I can get more specific with that last question if you like.
0Stuart_Armstrong10yDepends how independent the two are. Also, myself existing increases the probability of human-like life existing, while the alien civilization increases the probability of life similar to themselves existing. If we're similar, the combined effects will be particularly strong for theories of convergent evolution. The line of reasoning for immortal observers is similar. I thought that was exactly what I was doing? To be technical, I was using a variant of full non-indexical conditioning (FNC), which is an unloved bastard son of the SIA (self-indication assumption).
0rhollerith_dot_com10yCan I get a yes or no on my question of whether you take the existence of human civilization to be just as strong evidence for the probabilities we have been discussing as you would have taken it to be if you were a non-human observing human civilization from a position of invulnerability? Actually, "invulnerability" is not the right word: what I mean is, "if you were a non-human whose coming into existence was never in doubt and whose ability to observe the non-appearance of human civilization was never in doubt."
1Stuart_Armstrong10yIf the existence of the "invulnerable non-human" (INH) is completely independent from the existence of human-like civilizations, then: * If the INH gets the information "there are human-like civilizations in your universe" then this changes his prior for "lots of human-like civilizations" much less that we get from noticing that we exist. * If the INH gets the information "there are human-like civilizations in your immediate neighbourhood", then his prior is updated pretty similarly to ours.
1rhollerith_dot_com10yThanks for answering my question. I repeat that you and I are in disagreement about this particular application of observational selection effects, a.k.a., the anthropic principle and would probably also disagree about their application to an existential risk. I notice that last month saw the publication of a new paper, "Anthropic Shadow: Observation Selection Effects and Human Extinction Risk" []by Bostrum, Sandberg and my favorite astronomy professor, Milan M Circovic. As an aid to navigation, let me link to the ancestor to this comment [] at which the conversation turned to observation selection effects.
1gwern10yI have been meaning to write a post summarizing "Anthropic Shadow"; would anyone besides you and me be interested in it?
1rhollerith_dot_com10yI think you should write that post because thoughtful respected participants on LW use the anthropic principle incorrectly, IMHO. The gentleman who wrote great grandparent for example is respected enough to have been invited to attend SIAI's workshop on decision theory earlier this year. And thoughtful respected participant Cousin It probably misapplied the anthropic principle in the first paragraph of this comment []. I say "probably" because the context has to do with "modal realism" and other wooly thinking that I cannot digest, but I have not been able to think of any context in which Cousin It's "every passing day without incident should weaken your faith in the anthropic explanation" is a sound argument. (Many less thoughtful or less respected participants here have misapplied or failed to take into account the anthropic principle, too.)
1gwern10yIt has been a while since I skimmed "Anthropic Shadow", but IIRC a key point or assumption in their formula was that the more recent a risk would have occurred or not, the less likely 'we' are to have observed the risk occurring, because more recently = less time for observers to recover from the existential risk or fresh observers to have evolved. This suggests a weak version: the longer we exist, the fewer risks' absence we need to appeal to an observer-based principle. (But thinking about it, maybe the right version is the exact opposite. It's hard to think about this sort of thing.)
0gwern10yI've read "Anthropic Shadow" [] a few times now. I don't think I will write a post on it. It does a pretty good job of explaining itself, and I couldn't think of any uses for it. The Shadow only biases estimates when some narrow conditions are met: 1. your estimate has to be based strictly on your past 2. of a random event 3. the events have to be very destructive to observers like yourself 4. and also rare to begin with So it basically only applies to global existential risks, and there aren't that many of them. Nor can we apply it to interesting examples like the Singularity, because that's not a random event but dependent on our development.
0Stuart_Armstrong10yIndeed. I, for one, do not worry about the standard doomesday argument, and such. I would argue that SIA is the only consistent anthropic principle, but that's a long argument, and a long post to write one day. Fortunately, the Anthropic shadow argument can be accepted whatever type of anthropic reasoning you use.
1Pavitra10yI'm not convinced that works that way. Suppose I have the following (unreasonable, but illustrative) prior: 0.5 for P=(AGI is possible), 1 for Q=(if AGI is possible, then it will occur in 2011), 0.1 for R=(if AGI occurs, then I will survive), and 1 for S=(I will survive if AGI is impossible or otherwise fails to occur in 2011). The events of interest are P and R. P,R: 0.05. I survive. P,~R: 0.45. I do not survive. (This outcome will not be observed.) ~P: 0.5. I survive. R is irrelevant. After I observe myself to still be alive at the end of 2011 (which, due to anthropic bias, is guaranteed provided I'm there to make the observation), my posterior probability for P (AGI is possible) should be 0.05/(0.05+0.5) = 5/55 = 1/11 = 0.0909..., which is considerably less than the 0.5 I would have estimated beforehand. By updating on my own existence, I infer a lower probability of the possibility of something that could kill me.

It's an interesting analysis - have you confirmed the appearance of that distribution with real-world data? I suppose you'd need a substantial body of factual claims about which statistical information is available...

Not that many will care, but I should get a brief appearance on Dateline NBC Friday, Aug. 20, at 10 p.m. Eastern/Pacific. A case I prosecuted is getting the Dateline treatment.

Elderly atheist farmer dead; his friend the popular preacher's the suspect.


That's quite true. I mean, honestly, I would expect any AI to parallelize very well, although I'm loathe to trust my intuition about anything related to AGI. But I don't think we can take it as a given that the AI will be able to get linear or better gains in its speed of thought when going, say, from some big parallel supercomputer in a datacenter to trying to spread itself out through commodity hardware in other physical locations.

If a prospective AI had a tremendous, planet-sized amount of hardware available to it, it might hardly matter, but in the r... (read more)

4Morendil10yI suggest you Google the word "botnet". It isn't particularly hard for human-level intelligences to gain access to substantial computing power for selfish purposes.
1cata10yPoint taken.

Unless I'm really misinterpreting you, "simply copy the algorithm into more hardware" sounds totally silly to me. In general, tasks need to be designed from the ground up with parallelization in mind in order to be efficiently parallelizable. Rarely have I ever wanted to run a serial algorithm in parallel and had it be a matter of "simply run the same old thing on each one and put the results together." The more complicated the algorithm in question, the more work it takes to efficiently and correctly split up the work; and at really... (read more)

1Clippy10yI know. Does any human mathematician really doubt that?
4Unknowns10yI've been becoming more and more convinced that Kevin and Clippy are the same person. Besides Clippy's attempt to get money for Kevin, one reason is that both of them refer to people with labels like "User:Kevin". More evidence just came in here, namely these comments within 5 minutes of each other.
3multifoliaterose10yYes, there are humans mathematicians who doubt that P is not equal to NP. See "Guest Column: The P=?NP Poll" [] by William Gasarch where a poll was taken of 100 experts, 9 of whom ventured the guess that P = NP and 22 of whom offered no opinion on how the P vs. NP question will be resolved. The document has quotes from various of the people polled elaborating on what their beliefs are on this matter.
1Kevin10yHow do you know you know?
2JoshuaZ10yThere's a very good summary by Scott Aaronson describing why we believe that P is very likely to be not equal to NP []. However, Clippy's confidence seems unjustified. In particular, there was a poll a few years ago that showed that a majority of computer scientists believe that P=NP but a substantial fraction do not. (The link was here [] but seems to be not functioning at the moment (according to's main page today they have a scheduled outage of most Web services for maintenance so I'll check again later. I don't remember the exact numbers so I can't cite them right now)). This isn't precisely my area, but speaking as a mathematician whose work touches on complexity issues, I'd estimate around a 1/100 chance that P=NP.
1Sniffnoy10yURL is repeated twice in link?
1Clippy10yBecause if it were otherwise -- if verifying a solution were of the same order of computational difficulty of finding it -- it would be a lot harder to account for my observations than if it weren't so. For example, verifying a proof would be of similar difficulty to finding the proof, which would mean nature would stumble upon representations isomorphic to either with similar probability, which we do not see. The possibility that P = NP but with a "large polynomial degree" or constant is too ridiculous to be taken seriously; the algorithmic complexity of the set of NP-complete problems does not permit a shortcut that characterizes the entire set in a way that would allow such a solution to exist. I can't present a formal proof, but I have sufficient reason to predicate future actions on P ≠ NP, for the same reason I have sufficient reason to predicate future actions on any belief I hold, including beliefs about the provability or truth of mathematical theorems.
1Kevin10yMost human mathematicians think along similar lines. It will still be a big deal when P ≠ NP is proven, if for no other reason that it pays a million dollars. That's a lot of paperclips. Let me know if you think you can solve any of these! []

The part to feel guilty about is that I chose not to explain that the salesman is probably either lying, or insane, or both, and therefore probably shouldn't be considered "a trusted friend". And also that I chose to just try to avoid both of these people, rather than... thinking of a less blatantly unfriendly solution.

As people are probably aware, Hitchens has cancer, which is likely to kill him in the not-too-distant future. There does not seem to be much to be done about this; but I wonder if it's possible to pass the hat to pay for cryonics for him? Apart from the fuzzies of saving a life with X percent probability, which can be had much cheaper by sending food to Africa, it might serve as marketing for cryonics, causing others to sign up. Of course, this assumes that he would accept, and also that there wouldn't be a perception that he was just grasping at any straw available.

4Paul Crowley10yI'd love to persuade him, but no way am I passing a hat.
1dclayh10y1. Would Hitchens not be able to afford cryonics without donations? 2. What's wrong with this? Isn't that exactly what cryonics is: grasping the only available straw? (Hm, how do I get a sentence inside the numbering indentation but outside the quotation?)
2RolfAndreassen10yPerhaps so, but would he consider it the best use of his resources? While if he gets it for free, take it or lose it, that's a different matter. For marketing purposes it would be an epic fail. In interviews he has made the point that no, he will not be doing any deathbed conversions unless he goes mad from pain. If cryonics is seen as only a deathbed conversion to a different religion (easy pattern completions: "Rapture of the Nerds", "weird beliefs = cults") it'll merely reinforce the perception of cryonics as something rather kooky which serious people needn't spend time on. Your point is correct, but will only work as PR if that's how it gets across to the public: This is a straw with an actual chance of working.
1dclayh10yAh, I see. Certainly it would be better if he made the choice well before he's at death's door/in terrible pain/etc..

Knowing that medicine is often more about signaling care than improving health, it's hard for me to make a big fuss over some minor ailment of a friend or family member. Consciously trying to signal care seems too fake and manipulative. Unfortunately, others then interpret my lack of fuss-making as not caring. Has anyone else run into this problem, and if so, how did you deal with it?

6byrnema10yI feel like I've wrestled with this, or something similar. I will throw some thoughts out. In relating to your example, I recall times when I was expected to give care that I didn't think a person needed, and I guess my sense was that they were weak to expect it (and so I was unable to empathize with them), or that my fake care would encourage them to be weak. I also felt that the care was disingenuous because it wasn't really doing anything. I no longer feel that way, and what changed over several years, I guess, is a deeper realization (along an independent, separate path of experiences, including being a mother) of the human condition: we are all lonely, isolated minds trapped in physical bodies. We ache for connection -- more so at different times of our lives, and some more than others, with different levels of comfort for different levels -- but infants can't survive without affection and children and adults also need affection. (Alicorn's "love languages" appropriate here.) Whatever expressions of affection we prefer, I think we need all of them a little bit, and physical, platonic affection is something we just don't receive as often. (I hear this is especially true for the elderly.) Signaling medical care is token for physical care, thus it stands in for physical affection -- even if there is no physical contact involved. If there is physical contact involved -- the placement of a band-aid on a knee -- then that is even better. I think it is important to realize that people do have a need for such physical affection, and medical situations provide a context for this (often at times when people are in need of more affection anyway).
4orthonormal10yGood point. But the next question ought to be whether there's a creative third alternative [] that would allow us to better signal our caring while being less wasteful. In some cases (the rising popularity of hospice rather than hospital for terminal illness), we can see this already being done. (For a similar example, some couples planning weddings are moving away from the massively wasteful† registry option in favor of other ideas. It looks tacky to just ask for a cash donation, of course, but there really are third alternatives-- one couple asked for donations toward the specific events they planned for their honeymoon, while others ask for donations toward a favored list of charities. Etc.) † Guests signal their generosity and regard for the new couple by buying them something from a set of nice things. However, the couple typically asks for things that are uselessly nicer than what they would buy themselves if it were their money, so as to signal sophistication. The end result is that a lot of money gets wasted on overly specific kitchen gadgets which will gather dust, or overly nice china that rarely gets used, etc.
[-][anonymous]10y 1

One way to model someone's beliefs, at a given frozen moment of time, is as a real-valued function P on the set of all assertions. In an ideal situation, P will be subject to a lot of consistency conditions, for instance if A is a logical consequence of B, then P(A) is not smaller than P(B). This ideal P is very smart: if such a P has P(math axioms) very close to 1, then it will have P(math theorems) very close to 1 as well.

Clearly, even a Bayesian superintelligence is not going to maintain an infinitely large database of values of P, that it updates fro... (read more)

3wedrifid10yClarify for me what you are saying here. Why would a bounded-rational superintelligence maintain a logically inconsistent belief system? Are you making the following observation? * Even a superintelligence is not logically omniscient * There are inevitably going to be complicated mathematical properties of the superintelligence's map that are not worth spending processing time on. This is ensured by the limits of physics itself. * Outside the 'bounds' of the superintelligence's logical searching there will be logical properties for which the superintelligence's beliefs are not consistent. * A bayesian superintelligence is not logically omniscient so some of it's beliefs will be logically inconsistent. My impression is that the above holds true unless the superintelligence in question cripples itself, sacrificing most of it's instrumentally rational capability for epistemic purity.
2[anonymous]10yI am glad that a term like "bounded-rational" exists. If it's been discussed someplace very thoroughly then I likely don't have very much to add. What are some proposals for modeling bounded Bayesianism? I think what I'm saying is consistent with your bullet points, but I would go further. I'll focus on one point: I do not think it's possible for a bounded agent to be epistemically pure, even having sacrificed most or all of its instrumentally rational capability. Epistemic impurity is built right into math and logic. Let me make the following assumption about our bounded rational agent: given any assertion A, it has the capability of computing its prior P(A) in time that is polynomial in the length of A. That is, it is not strictly agnostic about anything. Since there exist assertions A which are logical consequences of some axioms, but whose shortest proof is super-polynomial (in fact it gets much worse) in the length of A, it seem very unlikely that we will have P(A) > P(Axioms) for all provable assertions A. (I think you could make this into a rigorous mathematical statement, but I am not claiming to have proved it--I don't see how to rule out the possibility that P always computes P(A) > P(Axioms) (and quickly!) just by luck. Such a P would be very valuable.)
1wedrifid10yI believe you are correct. A bounded rational agent that is not strictly agnostic about anything will produce outputs that contain logical inconsistencies. For a superintelligence to avoid such inconsistencies it would have to violate the 'not strictly agnostic about anything' assumption either explicitly or practically. By 'practically' I mean it could refuse to return output until such time as it has proven the logical correctness of a given A. It may burn up the neg-entropy of its future light cone before it returns but hey, at least it was never wrong. A bounded rational agent in denial about its 'bounds'.

Afaik, the government response after Katrina was unusually incompetent for a first world country, and it doesn't make sense to draw general conclusions from it.

The basic premise was pretty silly. (You have strong AI, perfect duplication of adult humans, regular flights to Earth - and you need hundreds of them on the Moon to do... nothing?)

A long time ago on Overcoming Bias, there was a thread started by Eliezer which was a link to a post on someone else's blog. The linked post posed a question, something like: "Consider two scientists. One does twenty experiments, and formulates a theory that explains all twenty results. Another does ten experiments, formulates a theory that adequately explains all ten results, does another ten experiments, and finds that eir theory correctly predicted the results. Which theory should we trust more and why?"

I remember Eliezer said he thought he ha... (read more)

3gwern10yI remember vaguely one discussion of Bayesianism vs. frequentism, where the frequentists held that a study in which the experimenters resolved to keep making observations until they observed Foo X times (for a total of X times and Y total observations) must be treated statistically different from a study where the experimenters resolved to make Y total observations but wound up X times observing Foo; while from the Bayesian perspective, these studies, with their identical objective facts, ought to be treated the same. Does this sound like it?
2Matt_Simpson10yHere's the post []
1sketerpot10yHow valuable was each experiment? If you make a theory after ten experiments, then you could design the next ten experiments to be very specific to the theory: if the theory is right, they come out one way, and they should come out a very different way if the theory is wrong. It's like writing test-cases for software: once you know what exactly you're testing, you can write test cases that aim at any potential weak spots, in a deliberate and targeted attempt to break it. So if we're assuming that these scientists are actual people (rather than really hypothetical abstractions), I would give more credence to the guy who did ten experiments, formulated a theory, and did ten more experiments, iff those later ten experiments look like they were designed to give new information and really stress-test the theory. If we're talking about the exact same 20 experiments, then I would generally favor doing maybe 13 experiments, making a theory, and then doing the other 7, to avoid overfitting. Or split the experiments up into two sets of ten, and have two scientists each look at ten of the experiments, make a theory, then test it with the other ten. This kind of thinking would have killed Ptolemy's theory of epicycles, which is a classic case of overcomplicating the theory to match your observations. I know that's hardly a formal answer, but I think the original question was oversimplifying.

Interesting article:

One reason why it's a good idea someone with OCD (or for that matter, Asperger, psychosis, autism, paranoia, schizophrenia — whatever) should make sure new acquaintances know of his/her condition:

I suppose that being presented by a third party, as in the example, should make a difference when compared to self-labeling (which may sound like excusing oneself)?

I think it's a quantum immortality argument. If you, the observer, are no longer on Earth, the Earth can be destroyed because its destruction no longer necessitates your death.

I am not sure what you mean about the "different levels of intelligence" point. Maybe this:

"A machine intelligence that is of "roughly human-level" is actually likely to be either vastly superior in some domans or vastly inferior in others - simply because machine intelligence so far has proven to be so vastly different from our own in terms of its strengths and weaknesses [...]"

I think that if The Future were commonly used, it would rapidly acquire all the weird connotations of The Singularity, or worse.

One problem with Suber's Initial Rules is that they do not enforce timing - that my (sadly expired) Livejournal nomic lasted as long as it did I attribute in part to having explicit deadlines for turn-based play.

[-][anonymous]10y 1

I'm going to be starting a game soon, but I'd planned on doing it on another forum (which is home to a few LW lurkers - who knows, we may vote to move the thread!) I've been working out the initial rules over the past few weeks. Should I make mention of it here when it starts?

All the more reason to be working on the problem now, while there's still time. I don't think the AGI problem is hardware-bound at this point, but it should be worth working on either way.

Missing the point. I wasn't arguing that there aren't reasons to think that the bad AI goes FOOM won't happen. Indeed, I said explicitly that I didn't think it would occur. My point was that if one is going to make an argument that relies on that here one needs to be aware that the premise is controversial and be clear about that (say giving basic reasoning for it, or even just saying "If one accepts that X then..." etc.).

[-][anonymous]10y 1

Alcumus is a lot like what you're describing.

I'm not familiar at all with Pearl's formalism. But from what I see on this site, I gather that the key insight of updateless decision theory is to maximize utility without conditioning on information about what world you're in, and the key insight of timeless decision theory is what you're describing (Eliezer summarizes it as "Choose as though controlling the logical output of the abstract computation you implement, including the output of all other instantiations and simulations of that computation.")

As an alternative to trying to figure out what you'd want if civilization fell apart, are there ways to improve how civilization deals with disasters?

If a first world country were swatted hard by a tsunami or comparable disaster, what kind of prep, tech, or social structures might help more than what we've got now if they were there in advance?

Has there ever been a practical proof-of-concept system, even a toy one, for futarchy? Not just a "bare" prediction market, but actually tying the thing directly to policy.

If not, I suggest a programming nomic (aka codenomic) for this purpose.

If you're not familiar with the concept of nomic, it's a little tricky to explain, but there's a live one here in ECMAScript/Javascript, and an old copy of the PerlNomic codebase here. (There's also a scholarly article [PDF] on PerlNomic, for those interested.)

Also, if you're not familiar with the concept of... (read more)

Well, sure - but if we build them appropriately, they will. We should be well motivated to do that - people are not going to want to buy a bad robots, or machine assistants that don't do what we tell them. Consumers buying potentially-dangerous machines will be looking for saftey features - STOP buttons and the like. The "bad" projects are less likely to get funding or mindshare - and so have less chance of getting off the ground.

2WrongBot10yYou are assuming the very thing that is being claimed to be astonishingly difficult. You also don't seem to accept the consequences of recursive self-improvement. May I ask why?
2timtyler10yI was not "assuming" - I said "if"! The issue needs evidence - and the idea that an unpleasant machine intelligence is easy to build is not - in itself - good quality evidence. It is easier to build many things that don't work properly. A pile of scrap metal is easier to build than a working car - but that doesn't imply that automotive engineers produce piles of scrap. The first manned moon rocket had many safety features - and in fact worked successfully the very first time - and then only a tiny handful of lives were at stake. If the claim is that safety features are likely to be seriously neglected, then one has to ask what reasoning supports that. The fact that nice agents are a small point in the search space is extremely feeble evidence on the issue. "The consequences of recursive self-improvement" seems too vague and nebulous to respond to. Which consequences. I have written a fair bit about self-improving systems. You can see some of my views on: []

I've heard many times here that Gargoyles involved some interesting multilevel plots, but the first few episodes had nothing like it, just standard Disneyishness. Anyone recommendations which episodes are best of the series, so I can check them out without going through the boring parts?

We don't actually disagree; I was envisioning lots of study groups (hopefully including many that meet in person), and you're obviously correct that computers wouldn't be able to grade anything too complicated. I just didn't communicate this effectively, since I was pressed for time.

I think it's important, if you're doing in-person study groups, that each student should have to answer questions in front of the rest of the class -- put them on the spot, both to wake them up and as an incentive to study so they don't look bad.

Here's a sketch of how a colleg... (read more)

1NancyLebovitz10yWhat's a good level of challenge for some would lead to paralyzing anxiety for others. One advantage to a mostly online system is that students can choose classes with policies that suit the way they learn.

Did you notice that, as phrased in the link, your bet is about the following event: "[at a certain point in time under a few conditions] it will be interesting to hear Eliezer's excuses"? Technically, all Eliezer will have to do to win the bet will be to write a boring excuse.

1Unknowns10yEliezer was the one who linked to that: the bet is about whether those conditions will be satisfied. Anyway, he has already promised (more or less) not to make excuses if I win.

Open source textbooks

I'm not sure if they're exactly open source-- what's in them is centrally controlled. However, they're at least free online.

Interestingly, now that you bring this up, I'm not at all certain that I'd be able to communicate especially effectively with a copy of myself. Probably better than with a randomly selected person, but perhaps not as well as I might hope.

What makes you reach that conclusion?

Any chance you and Eliezer could set a date on your bet? I'd like to import the 3 open bets to Prediction Book, but I need a specific date. (PB, rightly, doesn't do open-ended predictions.)

eg. perhaps 2100, well after many Singularitarians expect some sort of AI, and also well after both of your actuarial death dates.

I don't know what this bet is, and I don't see a link anywhere in your post.

I heard in a few places that a real neuron is nothing like a threshold unit, but more like a complete miniature computer. None of those places expanded on that, though. Could you?

7JanetK10yI am not sure that I understand the exact difference between a threshold unit and a miniature computer that you want to shine a light on. Below are some aspects that may be of use to you: 1. The whole surface of a neuron is not at the same potential (relative to the firing potential). Synapses are many (can be thousands), along branching dendrites, of different types and strengths so that the patterns of input that will cause firing from the dendrite end of the neuron are varied and a large number. This architecture is plastic with synapses being strengthened and weakened by use and by events far away from the neuron. At the axon root synapses are fewer and their effect more individual. If the neuron fires it delivers its signal to many synapses on many other neurons. Again this is plastic. 2. The potential of the neuron surface is affected by many other things besides other neurons through the synapses. It is affected by electrical and magnetic fields generated by the whole brain (the famous waves etc.). It is affected by the chemical environment such as the concentration of calcium ions. These factors are very changeable depending on the activity of the whole brain, hormone levels, general metabolism etc. 3. It is fairly easy to draw possible configurations of 1 or 2 neurons that mimic all the logic gates, discriminators (like a long tailed pairs in old fashioned electronics), delay lines, simple memory and so on. But it is unlikely that this is the root to understanding how neurons act. Networks, parallel feedback loops, glial communities, modules, architecture of different parts of the brain and the like are probably a better level of investigation. I hope this is of some help.

Suppose that inventing a recursively self improving AI is tantamount to solving a grand mathematical problem, similar in difficulty to the Riemann hypothesis, etc. Let's call it the RSI theorem.

This theorem would then constitute the primary obstacle in the development of a "true" strong AI. Other AI systems could be developed, for example, by simulating a human brain at 10,000x speed, but these sorts of systems would not capture the spirit (or capability) of a truly recursively self-improving super intelligence.

Do you disagree? Or, how likely is this scenario, and what are the consequences? How hard would the "RSI theorem" be?

2JoshuaZ10yThis seems like a bad analogy. If you could simulate a group of smart human going at 10,000 times normal speed, say copies of Steven Chu or of Terry Tao, I'd expect that they'd be able to figure out how to self-improve pretty quickly. In about 2 months they have had about 5000 years worth of time to think about things. The human brain isn't a great structure for recursive self-improvement (while some aspects are highly modular, other aspects are very much not so) but given enough time one could work on improving that architecture.

You are thinking about radio communication much more than I was. My thought process centered on encrypted use of the internet to allow insurgents to communicate both with each other and with sympathizers or allies that are elsewhere. I agree with your analysis that crypto is unlikely to do much in terms of radio communications.

The difficulty of creating an AGI drops slightly every time computational power increases. We know that people greatly underestimated the difficulty of creating AGI in the past, but we don't know how fast the difficulty is decreasing, how difficult it is now, whether it will ever stop decreasing, or where.

0PeterS10yI agree that those rates are hard to determine. I am also weary of "AI FOOM is a certainty" type statements, and appeals to the nebulous "powers that all computers inherently have".