It happens every now and then that someone encounters some of my transhumanist-side beliefs—as opposed to my ideas having to do with human rationality—strange, exotic-sounding ideas like superintelligence and Friendly AI. And the one rejects them.

If the one is called upon to explain the rejection, not uncommonly the one says, “Why should I believe anything Yudkowsky says? He doesn’t have a PhD!”

And occasionally someone else, hearing, says, “Oh, you should get a PhD, so that people will listen to you.” Or this advice may even be offered by the same one who expressed disbelief, saying, “Come back when you have a PhD.”

Now, there are good and bad reasons to get a PhD. This is one of the bad ones.

There are many reasons why someone might actually have an initial adverse reaction to transhumanist theses. Most are matters of pattern recognition, rather than verbal thought: the thesis calls to mind an associated category like “strange weird idea” or “science fiction” or “end-of-the-world cult” or “overenthusiastic youth.”1 Immediately, at the speed of perception, the idea is rejected.

If someone afterward says, “Why not?” this launches a search for justification, but the search won’t necessarily hit on the true reason. By “‘true reason,” I don’t mean the best reason that could be offered. Rather, I mean whichever causes were decisive as a matter of historical fact, at the very first moment the rejection occurred.

Instead, the search for justification hits on the justifying-sounding fact, “This speaker does not have a PhD.” But I also don’t have a PhD when I talk about human rationality, so why is the same objection not raised there?

More to the point, if I had a PhD, people would not treat this as a decisive factor indicating that they ought to believe everything I say. Rather, the same initial rejection would occur, for the same reasons; and the search for justification, afterward, would terminate at a different stopping point.

They would say, “Why should I believe you? You’re just some guy with a PhD! There are lots of those. Come back when you’re well-known in your field and tenured at a major university.”

But do people actually believe arbitrary professors at Harvard who say weird things? Of course not.

If you’re saying things that sound wrong to a novice, as opposed to just rattling off magical-sounding technobabble about leptical quark braids in N + 2 dimensions; and if the hearer is a stranger, unfamiliar with you personally and unfamiliar with the subject matter of your field; then I suspect that the point at which the average person will actually start to grant credence overriding their initial impression, purely because of academic credentials, is somewhere around the Nobel Laureate level. If that. Roughly, you need whatever level of academic credential qualifies as “beyond the mundane.”

This is more or less what happened to Eric Drexler, as far as I can tell. He presented his vision of nanotechnology, and people said, “Where are the technical details?” or “Come back when you have a PhD!” And Eric Drexler spent six years writing up technical details and got his PhD under Marvin Minsky for doing it. And Nanosystems is a great book. But did the same people who said, “Come back when you have a PhD,” actually change their minds at all about molecular nanotechnology? Not so far as I ever heard.

This might be an important thing for young businesses and new-minted consultants to keep in mind—that what your failed prospects tell you is the reason for rejection may not make the real difference; and you should ponder that carefully before spending huge efforts. If the venture capitalist says, “If only your sales were growing a little faster!” or if the potential customer says, “It seems good, but you don’t have feature X,” that may not be the true rejection. Fixing it may, or may not, change anything.

And it would also be something to keep in mind during disagreements. Robin Hanson and I share a belief that two rationalists should not agree to disagree: they should not have common knowledge of epistemic disagreement unless something is very wrong.2

I suspect that, in general, if two rationalists set out to resolve a disagreement that persisted past the first exchange, they should expect to find that the true sources of the disagreement are either hard to communicate, or hard to expose. E.g.:

  • Uncommon, but well-supported, scientific knowledge or math;
  • Long inferential distances;
  • Hard-to-verbalize intuitions, perhaps stemming from specific visualizations;
  • Zeitgeists inherited from a profession (that may have good reason for it);
  • Patterns perceptually recognized from experience;
  • Sheer habits of thought;
  • Emotional commitments to believing in a particular outcome;
  • Fear that a past mistake could be disproved;
  • Deep self-deception for the sake of pride or other personal benefits.

If the matter were one in which all the true rejections could be easily laid on the table, the disagreement would probably be so straightforward to resolve that it would never have lasted past the first meeting.

“Is this my true rejection?” is something that both disagreers should surely be asking themselves, to make things easier on the other person. However, attempts to directly, publicly psychoanalyze the other may cause the conversation to degenerate very fast, from what I’ve seen.

Still—“Is that your true rejection?” should be fair game for Disagreers to humbly ask, if there’s any productive way to pursue that sub-issue. Maybe the rule could be that you can openly ask, “Is that simple straightforward-sounding reason your true rejection, or does it come from intuition-X or professional-zeitgeist-Y ?” While the more embarrassing possibilities lower on the table are left to the Other’s conscience, as their own responsibility to handle.

1See “Science as Attire” in Map and Territory.

2See Hal Finney, “Agreeing to Agree,” Overcoming Bias (blog), 2006, http://www.overcomingbias.com/2006/12/agreeing_to_agr.html.

85

100 comments, sorted by Highlighting new comments since Today at 10:47 AM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

There need not be just one "true objection"; there can be many factors that together lead to an estimate. Whether you have a Ph.D., and whether folks with Ph.D. have reviewed your claims, and what they say, can certainly be relevant. Also remember that you should care lots more about the opinions of experts that could build on and endorse your work, than about average Joe opinions. Very few things ever convince average folks of anything unusual; target a narrower audience.

-8epigeios9y
1buybuydandavis8yAnd more than one of them may be considered decisive by themselves.

Immediate association: pick-up artists know well that when a girl rejects you, she often doesn't know the true reason and has to deceive herself. You could recruit some rationalists among PUAs. They wholeheartedly share your sentiment that "rational agents must WIN", and have accumulated many cynical but useful insights about human mating behaviour.

Most transhumanist ideas fall under the category of "not even wrong." Drexler's Nanosystems is ignored because it's a work of "speculative engineering" that doesn't address any of the questions a chemist would pose (i.e., regarding synthesis). It's a non-event. It shows that you can make fancy molecular structures under certain computational models. SI is similar. What do you expect a scientist to say about SI? Sure, they can't disprove the notion, but there's nothing for them to discuss either. The transhumanist community has a tendency to argue for its positions along the lines of "you can't prove this isn't possible" which is completely uninteresting from a practical viewpoint.

If I was going to depack "you should get a PhD" I'd say the intention is along the lines of: you should attempt to tackle something tractable before you start speculating on Big Ideas. If you had a PhD, maybe you'd be more cautious. If you had a PhD, maybe you'd be able to step outside the incestuous milieu of pop sci musings you find yourself trapped in. There's two things you get from a formal education: one is broad, you're exposed to a variety of subject matter t... (read more)

"There's two things you get from a formal education: one is broad, you're exposed to a variety of subject matter that you're unlikely to encounter as an autodidact;"

As someone who has a Ph.D., I have to disagree here. Most of my own breadth of knowledge has come from pursuing topics on my own initiative outside of the classroom, simply because they interested me or because they seemed likely to help me solve some problem I was working on. In fact, as a grad student, most of the things I needed to learn weren't being taught in any of the classes available to me.

The choice isn't between being an autodidact or getting a Ph.D.; I don't think you can really earn the latter unless you have the skills of the former.

7RichardKennaway9yBut being a grad student gave you the need to learn them.

If you had a PhD, maybe you'd be able to step outside the incestuous milieu of pop sci musings you find yourself trapped in.

That sounds like it's less "Once you get a Ph.D., I'll believe you," than "Once you get a Ph.D., you'll stop believing that."

Of course, those aren't so different: if I expect that getting a Ph. D would make one less likely to believe X, then believing X after getting a Ph.D is a stronger signal than simply believing X.

4capybaralet6yWhich transhumanist ideas are "not even wrong"? And do you mean simply 'not well specified enough'? Or more like 'unfalsifiable'? You also seem to be implying that scientists cannot discuss topics outside of their field, or even outside its current reach. My philosophy on language is that people can generally discuss anything. For any words that we have heard (and indeed, many we haven't), we have some clues as to their meaning, e.g. based on the context in which they've been used and similarity to other words. Also, would you consider being cautious an inherently good thing? Finally, from my experience as a Masters student in AI, many people are happy to give opinions on transhumanism, it's just that many of those opinions are negative.
1Mader_Levap5y"Which transhumanist ideas are "not even wrong"?" Technological Singularity, for example (as defined in Wikipedia). In my view, it is just atheistic version of Rapture or The End Of World As We Know It endemic in various cults and equally likely. Reason for that is that recursive self-improvement is not possible, since it requires perfect self-knowledge and self-understanding. In reality, AI will be black box to itself, like our brains are black boxes to ourself. More precisely, my claim is that any mind on any level of complexity is insuficient to understand itself. It is possible for more advanced mind to understand simpler mind, but it obviously does not help very much in context of direct self-improvement. AI with any self-preservation instincts would be as likely to willingly preform direct self-modification to its mind as you to get stabbed by icepick through eyesocket. So any AI improvement would have to be done old way. Slow way. No fast takeoff. No intelligence explosion. No Singularity.
5gjm5yOur brains are mysterious to us not simply because they're our brains and no one can fully understand themselves, but because our brains are the result of millions of years of evolutionary kludges and because they're made out of hard-to-probe meat. We are baffled by chimpanzee brains or even rabbit brains in many of the same ways as we're baffled by human brains. Imagine an intelligent agent whose thinking machinery is designed differently from ours. It's cleanly and explicitly divided into modules. It comes with source code and comments and documentation and even, in some cases, correctness proofs. Maybe there are some mysterious black boxes; they come with labels saying "Mysterious Black Box #115. Neural network trained to do X. Empirically appears to do X reliably. Other components assume only that it does X within such-and-such parameters.". Its hardware is made out of (notionally) discrete components with precise specifications, and comes with some analysis to show that if the low-level components meet the spec then the overall function of the hardware should be as documented. Suppose that's your brain. You might, I guess, be reluctant to experiment on it in any way in place, but you might feel quite comfortable changing EXPLICIT_FACT_STORAGE_SIZE from 4GB to 8GB, or reimplementing the hardware on a new semiconductor substrate you've designed that lets every component run at twice the speed while remaining within the appropriately-scaled specifications, and making a new instance. If it causes disaster, you can probably tell; if not, you've got a New Smarter You up and running. Of course, maybe you couldn't tell if some such change caused disasters of a sufficiently subtle kind. That's a reasonable concern. But this isn't an ice-pick-through-the-eye-socket sort of concern, and it isn't the sort of concern that makes it obvious that "recursive self-improvement is not possible".
1Lumifer5yWhile I agree with the overall thrust of your comment, this brought to mind an old anecdote [http://catb.org/jargon/html/magic-story.html]...
4gjm5ySuch things are why I said "maybe you couldn't tell if some such change caused disasters of a sufficiently subtle kind".

Vladimir, I don't quite think that's the "narrower audience" Robin is talking about...

Robin, see the Post Scriptum. I would be willing to get a PhD thesis if it went by the old rules and the old meaning of "Prove you can make an original, significant contribution to human knowledge and that you've mastered an existing field", rather than, "This credential shows you have spent X number of years in a building." (This particular theory would be hard enough to write up that I may not get around to it if a PhD credential isn't at stake.)

Eliezer,

See poke's comment above (which is so on the nose, it actually inspired me to register). Then consider the following.

You will never get a PhD in the manner you propose, because that would fulfill only a part of the purpose of a PhD. The number of years spent in the building can be (and in too many cases is) wasted time - but if things are done in a proper manner, this time (which can be only three or four years) is critical.

For science PhDs specifically, the idea isn't to just come up with something novel and write it up. The idea is to go into the field with a question that you don't have an answer for, not yet. To find ways to collect data, and then to actually collect it. To build intricate, detailed models that answer your question precisely and completely, fitting all the available data. To design experiments specifically so you can test your models. And finally, to watch these models completely and utterly fail, nine times out of ten.

They won't fail because you missed something while building them. They will fail because you could only test them properly after making them. If you just built the model that fit everything, and then never tested it with specific experim... (read more)

As a current grad student myself, I could not disagree with poke's comment and this comment more. I work for a very respected adviser in computer vision from a very prestigious university. The reason I was accepted to this lab is because I am an NDSEG fellow. Many other qualified people lost out because my attendance here frees up a lot of my adviser's money for more students. In the mean time, I have a lot of pretty worthwhile ideas in physical vision and theories of semantic visual representations. However, I spend most of my days building Python GUI widgets for a group of collaborating sociologists. They collect really mundane data by annotating videos and no off the shelf stuff does quite what they want... so guess who gets to do that grunt work for a summer? Grad students.

You should really read the good Economist article The Disposable Academic. Graduate studentships are business acquisitions in all but the utmost theoretical fields. Advisers want the most non-linear things imaginable. For example, I am a pure math guy, with heavy emphasis on machine learning methods and probability theory. Yet my day job is seriously creative-energy-draining Python programming. The programmin... (read more)

5kalla7249yOk, so - I hear what you're saying, but a) that is not the way it's supposed to be, and b) you are missing the point. First, a), even in the current academia, you are in a bad position. If I were you, I would switch mentors or programs ASAP. I understand where you're coming from perfectly. I had a very similar experience: I spent three years in a failed PhD (the lab I was working in went under at the same time as the department I was in), and I ended up getting a MS instead. But even in that position, which was all tedious gruntwork, I understood the hypothesis and had some input. I switched to a different field, and a different mentor, where most of my work was still tedious, but it was driven by ideas I came to while working with my adviser. If your position is, as it seems to be, even worse - that you have NO input whatsoever, and are purely cheap labor - then you should switch mentors immediately. If you don't, you might finish your PhD with a great deal of bitterness, but it is much more likely that you will simply burn out and drop out. Which brings me to b). As I said above, it would be pointless for Eliezer to go to grad school now. Even at best, it contains a lot of tedious, repetitive work. But the essential point stands: in a poorly constrained area such as transhumanism, grand ideas are not enough. That is where PhD does have a function, and does have a reason.

Actually, my mentor is among one of the nicest guys around and is a good manager, offers good advice, and has a consistent record of producing successful students. It's just that almost no grad student gets to have real input in what they are doing. If you do have that, consider yourself lucky, because the dozens of grad students that I know aren't in a position like that. I just had a meeting today where my adviser talked to me about having to balance my time between "whatever needs doing" (for the utility of our whole research group rather than just my own dissertation) and doing my own reading/research. His idea (shared by many faculty members) is that for a few years at the front end of the PhD, you mostly do about 80% general utility work and infrastructure work, just to build experience, write code, get involved... then after you get into some publications a few years later, the roles switch and you shift to more like 80% writing and doing your own thing (research). The problem is that if you're a passionate student with good ideas, then that first few years of bullshit infrastructure work is a complete waste of time. The run-of-the-mill PhD student (who generally i... (read more)

-2[anonymous]6yhttp://www.caseyresearch.com/cdd/doug-casey-education [http://www.caseyresearch.com/cdd/doug-casey-education]

Robin: Of course a PhD in "The Voodoo Sciences" isn't going to help convince anybody competent of much. I am actually more impressed with some of the fiction I vaguely remember you writing for Pournelle's "Endless Frontier" collections than a lot of what I've read recently here.

Poke: "formal education: one is broad, you're exposed to a variety of subject matter that you're unlikely to encounter as an autodidact"

I used to spend a lot of time around the Engineering Library at the University of Maryland, College Park before I mo... (read more)

Perhaps you are marginally ahead of your time Eliezer, and the young individuals that will flush out the theory are still traipsing about in diapers. In which case, either being a billionare or a phD makes it more likely you can become their mentor. I'd do the former if you have a choice.

Can't do basic derivatives? Seriously?!? I'm for kicking the troll out. His bragging about mediocre mathematical accomplishments isn't informative or entertaining to us readers.

billswift wrote:

…but the self-taught will simply extend their knowledge when a lack appears to them.

Yes, this point is key to the topic at hand, as well as to the problem of meaningful growth of any intelligent agent, regardless of its substrate and facility for (recursive) improvement. But in this particular forum, due to the particular biases which tend to predominate among those whose very nature tends to enforce relatively narrow (albeit deep) scope of interaction, the emphasis should be not on "will simply extend" but on "when a lack a... (read more)

Eliezer, I'm sure if you complete your friendly AI design, there will be multiple honorary PhDs to follow.

Sorry about the length of the post, there was just a lot to say.

I believe disagreements are easier to unpack if we stop presuming they are about difference in belief. Posts like this seem to confirm my own experience that the strongest factor in convincing people of something is not any notion of truth or plausibility but whether there are common allegiances with the other side. This seems to explain a number of puzzles of disagreement, including: (list incomplete to save space)

  • Why do people who aren't sure about Elizer's posts about physics/comp science/b
... (read more)

I have spent years in the Amazon Basin perfecting the art of run-on sentences and hubris it helps remind others of my shining intellect it also helps me find attractive women who love the smell of rich leather furnishings and old books.

Between bedding supermodels a new one each night, I have developed a scientific thesis that supersedes your talk of Solomonoff and Kolmogorov and any other Russian name you can throw at me. Here are a random snippet of conclusions a supposedly intelligent person will arrive having been graced by my mathematical superpowers:

  1. Everything you thought you knew about Probability is wrong.
  2. Existence is MADE of Existence.
  3. Einstien didn't know this, but slowly struggled toward my genius insight.
  4. They mocked me when I called myself a modern day Galileo, but like Bean I will come back after they have gone soft.

I can off the tip of my rather distinguished salt-and-pepper beard name at least 108 other conclusions that would startle lesser minds such as the John BAEZ the very devil himself or Adolf Hitler I have really lost my patience with you ElIzer.

They called me mad when I reinvented calculus! They will call me mad no longer oh I have to go make the Sweaty Wildebeest with a delicately frowning Victoria's Secret model.

5[anonymous]9ytime cube!
4SpaceFrank9yI wish I could upvote this post back into the positive. (It seems pretty obvious to me that is a direct satire of the previous post by a similar username. What, no love for sarcasm?)
7[anonymous]9yPoe's Law, anyone?
2SpaceFrank9yI had to look it up, but I definitely agree. Especially considering how quickly the karma changes reversed after I edited in that footnote.
3DanielLC9yI thought it was obvious. Maybe I'm just spoiled by the consistently good comments on LessWrong, and don't realize that there actually are comments that bad and cliched.
2[anonymous]9yGiven that particular misspelling of Einstein and the mention of Baez, it was nearly impossible that BrandNameThinker hadn't heard of the crackpot index... But what's obvious for me (and you) needn't be obvious for someone else. (Or maybe the downvoters did get the joke but just didn't find it funny.)

Crap. Will the moderator delete posts like that one, which appear to be so off the Mark?

Eliezer - 'I would be willing to get a PhD thesis if it went by the old rules and the old meaning of "Prove you can make an original, significant contribution to human knowledge and that you've mastered an existing field", rather than, "This credential shows you have spent X number of years in a building."'

British and Australasian universities don't require any coursework for their PhDs, just the thesis. If you think your work is good enough, write to Alan Hajek at ANU and see if he'd be willing to give it a look.

Ignoring the highly unlikely slurs about your calculus ability:

However, if any professor out there wants to let me come in and just do a PhD in analytic philosophy - just write the thesis and defend it - then I have, for my own use, worked out a general and mathematically elegant theory of Newcomblike decision problems. I think it would make a fine PhD thesis, and it is ready to be written - if anyone has the power to let me do things the old-fashioned way.

British universities? That's the traditional place to do that sort of thing. Oxbridge.

Specifically with regard to the apparent persistent disagreement between you and Robin, none of those things explain it. You guys could just take turns doing nothing but calling out your estimates on the issue in question (for example, the probability of a hard takeoff AI this century), and you should reach agreement within a few rounds. The actual reasoning behind your opinions has no bearing whatsoever on your ability to reach agreement (or more precisely, on your inability to maintain disagreement).

Now, this is assuming that you both are honest and rati... (read more)

And with that lovely exhibition of math talent, combined with the assertion that he skipped straight to grad school in mathematics, I do hereby request GenericThinker to cease and desist from further commenting on Overcoming Bias.

Generic,

The y appears on both sides of the equation, so these are differential equations. To avoid confusion, re-write as:

(1) (d/dt) F(t) = A*F(t) (2) (d/dt) F(t) = e^F(t)

Now plug e^At into (1) and -ln(C-t) into (2), and verify that they satisfy the condition.

You could recruit some rationalists among PUAs. They wholeheartedly share your sentiment that "rational agents must WIN"

Interesting. As a reasonable approximation, approaching women with confidence==one-boxing on Newcomb's problem. Eliezer's posts have increased my credence that the latter is correct, although it hasn't helped me with the former.

@Brian

I think Alec Greven may be your man. Or perhaps like Lucy van Pelt I should set up office hours offering Love Advice, 5 cents?

You could recruit some rationalists among PUAs. They wholeheartedly share your sentiment that "rational agents must WIN"

You have. We do. And yes, they must.

"Drexler's Nanosystems is ignored because it's a work of "speculative engineering" that doesn't address any of the questions a chemist would pose (i.e., regarding synthesis)."

It doesn't address any of the questions a chemist would pose after reading Nanosystems.

"As a reasonable approximation, approaching women with confidence==one-boxing on Newcomb's problem."

Interesting. Although I would say "approaching women with confidence is an instance of a class of problems that Newcomb's problem is supposed to represent but does ... (read more)

1Broggly10yIt does? Assuming a deterministic universe (which would seem to be necessary for Omega to predict with 100% certainty whether someone 1-boxes or 2-boxes) then it isn't the act of taking one or two boxes that causes the $1M to be present or absent, but rather both those events share a common cause (namely whatever circumstances were present to cause the subject to pick their Newcombe strategy). If Omega can't predict with certainty then two-boxing is possibly the right answer, depending on exactly how good Omega is at "reading" people and how much money is potentially in the boxes. If this really happened to me, I've got no idea how I would decide. I would probably think some kind of trick was going on and after checking all the fine print for loopholes and one-box the million with some kind of witnesses around to grab the old man behind the curtain and have him charged with fraud if the money wasn't there. If he had some legal escape hatch I could spot, then I'd probably one-box the $1k since I'd think it unlikely someone would give up $999,000 if they could get away with it. Agree that picking up women isn't newcombelike, though. I get how being nervous can be like 1-boxing the $1k, and being honestly confident might be like one-boxing the $1M (maybe?) but I have no idea what corresponds to 2-boxing.

Daniel, I knew it :-)

Phil, you can look at it another way: the commonality is that to win you have to make yourself believe a demonstrably false statement.

"However, if any professor out there wants to let me come in and just do a PhD in analytic philosophy - just write the thesis and defend it - then I have, for my own use, worked out a general and mathematically elegant theory of Newcomblike decision problems. I think it would make a fine PhD thesis, and it is ready to be written - if anyone has the power to let me do things the old-fashioned way."

I think this is a good idea for you. But don't be surprised if finding the right one takes more work than an occasional bleg. And I do recommend getting it at Harvard or the equivalent. And if I'm not mistaken, you may still have to do a bachelors and masters?

2apophenia10yMy university does not require a masters' degree to get a PhD.

If I have to do a bachelors degree, I expect that I can pick up an accredited degree quickly at that university that lets you test out of everything (I think it's called University of Phoenix these days?). No Masters, though, unless there's an org that will let me test out of that.

The rule of thumb here is pretty simple: I'm happy to take tests, I'm not willing to sit in a building for two years solely in order to get a piece of paper which indicates primarily that I sat in a building for two years.

1irrational7yIf you don't have a bachelor's degree, that makes it rather unlikely that you could get a PhD. I agree with folks that you shouldn't bother - if you are right, you'll get your honorary degrees and Nobel prizes, and if not, then not. (I know I am replying to a five-year-old comment). I also think you are too quick to dismiss the point of getting these degrees, since you in fact have no experience in what that involves.
-3[anonymous]6yThat's what John Gilmore did, among other cool things.http://www.toad.com/gnu/ [http://www.toad.com/gnu/]http://papersplease.org/id.html [http://papersplease.org/id.html] John Gilmore writes about his cut-rate degree, here: http://reason.com/archives/2005/04/01/letters [http://reason.com/archives/2005/04/01/letters] I would really like it if EY had the ability (money, engineering team, etc.) to begin producing brain-like hardware, replete with mirror neurons. I think that's the only way that the sociopaths (prosecutors, judges, politicians, bureaucrats, police) in the coercive sector will be challenged in a meaningful way. I think that it might be smarter for him to relocate MIRI to South Korea, because there's more of a culture of robotics there, and robotics is necessary for feedback regarding real-world problems. These desires of mine aren't tyrannical. I wouldn't cling to them or try to prescribe actions for EY if he didn't share the same desires. I'm just stating what I would do, if I were suddenly to occupy a leadership position at MIRI, or some similarly-capable organization. In many ways, this is a deep decision that is based on difficult to quantify innate value judgments. Hawkins was fascinated with brains, and logically, pursued brain design because attempts at getting intelligent, brainlike responses with technology have been so weak in the past, even given approximately adequate computing power. Deep-learning done by Schmidhuber has also recently been productive, given that all computational hardware is enabling far more intelligence, even from systems that were not optimal. This leads me to believe there will be "many kinds of minds" in the coming singularity. Some, of course, will be superior to others in terms of ability to restructure their environments. Let's hope they aren't sociopathic, or "coercive-human-directed." Remember, even intelligent people can act as sociopaths given good intentions, but the wrong (coercive collectivist) ideas.

Phil, you can look at it another way: the commonality is that to win you have to make yourself believe a demonstrably false statement.
But I don't. The problem, phrased in a real world situation that could possibly occur, is that a superintelligence is somehow figuring out what people are likely to do, or else is very lucky. The real-world solution is either

  1. if you know ahead of time that you're going to be given this decision, either pre-commit to one-boxing, or try to game the superintelligence. Neither option is irrational; it doesn't take any fancy

... (read more)

Phil, your commitment ahead of time is your own private business, your own cognitive ritual. What you need in order to determine the past in the right way is that you are known to perform a certain action in the end. Whether you are arranging it so that you'll perform that action by making a prior commitment and then having to choose the actions because of the penalty, or simply following a timeless decision theory, so that you don't need to bother with prior commitments outside of your cognitive algorithm, is irrelevant. If you are known to follow timeles... (read more)

Vladimir, I understand the PD and similar cases. I'm just saying that the Newcomb paradox is not actually a member of that class. Any agent faced with either version - being told ahead of time that they will face the Predictor, or being told only once the boxes are on the ground - has a simple choice to make; there's no paradox and no PD-like situation. It's a puzzle only if you believe that there really is backwards causality.

Phil, you said "if you didn't know ahead of time that you'd be given this decision, choose both boxes", which is a wrong answer. You didn't know, but the predictor knew what you'll do, and if you one-box, that is your property that predictor knew, and you'll have your reward as a result.

The important part is what predictor knows about your action, not even what you yourself know about your action, and it doesn't matter how you convince the predictor. If predictor just calculates your final action by physical simulation or whatnot, you don't need ... (read more)

"You didn't know, but the predictor knew what you'll do, and if you one-box, that is your property that predictor knew, and you'll have your reward as a result."

No. That makes sense only if you believe that causality can work backwards. It can't.

"If predictor can verify that you'll one-box (after you understand the rules of the game, yadda yadda), your property of one-boxing is communicated, and it's all it takes."

Your property of one-boxing can't be communicated backwards in time.

We could get bogged down in discussions of free will; ... (read more)

0Broggly10yPresumably the Predictor would be smart enough to calculate the result of that coin flip. If it was an actually random bit, then I don't know. In the real universe, as you require, then the Predictor would have a 50% certainty of being right. Probably if the Predictor thought you might do that it wouldn't offer you the challenge, in order to maintain its reputation for omniscience.

Compare: communicating the property of the timer that it will ring one hour in the future (that is, timer works according to certain principles that result in it ringing in the future) vs. communicating from the future the fact that timer ringed. If you can run a precise physical simulation of a coin, you can predict how it'll land. Usually, you can't do that. Not every difficult-seeming prediction requires things like simulation of physical laws, abstractions can be very powerful as well.

Vladimir, I don't mean to diss you; but I am running out of weekend, and think it's better for me to not reply than to reply carelessly. I don't think I can do much more than repeat myself anyway.

One boxing because of a lack of precommitment is a mistake. Backwards causality is irrelevant. Prediction based off psychological or physical simulation is sufficient.

Gaming a superintelligience with dice acheives little. You're here to make money not prove him wrong. Expect him to either give you a probabilistic payoff or count a probabilistic decision as two boxing. Giving pedentic answers requires a more formal description, it doesn't change anything.

If I'm ever stuck in a prison with a rational, competitive fellow prisoner, it'd be really damn handy to be omniscient and have my my buddy know it.

I may be wrong about Newcomb's paradox.

You could say that embracing timeless decision theory is a global meta-commitment, that makes you act as if you made commitment in all the situations where you benefit from having made the commitment.
I think this is correct.

It's perplexing: This seems like a logic problem, and I expect to make progress on logic problems using logic. I would expect reading an explanation to be more helpful than having my subconscious mull over a logic problem. But instead, the first time I read it, I couldn't understand it properly because I was not framing the problem p... (read more)

I don't think it did help, though. I think I failed to comprehend it. I didn't file it away and think about it; I completely missed the point. Later, my subconscious somehow changed gears so that I was able to go back and comprehend it. But communication failed.

Buddhists say that great truths can't be communicated; they have to be experienced, only after which you can understand the communication. This was something like that. Discouraging.

From my experience, the most productive way to solve a problem on which I'm stuck (that is, hours of looking at it produce no new insight or promising directions of future investigation), is to keep it in the background for long time, while avoiding forgetting it by recalling what's it about and visualizing its different aspects and related conjectures from time to time. And sure enough, in a few days or weeks, triggered by some essentially unrelated cue, a little insight comes, that allows to develop a new line of thought. When there are several such problem in the background, it's more or less efficient.

Inferential distance can make communication a problem worthy of this kind of reflectively intractable insight.

Phil - Changing your mind on previous public commitments is hard work. Respect!

It's a fascinating problem. I'm hoping Eleizer gets a chance to write that thesis of his. It's even more interesting once you see people applying newcomelike reasoning behaviorally. A whole lot more of human behavior started making sense after I grasped the newcome problem.

Phil, I think that's how logic (or math) normally works. You make progress on logic problems by using logic, but understanding another's solution usually feels completely different to me, completely binary.

Also, it's hard to say that your unconscious wasn't working on it. In particular, I don't know if communicating logic to me is as binary as it feels, whether I go through a search of complete dead ends, or whether intermediate progress is made but not reported.

Going back to this post, a lot of things that puzzled us then are way more obvious now. But one angle remained unexplored for some reason. Here it is: if people catch on that you got a PhD just to persuade them, your PhD won't help you persuade them. As Robin said, people often don't have "true rejections" on the object level because they don't understand the object level. Instead they feel (correctly) that controversial scientific arguments should not be sold directly to the public, and apply multiple heuristics on the meta level. And the positi... (read more)

Still - "Is that your true rejection?" should be fair game for Disagreers to humbly ask, if there's any productive way to pursue that sub-issue.

Perhaps it should, but the problem is that answering this question is one of the big problems in salesmanship: working out the customer's true obstacle to wanting to buy from you. Salesmen would love to be able to get a true answer to this question - and some even ask it directly - but people tend to receive this as manipulation: finding out their inner thoughts for purposes of getting their money. Thi... (read more)

Here's something I'd love to put into an entire article, but can't because my karma's bad (see my other comment on this thread):

Many people make the false assumption that the scientific method starts with the hypothesis. They think: first hypothesize, then observe, then make a theory from the collection of hypotheses.

The reality is quite the opposite. The first point on the scientific method is the observation. Any hypotheses before observation will only diminish the pool of possible observations. Second is building a theory. Along the process, many t... (read more)

1Zackmarty9yThis isn't related to the entire post so much as it is a response to the problem with the scientific method. The scientific method does start with a hypothesis in relation to individual experiments. The hypothesis starts with general observations made from previous experiments or just some kind of general observation. If we assume complete ignorance about NaCl (Sodium Chloride), but previously observed that pure sodium (Na) explodes in the presence of water, we might decide to devise an experiment to see what happens when we place other compounds with sodium in water. Our hypothesis might be something like, everything that has sodium explodes in water. It is not necessary to write down this hypothesis, because the hypotheses we generally make are made in our head when we are not in some kind of laboratory setting. By that I mean we constantly make hypotheses about subjects before we observe the results of our 'experiments.' Anyway, we toss a block of sodium chloride in a bucket of water and observe that an explosion does not immediately follow. We had previous observations from former experiments, or general observations, to suggest that stuff with sodium will explode, but our observational evidence suggests that our previously made hypothesis is not true. We make plenty of observations before making hypotheses, but we always make some sort of hypothesis before making some sort of observation when starting an experiment of any manner.
1epigeios9yMost people don't even go as far as to make a hypothesis there. It's arguable that every time someone forms a question, it is the same as forming a hypothesis; purely because questions can be reworded as hypotheses. However, in the case of someone who is just exploring, they would not go so far as to hypothesize. It's normal for the person to say something like "I wonder if it happens with all sodium compounds" or, "I wonder if there are any sodium compounds that explode as well". But in these cases, there is no basis and no reason to form a hypothesis. one could argue that the person is making a hypothesis like "all sodium compounds explode in water"; but the person doesn't care. The person could just as easily make the hypothesis "no sodium compounds explode in water". And there's no reason to make either of these, or any hypothesis at all, because no theory has been formed that can be tested. And further, making a hypothesis like this limits the amount of new information that can come in from these experiments. The information is now limited to "whether or not the substance explodes", when there are plenty of other reactions that can happen. The person who makes this hypothesis is liable to miss small bubbles appearing. That is anti-desired when exploring, when trying to observe as much as possible so as to build a theory. The point is that the person in your example is not doing a hypothesis experiment, the person is doing an exploration experiment. Unless a theory exists, there's no basis for choosing any hypothesis at all. Yet, let's say then that the person discovered some cool stuff and started to build a theory. He wants to tell you about his in-progress theory. Obviously he hasn't done any hypothesis experiments, because hypotheses haven't mattered yet. He tells you about his observations, and his conjectures. Many people, in response to this, say "can you prove it? Why should I believe you?". To which he has no answer, because he has nothing to prove

i'd say: you don't have a phd, therfore you're not qualified to judge whether or not yudkowsky should have a phd.

0[anonymous]8yHow does getting a Ph. D. (even in a related field) give one the qualification to judge?
1halcyon8yit doesn't, it's a jocular reductio ad absurdum based on the irrationality of the underlying premise. 9_9 and what field are we talking here, education? phdology?
1[anonymous]8yIf we're commenting on the same article, a Ph. D. in (in decreaing order of relevance) AI, Machine Learning, CS, or mathematics...
3halcyon8ynaw, i'm talking about what field qualifies you to judge how much not having a phd disqualifies you from judging statements on that subject.

When I was in Sales, we called this "finding their true objection."

Basically, if someone says "Well, I don't want it unless it has X!" You say "What if I could provide you with X?"

So if someone says "Come back when you have a PhD!" You say "What if I could provide you with PhDs who believe the same idea?" If they then say "There are tons of PhDs who believe crazy things!" then you say "Then what else would I need to convince you?"

Usually, between them dismissing their own criteria and the amount of ideas they can bring forward, you can bring it down to about three things. I've seen 5, but that was a hard case. Those aren't hard and fast rules: the rule is make sure you get them ALL, and make it specific, something like:

"So, if I can get you a published book by a PhD, respected in a field relevant to X, AND I can provide you with a for-profit organization that is working to accomplish goals relevant to X, AND I can make a flower appear out of my ear (or whatever)" THEN you will admit you were wrong and change your view?

And if you're REALLY invested, you should have been taking notes, and get them to 'in... (read more)

1Kenny7yIt would be awesome to see a transcript of a back and forth between Eliezer and Robin, or any two people denating, where they both provided this info!
3Jiro7y"Come back when you have a PhD" typically doesn't mean "if you have a PhD, I'll believe you", it means "if you have a PhD, I'll assign a high enough probability to you having something worthwhile to say that it's worth even listening to you and doing the work to figure out where you might be wrong". Also, it can't be answered by "I could show you some PhDs who agree with me". As there are a lot of PhDs in the world, finding one out of that large number who agrees with you doesn't update the probability of being right by much compared to combining PhD and agreement in a specific person named in advance (such as yourself). Furthermore, human beings are neither unsafe genies nor stupid AIs, and nobody will take kindly to you trying to act like one by giving someone something that matches their literal request but any human being with common sense can figure out isn't what they meant. My request would probably be something like "come back when you have a PhD and get your observations published in a peer-reviewed journal".

The one thing that actually has seemed to raise credibility, is famous people associating with the organization, like Peter Thiel funding us, or Ray Kurzweil on the Board.

Well, yeah.

You spend a lot of your time worrying about how to get an AI to operate within the interests of lesser beings. You also seem to spend a certain amount of time laying out Schelling fences around "dark side tactics". It seems to me that these are closely related processes.

As you have said, "people are crazy, the world is mad". We are not operating with a po... (read more)

4Jiro6yI am not very impressed by that. "Would you change your mind if you were convinced of X" carries the connotation "if I managed to give you an argument for X, and you couldn't rebut it, would you change your mind?" The answer to that should be "no" for many values of X even if the answer to the original question is "yes". The fact that you couldn't rebut the argument may mean that it's true. It may also just mean the argument is full of holes but the person is really good at convincing you. How do you know that the person who convinced you of X isn't another case of Eliezer convincing you to let the AI out of a box? If a lot of scientists or other experts vetted the claim of such an X and it was not only personally convincing, but had a substantial following in the community of experts, then I might change my mind.
-2RPMcMurphy6y.
0Jiro6yThat seems to suggest you believe peer review is a bad idea. Is that true?

I'll believe you're a gold miner when I see your gold.

Numenta http://www.numenta.org is building smart software brains that simulate hardware brains, with the ultimate goal of building energy-inexpensive hardware brains. Vicarious is building smart hardware brains. Robobrain is building --and educating-- smart hardware brains. IBM is building smart hardware brains, with its Neuromorphic computing project. OpenCog is building a software system, which is very good, but of unknown (to me) competitiveness with the other projects. Honda is building smart hardw... (read more)

[This comment is no longer endorsed by its author]Reply

As the years go on. I'm glad to review this and appreciate that you understand this. You definitely have a group of people that love you more than like you, and it is somewhat disheartening to see these people so vehemently they insist on their rejection without even giving some modicum of chance.

I only get more motivation to put in my extraordinary effort and see in what ways I can help.

There are some views of Yudkowsky I don't necessarily agree with, and none of them have anything to do with him having or not having a PhD.

Are you sure this type of rejection (or excuse of a rejection) is common and significant?

1dxu6yWould you be more inclined to agree with him if he did in fact have a PhD (in the relevant fields)? If your (honest) answer to this question is "yes", then your rejection does have something to do with him not having a PhD. Based on personal experience, I would say so, yes.
3Jiro6yI'd be more inclined to agree with him if he was God, too, but I wouldn't say "my rejection of his ideas has something to do with the fact that he is not God". For that matter I would be more inclined to agree with him if he used mind control rays on me, but "I reject Eliezer's ideas partly because he isn't using a mind control ray on me" would be a ludicrous thing to say. "My rejection has to do with X" connotes more than just a Bayseian probability estimate.
1dxu6yBoth of your hypothetical statements are correct, and both of them would be bad reasons to believe something (well, I'm a bit fuzzy on the God hypothetical--is God defined as always correct?), just as the presence or absence of a PhD would be a bad reason to believe something. (This is not to say that PhD's offer zero evidence one way or the other, but rather that the evidence they offer is often overwhelmed by other, more immediate factors.) The phrasing of your comment gives me the impression that you're trying to express disagreement with me about something, but I can't detect any actual disagreement. Could you clarify your intent for me here? Thanks in advance.
1Jiro6yI disagree that if you would be less likely to reject his ideas if X were true, that can always be usefully described as "my rejection has something to do with X". The statement "my rejection has something to do with X" literally means "I would be less likely to reject it if X were true", but it does not connote that.
0dxu6yAll right, I can accept that. So what does it connote, by your reckoning?
1Jiro6yGenerally, that you would be less likely to reject it if X were true, and a couple of other ideas: * X is specific to your rejectiion--that is, that the truthfulness of X affects the probability of your rejection to a much larger degree than it affects the probability of other propositions that are conceptually distant. * The line of reasoning from X to reducing the probability of your rejection proceeds through certain types of connections, such as ones that are conceptually closer. * The effect of X is not too small, where "not to small" depends on how strongly the other factors apply. (Human beings, of course, have lots of fuzzy concepts.)
1Val6yAn interesting question, and not an easy one to answer in the way I could be sure you understood the same thing what I meant. My original thought when composing the comment was that it never occurred to me that "he doesn't have a PhD so his opinion is less worth", and I would never use the fact that he doesn't have a PhD, neither in an assumed debate with him nor for any self-assurance. This means that even if I answered with a "clear yes" to your question, it still wouldn't mean that it was the cause of the rejection. Loss of credit because he doesn't have a PhD does not necessarily equal the gain of credit because he does have a PhD. Yes, I realize that this is the most attackable sentence in my answer. The disagreements are more centered on personal opinion, morality, philosophical interpretation, opinion about culture and/or religion, and in part on interpreting history. So, mostly soft sciences. This means that there would be no relevant PhD in these cases (a PhD in a philosophical field wouldn't matter to me as a deciding factor) On the other hand, if it was a scientific field, then I might unconsciously have given him a little higher probability of being right if he did have a PhD in the field, but this would be dwarfed by the much larger probability gain caused by him actually working in the relevant field, PhD or not. As of yet I don't have any really opposite views about any of his scientific views. Maybe I hold some possible future events a little more or less probable, or am unsure in things he is very sure about, but the conclusion is: I don't have scientific disagreements with him as of yet. About whether this type of rejection is common: if we take my explanation at the beginning of this answer, then I guess it is uncommon to reject him in that way (and his article might lean a tiny little bit in the direction of a strawman argument: "they only disagree because they think it's important that I don't have a PhD, so this means they just don't have
0Val6yThe two comments you and Jiro wrote while composing mine actually made me think about a possibly unclear formulation: I should have written "are not based in any significant way on" instead of "don't have anything to do with". You actually just described what I wrote as: "Loss of credit because he doesn't have a PhD does not necessarily equal the gain of credit because he does have a PhD"

I think there's a slight misconception about Aumann's agreement theorem here: "Common knowledge", as Aumann defines it (and is leveraged in the proof), doesn't just mean exchanging beliefs once: common knowledge means 1) knowing how they update after the first pass, then 2) knowing how they updated after knowing about the first pass, 3) knowing about how they updated after knowing about updating about knowing about the first pass, and so on

It's only at the end of a potentially infinite chain of exchanging beliefs, that two rational agents are guaranteed to

... (read more)