The probability for anything is non-zero. But when we see something to which our hypothesis assigns a sufficiently infinitesimal probability, we call it falsified - because even maximum entropy does better.

A small caveat: the word 'evolution' doesn't have to refer to the scientific theory of biological evolution. The word existed long before the theory; otherwise, the theory would have become attached to a different word. Since the word itself means "incremental change over time," then it is perfectly appropriate to refer to a new computer chip design, or a corporate reorganization as evolution. Make your own guesses about whether something totally different, such as uploading a personality, can be called "evolution."

Science (unlike religion) has proved its myths

See, this is exactly the sort of thing I have a problem with. Science is not just magic that works. People are learning science as though it were merely a true religion: passwords, attire, professions, and all.

The press has reports with things like "scientists attack creationist teachings", but I've never seen "scientists attack common misconceptions about evolutionary theory".

Press selectivity. Trust me, they do.

I'd add... speech, empathy... superior social skills to the list

I wasn't aware that speech, empathy, and social skills were functions of the kidneys rather than the brain.

Gray Area, the objections you list are objections from within a model. This is right and proper. A lot of people don't reason the same way you do, though. Quick replies: (1) We know from sheerly physical considerations that you can build a brain at least a million times as fast as a human brain, which gives us many interesting results of itself. (2, 3) Barring a specific model of cognitive science one cannot disprove a magical upper bound which lies exactly above human intelligence, even though raw evolution encountered no apparent difficulty in accidentally building humans out of chimps using only a threefold increase in brain and a sixfold increase in prefrontal cortex. But the principle of mediocrity weighs heavily against such an arbitrary presumption, as well as the general notion that evolution doesn't build optimal systems, plus all the known flaws we talk about on Overcoming Bias. Furthermore my own, specific model of cognitive science is already suggesting that we can go beyond human purely on the basis of writing better software, but this is too complex to justify in detail here - the common sense of this should be apparent, though.

Doug, it may help to think in terms of a log odds ratio, in which case a probability of zero corresponds to negative infinity. You could never get that much counterevidence - we are talking about a subjective probability, not an objective frequency. Perhaps you are hallucinating or deluded or a toy in a computer simulation of mathematical error. Infinity is not a number, and 0 is not a probability.

Jonvon, there is only one human superpower. It makes us what we are. It is our ability to think. Rationality trains this superpower, like martial arts trains a human body. It is not that some people are born with the power and others are not. Everyone has a brain. Not everyone tries to train it. Not everyone realizes that intelligence is the only superpower they will ever have, and so they seek other magics, spells and sorceries, as if any magic wand could ever be as powerful or as precious or as significant as a brain.

Unfortunately, you picked the only member of the x-men who turned out to be a goddess and not a mutant

Ha! I knew someone was going to say that! (Because I looked up the Wikipedia entry, thank you very much.) That's why I invoked the movie version of Storm, who is a mutant! So there!

I hate how movies promote fake 'science' like that and then completely disregard actual science that can be used in an exciting movie. For example: we had the awful, fancy-looking cryptographic thinger in Skyfall that leaked its own key! What?! It leaked its own key!! Meanwhile, an action/heist movie based around Shamir's secret sharing basically writes itself! There are k shares and we must travel around the world to collect them! That masked man is running away with one of them, chase him! Oh wait, it fell into the river and then exploded! Now we must journey into the mountains to find the other one! But then we shoot this dying man as he is saying there were really k+1 shares! Dramatic ending!

Actually, now that I think about it, that's exactly what Voldemort did.

There is no "general evolution". There is biological evolution. Period. Saying "small incremental changes over time" is not a causal model, it is a surface effect to be explained. If you are talking about the realm of causal forces, of underlying processes, then biological evolution is all there is in science. Natural selection IS NOT a special case of some deeper principle that also explains change in toaster ovens any more than gravity is a special case of a deeper principle that also makes the stock market "fall".

HA: Not that I have anything against the guy, but, Kevin Kelly.

Eliezer said: "I encounter people who are quite willing to entertain the notion of dumber-than-human Artificial Intelligence, or even mildly smarter-than-human Artificial Intelligence. Introduce the notion of strongly superhuman Artificial Intelligence, and they'll suddenly decide it's "pseudoscience"."

It may be that the notion of strongly superhuman AI runs into people's preconceptions they aren't willing to give up (possibly of religious origins). But I wonder if the 'Singularians' aren't suffering from a bias of their own. Our current understanding of science and intelligence is compatible with many non-Singularity outcomes:

(a) 'human-level' intelligence is, for various physical reasons, an approximate upper bound on intelligence (b) Scaling past 'human-level' intelligence is possible but difficult due to extremely poor returns (e.g., logarithmic rather than exponential growth past a certain point) (c) Scaling past 'human-level' intelligence is possible, is not difficult, but runs into an inherent 'glass ceiling' far below 'incomprehensibility' of the resulting intelligence

and so on

Many of these scenarios seem as interesting to me as a true Singularity outcome, but my perception is they aren't being given equal time. Singularity is certainly more 'vivid,' but is it more likely?

The general sense of the word 'evolution' outside of biology is established in English, and some of the people who use it to describe phenomena like 'the evolution of technology' do understand evolutionary theory.

Okay, but there's also people who say, "Corporations split - therefore they reproduce - therefore they evolve." These are the people I'm talking about.

(http://en.wiktionary.org/wiki/evolution general: a gradual process of development biology: change in the genetic composition of a population over time)

Eliezer,

It seems that when we talk about "evolution by natural selection" as opposed to "Lamarckian evolution," evolution is the explanandum, and evolutionary theory attempts to provide the explanans.

"I encounter people who very definitely believe in evolution, who sneer at the folly of creationists. And yet they have no idea of what the theory of evolutionary biology permits and prohibits." One can legitimately sneer at the folly of creationists based on evidence for the explanandum of evolutionary biology (dinosaur bones, dog breeding, Neanderthals, and comparative anatomy) and the creationist response. This level of understanding is inferior to one that incorporates evolutionary theory, but is not simply attire.

"Or even worse, they'll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they'll call that "evolution". If evolutionary biology could cover that, it could cover anything." The general sense of the word 'evolution' outside of biology is established in English, and some of the people who use it to describe phenomena like 'the evolution of technology' do understand evolutionary theory.

I have noticed that since using the word "progress" has become unseemly, many use "evolution" in its stead. Quite often in the sense of "incremental change", sometimes in the slightly biology-analogous sense of "the effect of broad trial and error learning" - but hiding the teleological assumption progress was at least open about.

It has been scientifically proven that people use science attire to make their views sound more plausible :-) Throw in some neuroscience, statistics or a claim by a Ph.D. in anything and you show that you are credible. And the worst thing is that it seems to work fairly often. At the price, of course, that increasingly manipulation-savy media consumer start to suspect a Sinister Conspiracy behind every scientific claim. "Gravitational slingshots - who benefits?" "Who is really behind the stem cells?"

But this attire-wearning is likely nothing new. Tartuffe wore the attire of a pious person to manipulate. It might be more problematic from our standpoint that it is currently a largely epistemological profession/activity that is being used as high-status attire to hide bias and bad epistemology. Having people dress up in moral attire might have been bad for morality and people involved in the moral business, but it didn't hurt truth-seeking and bias-overcoming directly.

key take-away:

• Many people can’t judge the difference between plausible and implausible scientific explanations, for them it’s just ‘science’

• Science as a new authority to explain things (similar to religion)

Influential usually implies higher status. Status makes people effectively stupid, as it makes it harder for them to update their public positions without feeling that they are loosing face.

HA, most influential folks are far beyond persuading. It is the hearts and minds of the unconvinced, who are often novices, for which I fight.

HA, I don't remember where all the fallacies I encounter come from. I have a difficult enough time remembering someone's name after speaking to them for four hours. But before you accuse me of manufacturing strawmen, spend eleven years in my shoes putting up with the likes of this:

http://science.slashdot.org/comments.pl?&sid=191964

Science (unlike religion) has proved its myths See, this is exactly the sort of thing I have a problem with. Science is not just magic that works. People are learning science as though it were merely a true religion: passwords, attire, professions, and all.

And I agree, that's terrible. Maybe I wasn't clear in my post: it's a disaster for people to learn science as magic. But for people who won't learn it, for whatever reason, then "science as magic that works" is a sensible view that gives them a cheap tool to assess scientific claims.

But even for people learning science, the fact that "science works" is still critical. That needs to come before anything else. Building models and making them pay for inaccuracies is the correct thing to do... because it works. This is a feature of our world. If our world were run as a huge morality tale by a new testament god with low-brow litterary tastes, then advanced model-building wouldn't work and doing so would be the wrong thing to do. Scientists in that world would be just a Pythagorean sect, proposing a way of doing stuff that didn't lead to anything. We need to have some indication that we are not in that sort of world, before we decide to learn the scientific method. The fact that science "put men on the moon" is such an indication.

Then, once we accept that science and rationality work, and decide to study them... that is the moment to put all thoughts of magic away.

The press has reports with things like "scientists attack creationist teachings", but I've never seen "scientists attack common misconceptions about evolutionary theory". Press selectivity. Trust me, they do.

I know they do, I don't lack biologists among my friends. I've even been in that position myself, trying to inject a proper understanding of evolution into the heads of people whose reasonings are "Telepathy would be useful (and Star Trek has it), so we will soon evolve it".

But if people have no true understanding of the scientific process (and no desire or need to learn more) then the most accurate picture they can get, for the least effort, is defering to popular press articles on scientific discoveries.

I wasn't aware that speech, empathy, and social skills were functions of the kidneys rather than the brain. Never implied they were. But they're definetly not rational thinking, according to the definition you're been using. But this is far away from the main discussion, anyway.

Unfortunately, you picked the only member of the x-men who turned out to be a goddess and not a mutant (can't remember what story arc). How disturbing.

I second Stuart's awful sentence. I'm not seconding the opinion that it is awful, just that it resembles my thoughts.

"I wasn't aware that speech, empathy, and social skills were functions of the kidneys rather than the brain. "

We know empathy and social skills don't require general intelligence; plenty of mammals show empathy and social skills. If the definition of "intelligence" is "whatever occurs in the brain", then a 4004 CPU shows "intelligence" every time it adds two hex digits.

anonymous, I think we have good empirical evidence that Eliezer is not "unaware of the fact that biological evolution is only a subset from general evolution".

Eliezer and Kyle, name names of serious or influential people who posit in a mockworthy "Corporations split - therefore they reproduce - therefore they evolve."

If you're just throwing up a foil so we have a smug sense of in-groupedness, are you wasting our time on an overcomingbias blog?

Damn the double post! Sorry for that; my incompetence is to blame.

"Science (unlike religion) has proved its myths - by putting men on the moon, mobile phones in people's pockets, and curing diseases. It's payed its dues to reliability. So unless I am willing to look into it myself, I should, as a default, believe most things scientists claim." Stuart, this in my opinion is an awful sentence, and I'm surprised to read it by an overcomingbias contributor.

It lacks something of the pithiness of "Veni, vedi, Vinci", I'll admit. But I stand by what I was trying to say; hopefully my more recent post articulates it better.

i can't help but see a few interesting ironies in this post.

the "mutants" in the world of the x-men are people who all have one and only one common "genetic" mutation. and that mutation is the ability to mutate, as you put it, "in one generation". that is itself the essential mutation that is common to all "mutants". the fact that they can move from normal human to super powered mutant in the space of one generation (their own lifetime) is exactly the point.

in other words, "control over lightning" is not the metaphor. shooting laser beams out of the eyes, or teleporting or flying or... whatever, none of those things are the metaphor. the metaphor is the ability to mutate in the space of one lifetime. and the stories they spin from that wellspring often have to do with the fallout, the blessings and curses that come from what the ability to mutate means for any particular individual.

it's a metaphor that works quite well, as long as you don't take it too literally. (pun not intended but i can't help noting it.) everyone wants to be able to adapt to circumstances within the course of their own lifetime in a way that puts them on top of their situation. an entire religion - buddhism - seems to rotate around this central idea. mostly buddhists seem to be saying that only in rare circumstances does one soul "mutate" into a state in which that soul truly achieves enlightenment. most of us have to keep spinning on the great wheel and reincarnating until we get it right. i'm not a buddhist so i may be getting this wrong, at least in terms of emphasis.

for you, your "super power" or "mutation" is your understanding of scientific models or rational ways of thinking. it seems to me that you have achieved quite a bit along these lines. you started thinking about these things at a young age, coincidentally (or not?) about the same time stan lee's mutants discover their powers, as a budding teenager. (hopefully i'm remembering your recent post correctly.) in my sparsely informed viewpoint, entirely based on the posts i've read here, you have been driven by something intrinsic to you to understand the world in just the way you do now. everything led you to the point you are at now. there were things that were obvious to you at a young age that were not, i assure you, obvious to most other people.

you might phrase that very differently, and with much greater precision, and without all the biases and faulty logic that i'm sure infect my own thinking.

if you think about it though, you'll see that in some way you are a kind of mutant. you are the exception, not the rule. you started to think about things, at a young age, in a way that most people never get to. it may be that you come from a long line of such thinkers. the further out i push the metaphor, the shakier it gets. at the end of the day it's about discovering your own inherent powers and using them in a way that is, hopefully, best for you and for the people around you.

::threadjack::

The probability for anything is non-zero.

I'd be willing to assign zero probability to mathematical falsehoods, such as "2+2=5".

On topic:

Apparently, a lot of people really don't understand biological evolution.

(Many people don't really understand what a physicist means by a "wave", either, but they tend to be familiar with examples.)

Even if biological evolution would allow one-generation mutation for "Storm abilities", it would not equal that evolution doesn't explain anything. Even if everything "is possible" in evolution, there can still be different probabilities for different outcomes. The probability for "Storm ability mutation" is non-zero.

Or even worse, they'll talk about something completely outside the domain of evolutionary biology, like an improved design for computer chips, or corporations splitting, or humans uploading themselves into computers, and they'll call that "evolution". If evolutionary biology could cover that, it could cover anything.

That seems unfair. Do they mean evolutionary biology when they say evolution? What if they just mean heredity, variation, and selection? I use "evolution" to describe how technologies spread throughout culture, because it spreads by those three mechanisms, and those seem to be the mechanisms of biology as well. I think of evolution as a class of algorithms which optimize for something (or "select") without doing prediction, but restraining the search space by keeping only those that pass certain tests, and trying only those nodes out which are closed to ones that passed. and we see the failures in optimization we would expect from such an algorithm in many non-biological mechanisms.

Why this? "because they want to be part of the scientific in-crowd"

Why would everybody want to be part of the "scientific" in-crowd? That wouldn't leave much for the rest of the world.

Eliezer, I understand the need to contest the wrong-but-influential, but not the wrong-but-insignificant. Kevin Kelly is definitely influential. I just think you have real, worthy opponents (the ultimate one, our apparently pending mortality), I don't like to see limited energy get sucked up on hack, performed disagreements. I'd rather you get your representational privilege the most useful way to us --solving the hard problems we face, as quickly as possible, not performing 1/2 of various dialectics.

Eliezer, you seem to be making a science vs. engineering distinction. You're obviously aware of how evolution is used in engineering (as described in the wikipedia entry on evolution).

Took a look at Kevin Kelly's site. Instead of occasional foilicious potshots, how about a serious critique of the error of these ideas. Let's not manufacture a dialectic, I think that's going to get in the way of building the best models of reality.

"Corporations split - therefore they reproduce - therefore they evolve."

Okay, now those guys have issues.

Science (unlike religion) has proved its myths See, this is exactly the sort of thing I have a problem with. Science is not just magic that works. People are learning science as though it were merely a true religion: passwords, attire, professions, and all.

And I agree, that's terrible. Maybe I wasn't clear in my post: it's a disaster for people to learn science as magic. But for people who won't learn it, for whatever reason, then "science as magic that works" is a sensible view that gives them a cheap tool to assess scientific claims.

But even for people learning science, the fact that "science works" is still critical. That needs to come before anything else. Building models and making them pay for inaccuracies is the correct thing to do... because it works. This is a feature of our world. If our world were run as a huge morality tale by a new testament god with low-brow litterary tastes, then advanced model-building wouldn't work and doing so would be the wrong thing to do. Scientists in that world would be just a Pythagorean sect, proposing a way of doing stuff that didn't lead to anything. We need to have some indication that we are not in that sort of world, before we decide to learn the scientific method. The fact that science "put men on the moon" is such an indication.

Then, once we accept that science and rationality work, and decide to study them... that is the moment to put all thoughts of magic away.

The press has reports with things like "scientists attack creationist teachings", but I've never seen "scientists attack common misconceptions about evolutionary theory". Press selectivity. Trust me, they do.

I know they do, I don't lack biologists among my friends. I've even been in that position myself, trying to inject a proper understanding of evolution into the heads of people whose reasonings are "Telepathy would be useful (and Star Trek has it), so we will soon evolve it".

But if people have no true understanding of the scientific process (and no desire or need to learn more) then the most accurate picture they can get, for the least effort, is defering to popular press articles on scientific discoveries.

I wasn't aware that speech, empathy, and social skills were functions of the kidneys rather than the brain. Never implied they were. But they're definetly not rational thinking, according to the definition you're been using. But this is far away from the main discussion, anyway.

Eliezer, are you unaware of the fact that biological evolution is only a subset from general evolution?

I think people should be more careful about the word "science." Here are some meanings I see attached to it:

1. knowledge of nature
2. naturalism
3. "the scientific method"
4. institutions practicing the scientific method
5. rules of a particular institution
6. the output of institutions

I feel compelled to add that what I mean by "the scientific method" is that observation should drive belief and that we can put effort into obtaining useful observations (experiments, stamp collecting). Also, it may be useful to distinguish between institutional rules intended to protect the institution from cheaters and rules intended to protect people from their own biases.

belief as scientific attire, like wearing a lab coat. Science (unlike religion) has proved its myths - by putting men on the moon, mobile phones in people's pockets, and curing diseases. It's payed its dues to reliability. So unless I am willing to look into it myself, I should, as a default, believe most things scientists claim. And, unless I'm willing to study the press extensively, I should defer to uncontradicted press stories about scientific claims, especially if they're repeated. This makes science into a litterary genre, but it's the only real option for non-jounalist non-scientists.

The press has reports with things like "scientists attack creationist teachings", but I've never seen "scientists attack common misconceptions about evolutionary theory". Ergo, bashing creationists is sensible, but worrying about my own poor understanding is not.

As for the issue you identified - attitude towards AI's - there are some other aspects reinforcing people's attitudes (this may be why you found this response so prevalent). There is no theoretical barrier to constructing human level AI's, since humans's exist. We can improve human intelligence in various ways (slightly), so we can improve these AI's too (so slightly better than human is possible too). For super-AIs, on the other hand, nothing exists in the world to show that they are possible. And if someone had created them, or were confident of creating them, this would have been reported in the press. So the man on the street, even without apocalyptic litterature, should conclude that super-AI's are far from today's technology (of course, the apocalyptic litterature doesn't help).

Some people I know then make the same mistake you mentioned - super-AI's aren't science (no repeatable experiment). They are, however, talked about in a scientific language. Hence they must be pseudo-science.

Jonvon, there is only one human superpower. It makes us what we are. It is our ability to think. I'd add speach, empathy, opposable thumb, quite a long lifespan, and superior social skills to the list. All of them have just as much claim to "making us who we are" (though thinking and social skills have the best claim to "making us who we will be").

"I'd be willing to assign zero probability to mathematical falsehoods, such as "2+2=5"."

You might be willing to, Doug S., but that doesn't mean that it's optimally rational for you to do so. I don't know as much about bayesian reasoning as I'd like to, but my understanding is that would not be bayesian of you.