The Design Space of Minds-In-General

"everything that isn't a duck"

Muggles?

@ Eli:

"Arguably Marcus Hutter's AIXI should go in this category: for a mind of infinite power, it's awfully stupid - poor thing can't even recognize itself in a mirror."

Have you (or somebody else) mathematically proven this?

(If you have then that's great and I'd like to see the proof, and I'll pass it on to Hutter because I'm sure he will be interested. A real proof. I say this because I see endless intuitions and opinions about Solomonoff induction and AIXI on the internet. Intuitions about models of super intelligent machines like AIXI just don't cut it. In my experience they very often don't do what you think they will.)

[-]Eliezer Yudkowsky17y10

Shane, there was a discussion about this on the AGI list way back when, "breaking AIXI-tl", in which e.g. this would be one of the more technical posts. I think I proved this at least as formally, as you proved that proof that FAI was impossible that I refuted.

But of course this subject is going to take a separate post.

[-]Shane_Legg17y00

@ Eli:

Yeah, my guess is that AIXI-tl can be broken. But AIXI? I'm pretty sure it can be broken in some senses, but whether these senses are very meaningful or significant, I don't know.

And yes, my "proof" that FAI would fail failed. But it also wasn't a formal proof. Kind of a lesson in that don't you think?

So until I see a proof, I'll take your statement about AIXI being "awfully stupid" as just an opinion. It will be interesting to see if you can prove yourself to be smarter than AIXI (I assume you don't view yourself as below awfully stupid).

[-]Roko17y00

I might pitch in with an intuition about whether AIXI can recognize itself in a mirror. If I understand the algorithm correctly, it would depend on the rewards you gave it, and the computational and time cost would depend on what sensory inputs and motor outputs you connected it to.

For example, if you ran AIXI on a computer connected to a webcam with a mirror in front of it, and rewarded it if and only if it printed "I recognize myself" on the screen, it would eventually learn to do this all the time. The time cost might be large, though.

[-]Tim_Tyler17y60

What if AIXI accidentally bashed its own brains out before it tried that?

[-][anonymous]17y00

Where would cats fit on the space. I would assume that they would be near humans, sharing as they do an amygdala, prefrontal cortex, cerebellum and the neurons fire at the same speed I assume. Not sure about the abstract planning. Could you have done the psychological unity of the mammals for your previous article?

[-]Silas17y-20

Does anyone know the ratio of discussion of implementations of AIXI/-tl, to discussion of its theoretical properties? I've calculated it at about zero.

[-]Shane_Legg17y10

@ Silas:

Given that AIXI is uncomputable, how is somebody going to discuss implementing it?

An approximation, sure, but an actual implementation?

[-]bambi17y20

What do you mean by a mind?

All you have given us is that a mind is an optimization process. And: what a human brain does counts as a mind. Evolution does not count as a mind. AIXI may or may not count as a mind (?!).

I understand your desire not to "generalize", but can't we do better than this? Must we rely on Eliezer-sub-28-hunches to distinguish minds from non-minds?

Is the FAI you want to build a mind? That might sound like a dumb question, but why should it be a "mind", given what we want from it?

[-]bambi17y40

Perhaps "mind" should just be tabooed. It doesn't seem to offer anything helpful, and leads to vast fuzzy confusion.

[-]Roko17y00

@Tim Tyler: yeah, this is always an issue. And there is the issue that AIXI might kill the person giving it the rewards. [I'm being sloppy here: you can't implement an uncomputable algorithm on physically real computer, so we should be talking about some kind of computable approximation to the algorithm being able to recognize "itself" in a mirror. ]

[-]Silas17y00

Shane_Legg: factoring in approximations, it's still about zero. I googled a lot hoping to find someone actually using some version of it, but only found the SIAI's blog's python implementation of Solomonoff induction, which doesn't even compile on Windows.

[-]poke17y10

So is the reason I should believe this space of minds-in-general exists at all going to come in a later post?

[-]Shane_Legg17y10

@ Silas:

I assume you mean "doesn't run" (python isn't normally a compiled language).

Regarding approximations of Solomonoff induction: it depends how broadly you want to interpret this statement. If we use a computable prior rather than the Solomonoff mixture, we recover normal Bayesian inference. If we define our prior to be uniform, for example by assuming that all models have the same complexity, then the result is maximum a posteriori (MAP) estimation, which in turn is related to maximum likelihood (ML) estimation. Relations can also be estab... (read more)

[-]Phil_Goetz417y80

The larger point, that the space of possible minds is very large, is correct.

The argument used involving ATP synthase is invalid. ATP synthase is a building block. Life on earth is all built using roughly the same set of Legos. But Legos are very versatile.

Here is an analogous argument that is obviously incorrect:

People ask me, "What is world literature like? What desires and ambitions, and comedies and tragedies, do people write about in other languages?"

And lo, I say unto them, "You have asked me a trick question."

"the" ... (read more)

[-]Caledonian217y00

I would point out, Mr. Goetz, that some languages do not have a "the".

It is not clear how this changes the content of things people say or write in those languages. Whorf-Sapir, while disproven in the technical sense, is surprisingly difficult to abolish.

[-]Cyan217y00

Phil, I'm not really sure what your criticism has to do with what Eliezer wrote. He's saying that evolution is contingent -- bits that work can get locked into place because other bits rely on them. Eliezer asserts that AI design is not contingent in this manner, so the space of possible AI designs does not form a natural class, unlike the space of realized Earth-based lifeforms. Your objection is... what, precisely?

[-]bambi17y00

Silas: you might find this paper of some interest:

http://www.agiri.org/docs/ComputationalApproximation.pdf

[-]Arosophos17y00

Eliezer, do you intend your use of "artificial intelligence" to be understood as always referencing something with human origins? What does it mean to you to place some artificial intelligences outside the scope of posthuman mindspace? Do you trust that human origins are capable of producing all possible artificial intelligences?

[-]Unknown17y40

Phil Goetz was not saying that all languages have the word "the." He said that the word "the" is something every ENGLISH document has in common. His criticism is that this does not mean that Hamlet is more similar to an English restaurant menu than an English novel is to a Russian novel. Likewise, Eliezer's argument does not show that we are more like petunias then like an AI.

[-]komponisto217y40

Caledonian, Sapir-Whorf becomes trivial to abolish once you regard language in the correct way: as an evolved tool for inducing thoughts in others' minds, rather than a sort of Platonic structure in terms of which thought is necessarily organized.

Phil, I don't see how the argument is obviously incorrect. Why can't two works of literature from different cultures be as different from each other as Hamlet is from a restaurant menu?

3wnoise15y

Even taken this way, I don't see how it abolishes Sapir-Whorf. Different languages are different tools for inducing thoughts, and may be better or worse at inducing specific kinds of thought, which will in turn influence "self"-generated thoughts.

-1sark15y

Nope because the whole point is that thought is already existent. We use language to induce thoughts in other people. With ourselves we do not have to use language to induce our own thinking, we just think.

3TheOtherDave15y

That might be true, but I'd want to see evidence of it. If you're just appealing to intuition, well, my intuition points strongly in the other direction: I frequently find that the act of saying things out loud, or writing them down, changes the way I think about them. I often discover things I hadn't previously thought about when I write down a chain of thought, for example. I suspect that's pretty common among at least a subset of humans. Not to mention that the act of convincing others demonstrably affects our own thoughts, so the distinction you want to draw between "inducing thoughts in other people" and "thinking" is not as crisp as you want it to be.

0jimrandomh15y

Does talking about or writing a thought down cause you to notice more things than if you had spent a similar amount of time thinking about it without writing anything? That's the proper baseline for comparison.

2TheOtherDave15y

I'm not sure it is the proper baseline, actually: if I am systematically spending more time thinking about a thought when writing than when not-writing, then that's a predictable fact about the process of writing that I can make use of. Leaving that aside, though: yes, for even moderately complex thoughts, writing it down causes me to notice more things than thinking about them for the same period of time. I am far more likely to get into loops, far less likely to notice gaps, and far more likely to rely on cached thoughts if I'm just thinking in my head. What counts as "moderately complex" has a lot to do with what my buffer-capacity is; when I was recovering from my stroke I noticed this effect with even simple logic-puzzles of the sort that I now just solve intuitively. But the real world is full of things that are worth thinking about that my buffers aren't large enough to examine in detail.

0sark15y

Your verbalizations can affect your own thinking the same way the utterances of other people can. Your original thoughts don't originate from or structured/organized by language however. They were already there, and when you hear what you said out loud, or read what you wrote down, your thoughts get modulated because language can induce thought. Perhaps 'induce' is not a suitable word. 'Trigger' might be better. Like how a poem can be read many ways. The meanings weren't contained in or organized by the words of the poem. Rather the words triggered emotions and thoughts in our brains. Nevertheless, those emotions and words can also be triggered by non-verbal experiences. 'Language affects thought' is then as trivial as 'The smell of a rose can trigger the thought of a rose, as much as the word "rose" can.'

5TheOtherDave15y

If you believe "the thought of a rose" is the same thing whether triggered by the smell of a rose, a picture of a rose, or the word "rose" then we disagree on something far more basic than the role of language. That said, I agree with what you actually said. The differences between the thoughts triggered by different languages is "as trivial" (which is to say, not at all trivial) a difference as that between the thoughts triggered by the smell of a rose and the word "rose" (also non-trivial).

0sark15y

Of course, usually different sets of thoughts get triggered by those different triggers. Can you express more explicitly what you think we disagree on? I use 'trivial' in the specific sense of language being no different than other environmental triggers. Not in the sense of 'magnitude of effect'. For example, the cultural differences which usually track language differences are probably more explanatory of the different thought patterns of various groups. For instance, lack of concept for 'kissing' could simply be from kissing not being prevalent in a culture. 'Language differences' as an explanation is usually screened-off, since naturally language use will track cultural practice. I just don't see the point of focusing specifically on language. Sapir-Whorf's ambition doesn't seem to merely be including language among the myriad influences on thought. Rather it seems to say thought is somehow systematically 'organized' or 'constrained' by language. I think this is only possible since language is so expressive. If you say 'language' influences thought you seem to have in your hands a very powerful explanatory tool which subsumes all other specific explanations.

0TheOtherDave15y

OK. When wnoise asserted that different languages influence thoughts differently, you disagreed, implying that the language we use doesn't affect the thoughts we think because the thoughts precede the language. I disagree with that: far from least among the many things that affect the thoughts we think is the linguistic environment in which we do our thinking. But you no longer seem to be claiming that, so I assume I misunderstood. Rather, you now seem to be claiming that while of course our language affects what we think, so do other things, some of them much more strongly than language. I agree with that. As far as I know, nobody I classify as even remotely sane _dis_agrees with that. You also seem to be asserting that the idea that language influences thought is incompatible with the idea that nonlinguistic factors influence thought, because language is expressive. I don't understand this well enough to agree or disagree with it, though I don't find it compelling at all. Put another way: I say that language influences thought, and I also say that nonlinguistic factors influence thought. What factors most relevantly influence thought depends on the specifics of the situation, but there are certainly nonlinguistic factors that reliably outweigh the role of language. As far as I know, nobody believes otherwise. Do you disagree with any of that? If not, what did you think I was saying?

2sark15y

Thanks for the clarification. I was mainly addressing the topic Komponisto set up: You see that just seems to be what this whole Sapir-Whorf debate is about. For one, I don't think there would be anything to talk about if it was simply asserting that 'language occasionally influences thought, like a rose sometimes would'. Since language somehow seems so concomitant (if not actually integral) to thought this seems to show that we are somehow severely constrained/organized by the language we speak. So apologies if i'm getting this all wrong, but I just don't think it is fair for you to say 'of course I agree that other things influences thought'. You seem to be ignoring the obvious implication of language being so intricately tied up with thought. We do have a substantial disagreement in that you seem to think that even though language is one of the many influences of thought, it's impact is especially significant since language is somehow intimately dependent on thought. I can simultaneously agree that language can influence thought and that speaking different languages has little influence on thought. This is because I think most of the time other factors, culture being the most obvious one, screens off language as an explanation for differing thought patterns among people who speak different languages. This simply means that the seeming influence of language on thought is actually the influence of culture on thought, and in turn of thought on language, and then of language on thought again. I mentioned the expressiveness of language because I wanted to show how it can seem like it is language affecting thought, when it is simply channeling the influences of other factors, which it can easily do because it is expressive. I'll try to summarize my position: If you somehow managed to change the language a person speaks without changing anything else, you will not see a systematic effect on his thought patterns. This is because he would soon adapt the language fo

0komponisto15y

(...) Excellently put. The view you express here coincides exactly with mine.

2TheOtherDave15y

For an English speaker, I expect that a picture of a rose will increase (albeit minimally) their speed/accuracy in a tachistoscopic word-recognition test when given the word "columns." For a Chinese speaker, I don't expect the same effect for the Chinese translation of "columns." I expect this difference because priming effects on speed/accuracy of tachistoscopic word-recognition tasks based on lexical associations are well-documented, and because for an English-speaker, the picture of a rose is associated with the word "rose," which is associated with the word "rows," which is associated with the word "columns," and because I know of no equivalent association-chain for a Chinese-speaker. Of course, how long it takes to identify "columns" (or its translation) as a word isn't the kind of thing people usually care about when they talk about differences in thought. It's a trivial example, agreed. I mention it not because I think it's hugely important, but because it is concrete. That is, it is a demonstrable difference in the implicit associations among thoughts that can't easily be attributed to some vague channeling of the influences of unspecified and unmeasurable differences between their cultures. Sure, it's possible to come up with such an explanation after the fact, but I'd be pretty skeptical of that sort of "explanation". It's far more likely due to the differences between languages that caused me to predict it in the first place. One could reply that, sure, differences in languages can create trivial influences in associations among thoughts, but not significant ones, and it's "just obvious" that significant influences are what this whole Sapir-Whorf discussion is about. I would accept that.

0wnoise15y

There is not a one-to-one correlation between culture and language of course. The screening off is fairly weak. Brazil and Portugal simply do not have the same culture, nor for that matter do Texas and California. Even stronger separation happens for those who can speak multiple languages, and for these people culture does not screen off language. We can actually "change the language a person speaks" in this case. Do the polylingual talk about being able to think differently in different languages?

0sark15y

I don't think that has anything to do with the strength of the screening-off at all. P(T|C&L)=P(A|C) means C screens off L, which does not mean P(T|L&C)=P(T|L) meaning L screens of C. Screening off is not symmetric. Or in other words, I have not said that if 2 cultures are not the same but their languages are, then the thinking could not be different. Nice one. My prediction is that their thinking would not be affected in any systematic way at the level of abstractions.

[-]Cyan217y00

Unknown, okay, I see it now. Thanks.

The word "the" is something every English document has in common.

... except for Gadsby, A Void, and other lipograms. ;-)

[-]Phil_Goetz417y10

Phil, I don't see how the argument is obviously incorrect. Why can't two works of literature from different cultures be as different from each other as Hamlet is from a restaurant menu?

They could be, but usually aren't. "World literature" is a valid category.

[-]Fabio_Franco17y00

This discussion reminds me of Frithjof Schuon's "The Transcendent Unity of Religions", in which he argues that a metaphysical unity exists which transcends the manifest world and which can be "univocally described by none and concretely aprehended by few".

[-]Nick_Tarleton17y00

poke: what are you trying to say? It "exists" in the same sense as the set of all integers, i.e. it's a natural and useful abstraction, regardless of what you think of it ontologically.

[-]Psy-Kosh17y20

Sapir-Whorf is disproven? Blinks I thought only the strong form is disproven and that the weak form has significant support. (But, on the other hand, this isn't a field I'm familiar with at all, so go ahead and correct me...)

[-]Unknown17y-10

In regard to AIXI: One should consider more carefully the fact that any self-modifying AI can be exactly modeled by a non-self modifying AI.

One should also consider the fact that no intelligent being can predict its own actions-- this is one of those extremely rare universals. But this doesn't mean that it can't recognize itself in a mirror, despite its inability to predict its actions.

[-]jmmcd17y10

If we focus on the bounded subspace of mind design space which contains all those minds whose makeup can be specified in a trillion bits or less, then every universal generalization that you make has two to the trillionth power chances to be falsified.

Conversely, every existential generalization - "there exists at least one mind such that X" - has two to the trillionth power chances to be true.

So you want to resist the temptation to say either that all minds do something, or that no minds do something.

This is fine where X is a property which has ... (read more)

0sark15y

Eliezer did qualify those statements:

[-]Kaj_Sotala15y00

Just happened to re-read this post. As before, reading it fills me with a kind of fascinated awe, akin to staring at the night sky and wondering at all the possible kinds of planets and lifeforms out there.

4Will_Newsome15y

Reading it makes me feel annoyed, because Eliezer seems to ignore convergence in the mindspace of increasingly general intelligences. All optimization processes have to approximate something like Bayesian decision theory, and there does seem to be a coherent universal vector in human preferencespace that gets absorbed into individual humans (instead of just being implicit in the overall structure of humane values). Egalitarianism, altruism, curiosity, et cetera are to some extent convergent features of minds. The universal AI drives are the most obvious example, but we might underestimate to what extent sexual selection, game theoretically-derived moral sentiments, et cetera are convergent, especially in the limit as intelligence/wisdom/knowledge approaches a singularity. I wonder if the Babyeater equivalent of a Buddha would give up babyeating the same way that a human Buddha gives up romantic love. I suspect that Buddhahood or something close is the only real attractor in mindspace that could be construed as reflectively consistent given the vector humanity seems to be on. We should enjoy the suffering and confusion while we can.

-1Will_Newsome15y

Relatedly, I've formed the tentative intuition that paper clip maximizers are very hard to build; in fact, harder to build than FAI. What you do get out of kludge superintelligences is probably just going to be the pure universal AI drives (or something like that), or possibly some sort of approximately objective convergent decision theoretic policy, perhaps dictated by the acausal economy.

2Mitchell_Porter15y

The only really hard part about making a superintelligent paperclip maximizer is the superintelligence. If you think that specifying the goal of a Friendly AI is just as easy, then making a superintelligent FAI will be just as hard. The "fragility of value" thesis argues that Friendliness is significantly harder to specify, because there are many ways to go wrong. I don't. Simple selfishness is definitely an attractor (in the sense that it's an attitude that many people end up adopting), and it wouldn't take much axiological surgery to make it reflectively consistent.

3Will_Newsome15y

No, it takes a lot of work to specify paperclips, and thus it's not as easy as just superintelligence. You need goal stability, a stable ontology of paperclips, et cetera. I'm positing that it's easier to specify human values than to specify paperclips. I have a long list of considerations, but the most obvious is that paperclips have much higher Kolmogorov complexity than human values given some sort of universal prior with a sort of prefix you could pick up by looking at the laws of physics.

3Paul Crowley15y

I agree with your first sentence, but I suspect that building a friendly AI is harder than building a paperclipper.

2wnoise15y

Reference class confusion. "Paperclipper" refers to any "universe tiler", not just one that tile with paperclips. Specifying paperclips in particular is hard. If you don't care about exactly what gets tiled, it's much easier. For well-understood goals, that's easy. Just hardcode the goal. It's making goals that can change in a useful way that's hard. Part of the hardness of FAI is we don't understand friendly, we don't know what humans want, and we don't know what's good for humans, and any simplistic fixed goal will cut off our evolution. I don't see how you could possibly believe that, except out of wishful thinking. Human values are contingent on our entire evolutionary history. My parochial values are contingent upon cultural history and my own personal history. Our values are not universal. Different types of creature will develop radically different values with only small points of contact and agreement.

4JGWeissman15y

Hardcoding is not necessarily stable in programs that can edit their own source code.

0wnoise15y

Really? Isn't editing one's goal directly contrary to one's goal? If an AI self-edits in such a way that its goal changes, it will predictably no longer be working towards that goal, and will thus not consider it a good idea to edit its goal.

0Vaniver15y

It depends on how it decides whether or not changes are a good thing. If is trying out two utility functions- Ub for utility before and Ua for utility after- you need to be careful to ensure it doesn't say "hey, Ua(x)>Ub(x), so I can make myself better off by switching to Ua!". Ensuring that doesn't happen is not simple, because it requires stability throughout everything. There can't be a section that decides to try being goalless, or go about resolving the goal in a different way (which is troublesome if you want it to cleverly use instrumental goals). [edit] To be clearer, you need to not just have the goals be fixed and well-understood, but every part of everywhere else also needs to have a fixed and well-understood relationship to the goals (and a fixed and well-understood sense of understanding, and ...). Most attempts to rewrite source code are not that well-planned.

2TheOtherDave15y

Well, a number of voices here do seem to believe that there is something instantiable which is the thing that humans actually want and will continue to want no matter how much they improve/modify. Presumably those voices would not call something that tiles the universe with that thing a paperclipper, despite agreeing that it's a universe tiler, at least technically.

2wnoise15y

I believe that most voices here think that there are conditions that humans actually want and that one of them is variety. This in no way implies a material tiling. Permanency of these conditions is more questionable, of course. The only plausible attractive tiling with something material that I could see even a large minority agreeing to would be a computational substrate, "computronium".

0TheOtherDave15y

If we're talking about conditions that don't correlate well with particular classes of material things, sure. That's rarely true of the conditions I (or anyone else I know) value in real life, but I freely grant that real life isn't a good reference class for these sorts of considerations, so that's not evidence of much. Still, failing a specific argument to the contrary, I would expect whatever conditions humans maximally value (assuming there is a stable referent for that concept, which I tend to doubt, but a lot of people seem to believe strongly) to implicitly define a class of objects that optimally satisfies those conditions. Even variety, assuming that it's not the trivial condition of variety wherein anything is good as long as it's new. Computronium actually raises even more questions. It seems that unless one values the accurate knowledge (even when epistemically indistinguishable from the false belief) that one is not a simulation, the ideal result would be to devote all available mass-energy to computronium running simulated humans in a simulated optimal environment.

0wnoise15y

I think that needs a bit of refinement. Having lots of food correlates quite well with food. Yet no one here wants to tile the universe with white rice. Other people are a necessity for a social circle, but again few want to tile the universe with humans (okay, there are some here that could be caricatured that way). I think we all want to eventually have the entire universe able to be physically controlled by humanity (and/or its FAI guardians), but tiling implies a uniformity that we won't want. Certainly not on a local scale, and probably not on a macroscale either. Right, which is why I borught it up as one of the few reasonable counterexamples. Still, it's the programming makes a difference between heaven and hell.

0Will_Newsome15y

My arguments apply to most kinds of things that tile the universe with highly arbitrary things like paperclips, though they apply less to things that tile the universe with less arbitrary things like bignums or something. I do believe arbitrary universe tilers are easier than FAI; just not paperclippers. There are currently no well-understood goals, nor are there obvious ways of hardcoding goals, nor are hardcoded goals necessarily stable for self-modifying AIs. We don't even know what a goal is, let alone do we know how to solve the grounding problem of specifying what a paperclip is (or a tree is, or what have you). With humans you get the nice trick of getting the AI to look back on the process that created it, or alternatively just use universal induction techniques to find optimization processes out there in the universe, which will also find humans. (Actually I think it would find mostly memes, not humans per se, but a lot of what humans care about is memes.) Paperclips are contingent on that, plus a whole bunch of random cultural stuff. Again, if we're talking about universe tilers in general this does not apply. Also, as a sort of appeal to authority, I've been working at the Singularity Institute for a year now, and have spent many many hours thinking about the problem of FAI (though admittedly I've given significantly less thought to how to build a paperclipper). If my intuitions are unsound, it is not for reasons that are intuitively obvious.

6jimrandomh15y

Really? Maybe if you wanted it to be able to classify whether any arbitrary corner case count as a paperclip or not, but that isn't required for a paperclipper. If you're just giving it a shape to copy then I don't see why that would be more than a hundred bytes or so - trivial compared to the optimizer.

3Will_Newsome15y

A hundred bytes in what language? I get the intuition, but it really seems to me like paper clips are really complex. There are lots of important qualities of paperclips that make them clippy that seem to me like they'd be very hard to get an AI to understand. You say you're giving it a shape, but that shape is not at all easily defined. And its molecular structure might be important, and its size, and its density, and its ability to hold sheets of paper together... Shape and molecular component aren't fundamental attributes of the universe that an AI would have a native language for. This is why we can't just keep an oracle AI in a box -- it turns out that our intuitive idea of what a box is is really hard to explain to a de novo AI. Paperclips are similar. And if the AI is smart enough to understand human concepts that well, then you should also be able to just type up CEV and give it that instead... CEV is easier to describe than a paperclip in that case, since CEV is already written up. (Edit: I mean a description of CEV is written up, not CEV. We're still working on the latter.)

2Will_Sawin15y

If you can understand human concepts, "paperclip" is sufficient to tell you about paperclips. Google "paperclip," you get hundreds of millions of results. "Understanding human concepts" may be hard, but understanding arbitrary concrete concepts seems harder than understanding arbitrary abstract concepts that take a long essay to write up and have only been written up by one person and use a number of other abstract concepts in this writeup.

1JoshuaZ15y

Do you mean easier here?

1Will_Sawin15y

Yes.

5Normal_Anomaly15y

Well, here's a recipe for a paperclip in English: Make a wire, 10 cm long and 1mm in diameter, composed of an alloy of 99.8% iron and 0.2% carbon. Start at one end and bend it such that the segments from 2-2.5cm, 2.75-3.25cm, 5.25-5.75cm form half-circles, with all the bends in the same direction and forming an inward spiral (the end with the first bend is outside the third bend).

0datadataeverywhere15y

Sample a million paperclips from different manufacturers. Automatically cluster to find the 10,000 that are most similar to each other. Anything is a paperclip if it is more similar to one of those along every analytical dimension than another in the set. Very easy, requires no human values. The AI is free to come up with strange dimensions along which to analyze paperclips, but it now has a clearly defined concept of paperclip, some (minimal) flexibility in designing them, and no need for human values.

0shokwave15y

Counter: Sample a million people from different continents. Automatically cluster to find the 10,000 that are most similar to each other. Anything is a person if it is more similar to one of those along every analytical dimension than another in the set. This is already tripping majoritarianism alarm bells. I would meditate on this for a while when trying to define a paperclip.

0TheOtherDave15y

Sure. For that matter, one could play all kinds of Hofstadterian games along these lines... is a staple a paperclip? After all, it's a thin piece of shaped metal designed and used to hold several sheets of paper together. Is a one-pound figurine of a paperclip a paperclip? Does it matter if you use it as a paperweight, to hold several pieces of paper together? Is a directory on a computer file system a virtual paperclip? Would it be more of one if we'd used the paperclip metaphor rather than the folder metaphor for it? And on and on and on. In any case, I agree that the intuition that paperclips are an easy set to define depends heavily on the idea that not very many differences among candidates for inclusion matter very much, and that it should be obvious which differences those are. And all of that depends on human values. Put a different way: picking 10,000 manufactured paperclips and fitting a category definition to those might exclude any number of things that, if asked, we would judge to be paperclips... but we don't really care, so a category arrived at this way is good enough. Adopting the same approach to humans would similarly exclude things we would judge to be human... and we care a lot about that, at least sometimes.

0datadataeverywhere15y

TheOtherDave got it right; I'm wasn't trying to give a complete definition of what is and isn't a paperclip, I was just offering forth an easy to define (without human values) subset that we would still call paperclips. It has plenty of false negatives, but I don't really see that as a loss. Likewise, your personhood algorithm doesn't bother me as long as we don't use it to establish non-personhood.

0shokwave15y

Ah. This distinction escaped me; I tend to use definitions in a formal logical style (P or ~P, no other options).

4cousin_it15y

"Find a valid proof of this theorem from the axioms of ZFC". This goal is pretty well-understood, and I don't believe an AI with such a goal will converge on Buddha. Or am I misunderstanding your position?

0Vladimir_Nesov15y

This doesn't take into account logical uncertainty. It's easy to write a program that eventually computes the answer you want, and then pose a question of doing that more efficiently while provably retaining the same goal, which is essentially what you cited, with respect to a brute force classical inference system starting from ZFC and enumerating all theorems (and even this has its problems, as you know, since the agent could be controlling which answer is correct). A far more interesting question is which answer to name when you don't have time to find the correct answer. "Correct" is merely a heuristic for when you have enough time to reflect on what to do. (Also, even to prove theorems, you need operating hardware, and manging that hardware and other actions in the world would require decision-making under (logical) uncertainty. Even nontrivial self-optimization would require decision-making under uncertainty that has a "chance" of turning you from the correct question.)

2cousin_it15y

What's more interesting about it? Think for some time and then output the best answer you've got.

0Vladimir_Nesov15y

Try to formalize this intuition. With provably correct answers, that's easy. Here, you need a notion of "best answer I've got", a way of comparing possible answers where correctness remains inaccessible. This makes it "more interesting": where the first problem is solved (to an extent), this one isn't.

0Will_Newsome15y

Individual humans sometimes become more selfish, but not consistently reflectively so, and humanity seems to be becoming more humane over time. Obviously there's a lot of interpersonal and even intrapersonal variance, but the trend in human values is both intuitively apparent and empirically verified by e.g. the World Values Survey and other axiological sociology. Also, I doubt selfishness is as strong an attractor among people who are the smartest and most knowledgeable of their time. Look at e.g. Maslow's research on people at the peak of human performance and mental health, and the attractors he identified as self actualization and self transcendence. Selfishness (or simple/naive selfishness) mostly seems like a pitfall for stereotypical amateur philosophers and venture capitalists.

2Mitchell_Porter15y

I'm taking another look at this and find it hard to sum up just how many problems there are with your argument. What about the people who are the most powerful of their time? Think about what the psychology of a billionaire must be. You don't accumulate that much wealth just by setting out to serve humanity. You care about offering your customers a service, but you also try to kill the competition, and you cut deals with the already existing powers, especially the state. Most adults are slaves, to an economic function if not literally doing what another person tells them, and then there is a small wealthy class of masters who have the desire and ability to take advantage of this situation. I started out by opposing "simple selfishness" to your hypothesis that "Buddhahood or something close" is the natural endpoint of human moral development. But there's also group allegiance: my family, my country, my race, but not yours. I look out for my group, it looks out for me, and caring about other groups is a luxury for those who are really well off. Such caring is also likely to be pursued in a form which is advantageous, whether blatantly or subtly, for the group which gets to play benefactor. We will reshape you even while we care for you. How close do you think anyone has ever come to reflective consistency? Anyway, you are reflectively consistent if there's no impulse within you to change your goals. So anyone, whatever their current goals, can achieve reflective consistency by removing whatever impulses for change-of-values they may have. Reflective consistency isn't a matter of consistency with your trajectory so far, it's a matter of consistency when examined according to your normative principles. The trajectory so far did not result from any such thorough and transparent self-scrutiny. Frankly, if I ask myself, what does the average human want to be, I'd say a benevolent dictator. So yes, the trend of increasing humaneness corresponds to something - increased

1TheOtherDave15y

Seriously? For my part, I doubt that a typical human wants to do that much work. I suspect that "favored and privileged subject of a benevolent dictator" would be much more popular. Even more popular would be "favored and privileged beneficiary of a benevolent system without a superior peer." But agreed that none of this implies a reduced interest in having one's desires satisfied. (ETA: And, I should note, I agree with your main point about nonuniversality of drives.)

1Mitchell_Porter15y

Immediately after I posted that, I doubted it. A lot of people might just want autonomy - freedom from dependency on others and freedom from the control of others. Dictator of yourself, but not dictator of humanity as a whole. Though one should not underestimate the extent to which human desire is about other people. Will Newsome is talking about - or I thought he was talking about - value systems that would be stable in a situation where human beings have superintelligence working on their side. That's a scenario where domination should become easy and without costs, so if people with a desire to rule had that level of power, the only thing to stop them from reshaping everyone else would be their own scruples about doing so; and even if they were troubled in that way, what's to stop them from first reshaping themselves so as to be guiltless rulers of the world? Also, even if we suppose that that outcome, while stable, is not what anyone would really want, if they first spent half an eternity in self-optimization limbo investigating the structure of their personal utility function... I remain skeptical that "Buddhahood" is the universal true attractor, though it's hard to tell without knowing exactly what connotations Will would like to convey through his use of the term.

0TheOtherDave15y

I am skeptical about universal attractors in general, including but not limited to Buddhahood and domination. (Psychological ones, anyway. I suppose entropy is a universal attractor in some trivial sense.) I'm also inclined to doubt that anything is a stable choice, either in the sense you describe here, or in the sense of not palling after a time of experiencing it. Of course, if human desires are editable, then anything can be a stable choice: just modify the person's desires such that they never want anything else. By the same token, anything can be a universal attractor: just modify everyone's desires so they choose it. These seem like uninteresting boundary cases. I agree that some humans would, given the option, choose domination. I suspect that's <1% of the population given a range of options, though rather more if the choice is "dominate or be dominated." (Although I suspect most people would choose to try it out for a while, if that were an option, then would give it up in less than a year.) I suspect about the same percentage would choose to be dominated as a long-term lifestyle choice, given the expectation that they can quit whenever they want. I agree that some would choose autonomy, though again I suspect not that many (<5%, say) would choose it for any length of time. I suspect the majority of humans would choose some form of interdependency, if that were an option.

-1Strange715y

Entropy is the lack of an identifiable attractor.

4TheOtherDave15y

If you ever felt inclined to lay out your reasons for believing that general intelligences without a shared heritage are likely to converge on egalitarianism, altruism, curiosity, game theoretical moral sentiments, or Buddhahood... or, come to that, lay out more precisely what you mean by those terms... I would be interested to read them.

0Will_Newsome15y

Evolution works on species, smart species' members will either evolve together or eventually get smart enough to learn to copy each other even if adversarial, such interactions will probably roughly approximate evolutionary game theory, and iterated games for social animals will probably yield cooperation and possibly altruism. Knowing more about yourself, the process that created you, and the arbitrarity in how you ended up with your preferences, intuitively seems like it would promote egalitarianism both on aesthetic grounds and pragmatic game theoretic grounds. Curiosity is just a necessary prerequisite for intelligence, it's obviously convergent. Something like Buddhahood is just a necessary prerequisite for a reasonable decision theory approximation, and is thus also convergent. That one is horribly imprecise, I know, but Buddhahood is hard enough to explain in itself, let alone as a decision theory approximation, let alone as a normative one. That's just the scattershot sleep-deprived off-the-top-of-my-head version that's missing all the good intuitions. If I end up converting my mountain of intuitions into respectable arguments in text I will let you know. It's just so much easier to do in-person, where I can get quick feedback about others' ontologies and how they mesh with mine, et cetera.

0TheOtherDave15y

Thanks, on both counts. And, yes, agreed that it's easier to have these sorts of conversations with known quantities.

[-]jacob_cannell13y180

This post is cute, but there are several flaws/omissions that can lead to compound propagating errors in typical interpretations.

Any two AI designs might be less similar to each other than you are to a petunia.

Cute. The general form of this statement:

(Any two X might be less similar to each other than you are to a petunia) is trivially true if our basis of comparison is based solely on genetic similarity.

This leads to the first big problem with this post: The idea that minds are determined by DNA. This idea only makes sense if one is thinking of a mind as a sort of potential space.

Clone Einstein and raise him with wolves and you get a sort of smart wolf mind inhabiting a human body. Minds are memetic. Petunias don't have minds. I am my mind.

The second issue (more of a missing idea really) is that of functional/algorithmic equivalence. If you take a human brain, scan it, and sufficiently simulate out the key circuits, you get a functional equivalent of the original mind encoded in that brain. The substrate doesn't matter, and nor even do the exact algorithms, as any circuit can be replaced with any algorithm that preserves the input/output relationships.

Functional equival... (read more)

1azergante5mo

Reversing your analogy, if you clone a wolf and raise it with Einsteins, you do not get another Einstein. That is because hardware (DNA) matters and wolves do not have the required brain hardware to instantiate Einstein's mind. Minds are determined to a large extent (though not fully as you rightly point out) by the specific hardware or hardware architecture that instantiates them, Eliezer stresses this point in his post Detached Lever Fallacy that comes earlier in the sequence: learning a human language and culture requires specialized hardware that has been evolved (optimized) specifically for the task, the same goes for deciding to grow fur when it's cold (you need sensors) or for parsing what is seen (eyes + visual cortex). The right environment is not enough, you need a mind that already works to even be able to learn from the environment. A quick search tells me this is not true for reptiles, amphibians, fish and insects. Primates (including humans) and birds have specific hardware dedicated to this task: mirror neurons. In practice it does as soon as you try to make such a mind because you do not have the same runtime complexity or energy consumption etc depending on the substrate. For example you can emulate a Quantum Computer on a classical computer, but you get an exponential slow-down.

LESSWRONG
LW

LESSWRONG
LW

46

The Design Space of Minds-In-General

46

46