All of luzr's Comments + Replies


I am starting to be sort of frightened by your premises - especially considering that there is non-zero probablity of creating some nonsentient singleton that tries to realize your values.

Before going any further, I STRONGLY suggest that you think AGAIN what might be interesting in carving wooden legs.

Yes, I like to SEE MOVIES with strong main characters going through the hell. But I would not want any of that.

It does not matter that AI can do everything better than me. Right now, I am not the best carving the wood either. But working with wood is ... (read more)


"Narnia as a simplified case where the problem is especially stark."

I believe there are at least two significant differences:

  • Aslan was not created by humans, it does not represent the "story of intelligence" (quite contrary, lesser intelligence was created by Aslan, as long as you interpret it as God).

  • There is only single Aslan with single predetermined "goal" while there are millions of Culture minds, with no single "goal".

(actually, second point is what I dislike so much about the idea of singleton - it can turn into something like benevolent but oppressing God too easily. Aslan IS Narnia Singleton).


"asks a Mind whether it could create symphonies as beautiful as it and how hard it would be"

On somewhat related note, there are still human chess players and competitions...

Brilliant observation! Damn, we really are living in the future already...


It is really off-topic, and I do not have a copy of Consider Phlebas at hand now, but

Even if Banks have not mentioned 'sublimed' in the first novel, the concept exactly fits Dra'Azon.

Besides, Culture is not really advancing its 'base' technology, but rather rebuilding its infrastructure to war-machine.

Eliezer (about Sublimation):

"Ramarren, Banks added on that part later, and it renders a lot of the earlier books nonsensical - why didn't the Culture or the Idarans increase their intelligence to win their war, if it was that easy? I refuse to regard Excession as canon; it never happened."

Just a technical (or fandom?) note:

Sublimed civilization is the central plot of Consider Phlebas (Schar's world, where Mind escapes, is "protected" by sublimed civilization - that is why direct military action by either Iridans or Culture is impossible).

Julian Morrison:

Or you can revert the issue once again. You can enjoy your time on obsolete skills (like sports, arts or carving table legs...).

There is no shortage of things to do, there is only a problem with your definition of "worthless".

"If you already had the lifespan and the health and the promise of future growth, would you want new powerful superintelligences to be created in your vicinity, on your same playing field?"

Yes, definititely. If nothing else, it means diversity.

"Or would you prefer that we stay on as the main characters in the story of intelligent life, with no higher beings above us?"

I do not care, as long as story continues.

And yes, I would like to hear the story - which is about the same thing I would get in case Minds are prohibited. I will not be th... (read more)

anon: "The cheesecake is a placeholder for anything that the sentient AI might value highly, while we (upon sufficient introspection) do not."

I am quite aware of that. Anyway, using "cheescake" as placeholder adds a bias to the whole story.

"Eliezer thinks that some/most of our values are consequences of our long history, and are unlikely to be shared by other sentient beings."

Indeed. So what? In reality, I am quite interested what superintelligence would really consider valueable. But I am pretty sure that "big cheescake&... (read more)

Indeed, when we substitute for "cheesecake" the likely things that a superintelligent AI might value, the problem becomes a whole lot less obvious. "We want to create a unified superintelligence that encompasses the full computational power of the universe." "We want to create the maximum possible number of sentient intelligences the universe can sustain." "We want to create a being of perfect happiness, the maximally hedonic sentient." "We want to eliminate the concepts of 'selfishness' and 'hierarchy' in favor of a transcendental egalitarian anarchy." Would humans resist these goals? Yes, because they probably entail getting rid of us puny flesh-bags. But are they worth doing? I don't know... it kinda seems like they might be.

Uhm, maybe it is naive, but if you have a problem that your mind is too weak to decide, and you have real strong (friendly) superintelligent GAI, would not it be logical to use GAIs strong mental processes to resolve the problem?

"The counter-argument that completely random behavior makes you vulnerable, because predictable agents better enjoy the benefits of social cooperation, just doesn't have the same pull on people's emotions."

BTW, completely deterministic behaviour makes you vulnerable as well. Ask computer security experts.

Somewhat related note: Linux strong random number generator works by capturing real world actions (think user moving mouse) and hashing them into random number that is considered for all practical purposes perfect.

Taking or not taking action may ... (read more)

"If this were all the hope the future held, I don't know if I could bring myself to try. Small wonder that people don't sign up for cryonics, if even SF writers think this is the best we can do."

Well, I think that the points missed is that you are not FORCED to carve those legs. If you find something else interesting, do it.


"The "Culture" sequence of novels by Iain M. Banks suggests how people might cope with machines doing all the work."

Exactly, I think Culture is highly relevant to most topics discussed here. Obviously, it is just a fictional utopia, but I believe it gives plausible answer to "unlimited power future".

For the reference:

"Wait for the opponents to catch up a little, stage some nice space battles... close the game window at some point. What if our universe is like that?"

Wow, what a nice elegant Fermi paradox solution:)

"because you don't actually want to wake up in an incomprehensible world"

Is not it what all people do each morning anyway?

"Errr.... luzr, why would I assume that the majority of GAIs that we create will think in a way I define as 'right'?"

It is not about what YOU define as right.

Anyway, considering that Eliezer is existing self-aware sentient GI agent, with obviously high intelligence and he is able to ask such questions despite his original biological programming makes me suppose that some other powerful strong sentient self-aware GI should reach the same point. I also believe that more general intelligence make GI converge to such "right thinking".

What m... (read more)

Could reach the same point. Said Eliezer agent is programmed genetically to value his own genes and those of humanity. An artificial Elizer could reach the conclusion that humanity is worth keeping but is by no means obliged to come to that conclusion. On the contrary, genetics determines that at least some of us humans value the continued existence of humanity.


"If we are so unfortunate as to live in a universe in which knowledge is finite, then conflict may serve as a substitute for ignorance in providing us a challenge."

This is inconsistent. What would conflict really do is to provide new information to process ("knowledge").

I guess I can agree with the rest of post. What IMO is worth pointing out that the most pleasures, hormones and insticts excluded, are about processing 'interesting' infromations.

I guess, somewhere deep in all sentient beings, "interesting informations" ar... (read more)

"But considering an unlimited amount of ice cream forced me to confront the issue of what to do with any of it."

"If you invoke the unlimited power to create a quadrillion people, then why not a quadrillion?"

"Say, the programming team has cracked the "hard problem of conscious experience" in sufficient depth that they can guarantee that the AI they create is not sentient - not a repository of pleasure, or pain, or subjective experience, or any interest-in-self - and hence, the AI is only a means to an end, and not an end i... (read more)


Well, as off-topic recourse, I see only cited some engineering problems in your "Against Cyborgs" essay as contraargument. Anyway, let me to say that in my book:

"miniaturizing and refining cell phones, video displays, and other devices that feed our senses. A global-positioning-system brain implant to guide you to your destination would seem seductive only if you could not buy a miniature ear speaker to whisper you directions. Not only could you stow away this and other such gear when you wanted a break, you could upgrade without brain ... (read more)


"Will, your example, good or bad, is universal over singletons, nonsingletons, any way of doing things anywhere."

I guess there is significant difference - for singleton, each mistake can be fatal (and not only for it).

I believe that this is the real part I dislike about the idea, except the part where singleton either cannot evolve or cannot stay singleton (because of speed of light vs locality issue).


"Tim, your page doesn't say anything about FOR loops or self-optimizing compilers not being able to go a second round, which is the part you got from me and then thought you had invented."

Well, it certainly does:

"Today, machines already do a lot of programming. They perform refactoring tasks which would once have been delegated to junior programmers. They compile high-level languages into machine code, and generate programs from task specifications. They also also automatically detect programming errors, and automatically test existi... (read more)


"Tim probably read my analysis using the self-optimizing compiler as an example, then forgot that I had analyzed it and thought that he was inventing a crushing objection on his own."

Why do you think it is crushing objection? I believe Tim just repeats his favorite theme (which, in fact, I tend to agree with) where machine augmented humans build better machines. If you can use automated refactoring to improve the way compiler works (and today, you often can), that is in fact pretty cool augmentation of human capabilities. It is recursive ... (read more)

"FOOM that takes two years"

In addition to comments by Robin and Aron, I would also pointed out the possibility that longer the FOOM takes, larger the chance it is not local, regardless of security - somewhere else, there might be another FOOMing AI.

Now as I understand, some consider this situation even more dangerous, but it as well might create "take over" defence.

Another comment to FOOM scenario and this is sort of addition to Tim's post:

"As machines get smarter, they will gradually become able to improve more and more of themsel... (read more)

Tim Tyler:

As much as I like your posts, one formal note:

If you are responding to somebody else, it is always a good idea to put his name at the beginning of post.

"because parallelizing is programmatically difficult"

Minor note: "Parallelization is programmatically difficult" is in fact another example of recursion.

The real reason why programming focused on serial execution was the fact that the most hardware was serial. There is not much point learning mysteries of multithreaded development if chances that your SW will run on multicore CPU is close to zero.

Now when multicore CPUs are de facto standard, parallel programming is no longer considered prohibitively difficult, it is just another thing ... (read more)

A powerful world government that took control of human reproduction could effectively direct human evolution, reducing natural selection between agents to a subsidiary role. 6 minutes is an irrelevant delay in the context of this type of system.

Nice thread.

Seriously, I guess Eliezer really needs this kind of reality check wakeup, before his whole idea of "FOOM" and "recursion" etc... turns into complete cargo cult science.

While I think the basic premise (strong AI friendliness) is quite concern, many of his recent posts sound like he had read too much science fiction and watched Terminator movie too many times.

There are some very basic issues with the whole recursion and singleton ideas... GenericThinker is right, 'halting problem' is very relevant there, in fact it proves tha... (read more)

They do that using dedicated hardware. Try to paint Crysis in realtime 'per pixel', using a vanilla CPU.

Interestingly, today's high-end vanilla CPU (quadcore at 3Ghz) would paint 7-8 years old games just fine. Means in another 8 years, we will be capable of doing Crysis without GPU.


"You're currently using a program which can access the internet. Why do you think an AI would be unable to do the same?"

I hope it will. Still, that would get it only to preexisting knowledge.

It can draw many hypothesis, but it will have to TEST them (gain empirical knowledge). Think LHC.

BTW, not that there are problems in quantum physics that do not have analytical solution. Some equations simply cannot be solved. Now of course, perhaps superintelligence will find how to do that, but I believe there are quite solid mathematic proofs that it ... (read more)

"Intelligence tests are timed for a good reason. If you see intelligence as an optimisation process, it is obvious why speed matters - you can do more trials."

Inteligence tests are designed to measure performance of human brain.

Try this: Strong AI running on 2Ghz CPU. You reduce it to 1Ghz, without changing anything else. Will it make less intelligent? Slower, definitely.

" The faster they get the smarter they are - since one component of intelligence is speed."

I think this might be incorrect. The speed means that you can solve the problem faster, not that you can solve more complex problem.

"I suspect that, if correctly designed, a midsize computer cluster would be able to get high-grade thinking done at a serial speed much faster than human, even if the total parallel computing power was less."

I am glad I can agree for once :)

"The main thing I'll venture into actually expecting from adding "insight" to the mix, is that there'll be a discontinuity at the point where the AI understands how to do AI theory, the same way that human researchers try to do AI theory. An AI, to swallow its own optimization chain, must not j... (read more)

Perfect simulation is not the only means of self-knowledge. As for empirical knowledge, I'm not sure Eliezer expects an AI to take over the world with no observations/input at all, but he does think that people do far overestimate the amount of observations an effective AI would need. (Also, for an AI, "building a new AI" and "self-improving" are pretty much the same thing. There isn't anything magic about "self". If the AI can write a better AI, it can write a better AI; whether it calls that code "self" or not makes no difference. Granted, it may be somewhat harder for the AI to make sure the new code has the same goal structure if it's written from scratch, but there's no particular reason it has to start from scratch.)
"I think it all boils down to very simple showstopper - considering you are building perfect simulation, how many atoms you need to simulate a atom?" The answer to that question is a blatant "At most, one." The universe is already shaped like itself.


"Religion has great benefits for a society and for individuals, even if there is no creator God."

And these are?

Well, where I live, there is about 70% atheists and 30% believers. I have a couple of believers as friends. From my experience, no, they are not better people than we are.


"I'm not sure if this has been discussed here before, but how isn't atheism as religion? It has to be accepted on faith, because we can't prove there isn't a magical space god that created everything. I happen to have more faith in atheism than Christ... (read more)


"Religion might be more understandable to atheists"

What makes you think religion is not understandable to atheists?

As for your explanation, yeah, nice, but do you really think this is what most christians really believe in? (You can try - find some and ask if he thinks that God equals to Society).

Also, what is a use for such abstraction?


"The "foom" is now."

I like that. Maybe we can get some T-shirts? :)

"There are not many guarantees that we can make about the behavior of society as a whole. Does society act like it values human life? Does society act like it values human comfort?"

Good point. Anyway, it is questionable whether we can apply any Elizier's friendlines guidelines to the whole society instead of single strong general AI entity.


Thanks for the link. I have the website to my AI portfolio.

BTW, I guess you got it right:) I have came to similar conclusions, including the observation about exponential functions :)


I guess that in Tim's scenario, "friendliness" is no near as important subject. Without "foom" there is a plenty of time for debugging...

I think you should try to consider one possible thing:

In your story, Engelbart failed to produce the UberTool.

Anyway, looking around and seeing the progress since 1970, I would say, he PRETTY MUCH DID. He was not alone and we should rather speak about technology that succeeded, but what else is all the existing computing infrastructure, internet, Google etc.. than the ultimate UberTool, augmenting human cognitive abilities?

Do you think we could keep the Moore's law going without all this? Good luck placing those two billions transistors of next generation ... (read more)


Quick reply: I think we will not know exactly, but that does not mean we cannot do it.

More detailed: I believe that the process will go with some sort of connectionist approach maybe mixed with some genetic algorithm voodoo. The key is reaching high complexity system that does something and starts to self-evolve (which in fact is nearly the same thing as learn).

Anyway, I believe that when all is done, we will have the same problem understanding Strong-AI mental processes as we have human brain ones.

Interesting... Reading this entry somewhat awakened my interest in computer chess... And I think I have stumled upon a nice example "inteligence as book smart" misconception.

"Fischer believed that eliminating memorized book moves would level the playing field; as an accidental consequence, it makes computer chess programs much weaker, as they rely heavily on the opening book to beat humans."

Classic example of "bias". The reality:

"In 2005, the chess program Shredder, developed by Stefan... (read more)