# All of derekz2's Comments + Replies

Nonsentient Optimizers

These last two posts were very interesting, I approve strongly of your approach here. Alas, I have no vocabulary with any sort of precision that I could use to make a "nonperson" predicate. I suppose one way to proceed is by thinking of things (most usefully, optimization processes) that are not persons and trying to figure out why... slowly growing the cases covered as insight into the topic comes about. Perform a reduction on "nonpersonhood" I guess. I'm not sure that one can succeed in a universal sense, though... plenty of peopl... (read more)

Not Taking Over the World

Oops, Julian Morrison already said something similar.

Not Taking Over the World

I think Robin's implied suggestion -- to not be so quick to discard the option of building an AI that can improve itself in certain ways but not to the point of needing to hardcode something like Coherent Extrapolated Volition. Is it really impossible to make an AI that can become "smarter" in useful ways (including by modifying its own source code, if you like), without it ever needing to take decisions itself that have severe nonlocal effects? If intelligence is an optimization process, perhaps we can choose more carefully what is being optim... (read more)

Sustained Strong Recursion

Robin, perhaps you could elaborate a little bit... assuming I understand what's going on (I'm always hopeful), the "recursion" here is the introduction of output being a function of "subjective time" (y) instead of "clock time" (t), and, further, y -- it is postulated -- is related to t by:

dy/dt = e^y

because the ratio of y to t is directly related to output (which as noted above is said to be an exponential function of y due Moore's law-type arguments).

That's seriously "strange". It is very different than a non-&quo... (read more)

Sustained Strong Recursion

Good post, thanks for making it. Besides the issue of whether Intel gets some of the recursive benefits, there is also the question of how FOOMable Intel would be if its engineers ran on its own hardware. Since Intel is embedded in the global economy and chip fabs are monstrously expensive undertakings, speeding up certain design issues would only go so far. I suppose the answer is that Intel will shortly invent molelcular nanotechnology but it's not really clear to what extent Drexler's vision or a completely flexible variant is even possible.

Still, your point here was to illustrate mathematically the way "recursion" of the type you are talking about increases growth and you did a good job of that.

Recursive Self-Improvement

From a practical point of view, a "hard takeoff" would seem to be defined by self-improvement and expansion of control at a rate too fast for humans to cope with. As an example of this, it is often put forward as obvious that the AI would invent molecular nanotechnology in a matter of hours.

Yet there is no reason to think it's even possible to improve molecular simulation, required to search in molecular process-space, much beyond our current algorithms, which on any near-term hardware are nowhere near up to the task. The only explanation is th... (read more)

Surprised by Brains

At some random time before human brains appeared, the time estimate is more difficult. Believer's imminence is helped a lot by humans' existence.

This public exercise, where two smart people hunt for the roots of their disagreement on a complex issue, has the potential to be the coolest thing I've seen yet on this blog. I wonder if it's actually possible -- there are so many ways that it could go wrong, humans being what they are. I really appreciate the two of you giving it a try.

Not that you asked for people to take shots from the cheap seats, but Eliezer, your ending question: "I mean... is that really what you want to go with here?" comes across (to me at least) as an unnecessarily belligerent way to proceed, likely to lead the process in unfortunate directions. It's not an argument to win, with points to score.

Failure By Analogy

Analogies are great guess generators, sources of wondrous creativity. It's very cool that the universe works in such a way that analogy often leads to fruitful ideas, and sometimes to truths.

But determining whether a particular analogy has really led anywhere requires more and different work, because analogy is not a valid form of inference. Nothing is ever true because of an analogy.

Selling Nonapples

I think subsumption is still popular amongst students and hobbyists.

It does raise an interesting "mini Friendliness" issue... I'm not really comfortable with the idea of subsumption software systems driving cars around on the road. Robots taking over the world may seem silly to most of the public but there are definite decisions to be made soon about what criteria we should use to trust complex software systems that make potentially deadly decisions. So far I think there's a sense that the rules robots should use are completely clear, the situations enumerable -- because of the extreme narrowness of the task domains. As tasks become more open ended, that may not be true for much longer.