Singularity Summit 2011 Workshop Report

lukeprog

Here is a short new publication from the Singularity Institute, on the 2-day workshop that followed Singularity Summit 2011.

Note the new publication design. We are currently porting our earlier publications to this template, too.

Here is a short new publication from the Singularity Institute, on the 2-day workshop that followed Singularity Summit 2011.

Note the new publication design. We are currently porting our earlier publications to this template, too.

What changes with WBE? Waiting for it already forsakes significant part of the expected future, but then what? It accelerates the physical timeline towards FAI and catastrophe both, and squeezing more FAI than catastrophe out of it (compared to pre-WBE ratio) requires rather unlikely circumstances.

The way I see it, even if we completely solve decision theory, there are so many other problems involved with building an FAI that the success probability (unless we first develop WBE or intelligence amplification) is still tiny.

Yes. (With the caveat that I'm regarding decision theory as a currently salient instrumental focus in pursuing the overall FAI problem, not a goal in itself.)

So working on decision theory is counterproductive if it raises the probability of UFAI coming before WBE/IA by even a small delta.

There is probability and then probability in given period of time. Raising probability of catastrophe before WBE doesn't necessarily raise it overall (for example, suppose catastrophe is inevitable, then moving it closer doesn't change the probability of it eventually occurring, and introducing a bit of probability of lack of catastrophe in exchange for moving the catastrophe branch closer does reduce the long term risk).

Lack of a catastrophe is not a stable state, it can be followed by a catastrophe, while implemented FAI is stable. You seem to be considering two alternatives: (1) reduction in obscure more immediate risk of UFAI, achieved by a few people deciding not to think about decision theory and hence not talking about it in public (this decision, as it's being made, doesn't take many out of the pool of those who would make progress towards UFAI); and (2) taking a small chance to get FAI right. Estimating FAI ever being solved as low, I think choosing (2) is the correct play. Alternatively, don't talk about the results openly, but work anyway (which is an important decision, but the community is too weak right now for closing itself off, and our present ideas are too feeble to pose significant risk; likely should turn to secrecy later).

(I notice that I don't understand this clearly enough, will have to reflect in more detail.)

The workshop report gave "a roughly 14% chance of win if de novo AI (Friendly AI or not) came first" so the two of us seem to be much more pessimistic than average. Do you think we should be updating in their direction, or vice versa? (Unless the workshop is counting "partial wins" like an AI that fills the universe with orgasmium, or what I called "Instrumentally Friendly AI"?)

4Wei Dai14y

Did you read the part of the workshop report that talked about this? [...] Getting decision theory "right enough" could be important for building a viable UFAI (or at least certain types of it, e.g., non-neuromorphic). There's reason to think for example that AIXI would fail due to incorrect decision theory (but people trying to make AIXI practical do not seem to realize this yet). Given that we seem to constitute a large portion of all people trying to get decision theory right for AI purposes, the effect of our decisions might be larger than you think. [...] Yes, but of course that reduces the positive effects of working on decision theory, so you might decide that you should do something else instead. For example I think that thinking about strategy and meta-philosophy might be better uses of my time. (Also, I suggest that keeping secrets is very hard so even this alternative of working in secret may be a net negative.)

12

Singularity Summit 2011 Workshop Report

12

12

12

Singularity Summit 2011 Workshop Report

12

12