What is the value of strategy vs. object-level progress toward a positive Singularity?
I think strategy is currently being undervalued, although this workshop and other recent SIAI work is alleviating that concern quite a bit. I'm still not sure why there is so little discussion of Singularity strategies on LW, compared to say decision theory, CEV, or Oracle AI, even as SIAI has started doing a lot more work in that area. Is it that most LW posts can be considered replies/followups to Eliezer's posts, and he never wrote one about Singularity strategies?
Almost none of the strategy work has been published, but we've had thousands upon thousands of hours of dialogue between SIAI and FHI researchers and others like Paul Christiano (who is now a research associate) and Owain Evans. Carl has been in Oxford the last two weeks working with FHI folk on strategy issues almost every day. SIAI had a series of 10-hour strategy meetings in January. Believe me, lots of strategy work is being done, but I understand that you're not seeing hardly any of it. Much of the strategy work that's been done by SIAI and FHI in the past five years is, thankfully, ending up in Bostrom's forthcoming monograph on machine superintelligence.
I loved the list of important questions. It would be nice if they were organized (a wiki seems like a good way to do it) and weighted (or had their implicit import explained). I think some of the answers can be crowdsourced within this community.
The tiebreaker consideration deemed most important was timelines. Pushing on WBE-related tech would, if anything, cause machine superintelligence to be created sooner. All else being equal, sooner timelines would seem to decrease the probability of a win scenario. Thus, despite many shortcomings in the design of this expert elicitation, the group’s current collective best guess, given the considerations evaluated so far, is that humanity should not accelerate WBE-related tech.
If this argument is correct, doesn't it also imply that humanity should not accelerate de novo AI-related tech? Did the workshop notice that, or am I misunderstanding something?
(BTW, when I tried to copy a section from the PDF to quote it, some of the characters (for example, "Th" in the first word of the quoted paragraph) couldn't be copied correctly. Whoever is doing the PDF production, can you please check if there is some option you can set to correct this?)
I assume the th couldn't be copied because it's a ligature, which makes the publication look better.
doesn't it also imply that humanity should not accelerate de novo AI-related tech
Correct. This is a key piece of recommended differential technological progress, and this recommendation is stated explicitly in both "AI as a negative and positive factor in global risk" and in "Intelligence Explosion: Evidence and Import."
A couple of minutes of research points to this post about ligature in pdf: Make ligatures in Linux Libertine copyable (and searchable). It appears that there are ways of configuring a TeX document so that ligature can be more reliably copied out of PDF as plain text (this depends on correct support of copying out by PDF readers, but some ways of creating PDFs make more readers handle this operation correctly).
I have forwarded this request to the person heading up the project to TeXify our publications.
I didn't try copy/paste before. I now tried Wei's fragment, and copying out of "Th" (as well as search) doesn't work in Foxit Reader 2.3.2923 (I got "Te tiebreaker" instead of "The tiebreaker").
I tried opening the PDF in Google Chrome (17.0.963.66 m/win32 XP) using its in-built viewer, and it doesn't open at all.
I didn't try copy/paste before. I now tried Wei's fragment, and copying out of "Th" (as well as search) doesn't work in Foxit Reader 2.3.2923 (I got "Te tiebreaker" instead of "The tiebreaker").
2.3? Wow. I'm on Foxit Reader 5.4.1. Perhaps that's what is making the difficulties?
Did you try copying the 'Th' by itself? My reader (Mac OS X's Preview) gives me only the first letter when I try to copy a single ligature but works if I include the next character in the selection.
Also, initially Chrome failed to open it for me too but succeeded on another try. Weird.
Edit: added name of the reader.
Reloading in Chrome worked (I checked with md5sum that it's the same file as before, when it didn't work), and copying out of "The" works from Chrome. Foxit Reader doesn't work, no matter how "The" is copied.
(Both you and Luke didn't name their readers where copying works, which is unhelpful if you're mentioning the fact that it works somewhere at all.)
I just clicked on the link in the opening post and it opened in my foxit reader. I then copy and pasted this:
came first. The original
Can you confirm this for me please Vladimir? Perhaps do a "Save file as" then open externally? Or perhaps some sort of force reload? We need to be sure it isn't a cached version. Actually, I'll email you one.
If you are having problems with Foxit Reader (my primary test case) then we have something of a problem! (In that case I will try, for example, installing foxit on a virtual machine that doesn't have the fonts installed to check if it is an embedding issue.)
While you are at it, could you see if it works in Adobe Acrobat Reader? It certainly should but if you have the same problem with adobe as you do with foxit it would tell us something.
I assume the th couldn't be copied because it's a ligature, which makes the publication look better.
Ok, just be aware it makes it harder for others to quote your papers. (I originally intended to quote the last 3 paragraphs, but there were too many characters to fix so I cut it down to 1.)
Correct. This is a key piece of recommended differential technological progress, and this recommendation is stated explicitly in both "AI as a negative and positive factor in global risk" and in "Intelligence Explosion: Evidence and Import."
I see, so the argument wasn't meant to rule out the possibility that there are some sub-topics within WBE that we should accelerate (similar to how SIAI thinks there are sub-topics within de novo AI like decision theory that we should accelerate).
You are correct.
(Also: different people have different opinions within SIAI, but it's simplifying and not grossly inaccurate to sometimes talk about "what SIAI thinks.")
So, the workshop discussion (plus your "Intelligence Explosion" paper) lead to three possible approaches:
It seems really hard to differentially push for FAI. For example I've mostly stopped working on decision theory because it seems to help UFAI as much as FAI. The only safe topics within FAI that I can see are ethics (normative and meta) and meta-philosophy, which are not really things you can throw resources at. I'm much less familiar with WBE but naively I would think that there are more opportunities for research in WBE that don't contribute too much to neuromorphic AI.
Has anyone been working on these questions?
For example I've mostly stopped working on decision theory because it seems to help UFAI as much as FAI.
I think there are potential avenues of development of decision theory that might help FAI more than uFAI; I think maybe you should talk to Steve Rayhawk to see if he has any thoughts about this.
Anyway I praise your prudence, especially as it seems like a real logical possibility that AGI can't be engineered without first solving self-reference and logical uncertainty.
For example I've mostly stopped working on decision theory because it seems to help UFAI as much as FAI. The only safe topics within FAI that I can see are ethics (normative and meta) and meta-philosophy, which are not really things you can throw resources at.
I see many examples where ideas associated with decision theory (and surfacing from thinking about it) clarify philosophical or metaethical questions. For example, meaning of beliefs, of decisions, or observations, what influences decisions (and so where to look for preference), the way preference could be accessed through beliefs about it, all these things happen in context of certain setups of control and interpretation. I don't think it's possible to separate decision theory from FAI theory, and if we all stop developing FAI theory, we lose automatically.
The way I see it, even if we completely solve decision theory, there are so many other problems involved with building an FAI that the success probability (unless we first develop WBE or intelligence amplification) is still tiny. So working on decision theory is counterproductive if it raises the probability of UFAI coming before WBE/IA by even a small delta.
I don't think it's possible to separate decision theory from FAI theory, and if we all stop developing FAI theory, we lose automatically.
Of course I'm not suggesting we stop such work permanently, only until WBE or IA arrives. (Or when they are close enough that the probability of UFAI coming first is no longer significant, or if it becomes clear that de novo AI is much easier than WBE and IA and we have no choice but to push for FAI directly.)
What changes with WBE? Waiting for it already forsakes significant part of the expected future, but then what? It accelerates the physical timeline towards FAI and catastrophe both, and squeezing more FAI than catastrophe out of it (compared to pre-WBE ratio) requires rather unlikely circumstances.
The way I see it, even if we completely solve decision theory, there are so many other problems involved with building an FAI that the success probability (unless we first develop WBE or intelligence amplification) is still tiny.
Yes. (With the caveat that I'm regarding decision theory as a currently salient instrumental focus in pursuing the overall FAI problem, not a goal in itself.)
So working on decision theory is counterproductive if it raises the probability of UFAI coming before WBE/IA by even a small delta.
There is probability and then probability in given period of time. Raising probability of catastrophe before WBE doesn't necessarily raise it overall (for example, suppose catastrophe is inevitable, then moving it closer doesn't change the probability of it eventually occurring, and introducing a bit of probability of lack of catastrophe in exchange for moving the catastrophe branch closer does reduce the long term risk).
Lack of a catastrophe is not a stable state, it can be followed by a catastrophe, while implemented FAI is stable. You seem to be considering two alternatives: (1) reduction in obscure more immediate risk of UFAI, achieved by a few people deciding not to think about decision theory and hence not talking about it in public (this decision, as it's being made, doesn't take many out of the pool of those who would make progress towards UFAI); and (2) taking a small chance to get FAI right. Estimating FAI ever being solved as low, I think choosing (2) is the correct play. Alternatively, don't talk about the results openly, but work anyway (which is an important decision, but the community is too weak right now for closing itself off, and our present ideas are too feeble to pose significant risk; likely should turn to secrecy later).
(I notice that I don't understand this clearly enough, will have to reflect in more detail.)
What changes with WBE?
Did you read the part of the workshop report that talked about this?
this decision, as it's being made, doesn't take many out of the pool of those who would make progress towards UFAI
Getting decision theory "right enough" could be important for building a viable UFAI (or at least certain types of it, e.g., non-neuromorphic). There's reason to think for example that AIXI would fail due to incorrect decision theory (but people trying to make AIXI practical do not seem to realize this yet). Given that we seem to constitute a large portion of all people trying to get decision theory right for AI purposes, the effect of our decisions might be larger than you think.
Alternatively, don't talk about the results openly, but work anyway
Yes, but of course that reduces the positive effects of working on decision theory, so you might decide that you should do something else instead. For example I think that thinking about strategy and meta-philosophy might be better uses of my time. (Also, I suggest that keeping secrets is very hard so even this alternative of working in secret may be a net negative.)
What changes with WBE?
Did you read the part of the workshop report that talked about this?
Yes, and I agree, but it's not what I referred to. The essential part of the claim (as I accept it) is that given WBE, there exist scenarios where FAI can be developed much more reliably than in any feasible pre-WBE scenario. At the very least, dominating WBE theoretically allows to spend thousands of subjective years working on the problem, while in pre-WBE mode we have at most 150 and more likely about 50-80 years.
What I was talking about is probability of success. FAI (and FAI theory in particular, as a technology-independent component) is a race against AGI and other disasters, which become more likely as technology develops. In any given time interval, all else equal, completion of an AGI seems significantly more likely than of FAI. It's this probability of winning vs. losing the race in any given time that I don't see expected to change with the WBE transition. Just as FAI research gets more time, so is AGI research expected to get more time, unless somehow FAI researchers outrun everyone else for WBE resources, what I called "dominating WBE" above, but that's an unlikely feat, I don't have reasons for seeing that as more likely than just solving FAI pre-WBE.
In other words, we have two different low-probability events that bound success pre-WBE and post-WBE: solving the likely-too-difficult problem of FAI in a short time (pre-WBE), and outrunning "competing" AGI projects (post-WBE). If AGI is easy, pre-WBE is more important, because probability of surviving to post-WBE is then low. If AGI is hard, then FAI is hard too, and so we must rely on the post-WBE stage.
The gamble is on uncertainty about how hard FAI and AGI are. If they are very hard, we'll probably get to the WBE race. Otherwise, it's worth trying now, just in case it's possible to solve FAI earlier, or perhaps to develop the theory well enough to gain high-profile claim on dominating WBE and finishing the project before competing risks.
Just as FAI research gets more time, so is AGI research expected to get more time, unless somehow FAI researchers outrun everyone else for WBE resources, what I called "dominating WBE" above, but that's an unlikely feat, I don't have reasons for seeing that as more likely than just solving FAI pre-WBE.
In order for FAI to win pre-WBE, FAI has to get more resources than AGI (e.g., more, smarter researchers, computing power), but because FAI is much harder than AGI, it needs a large advantage. The "race for WBE" is better because it's a fairer one and you may only need to win by a small margin.
Also, if someone (who isn't necessarily an FAI group to start with) dominates WBE, they have no strong reason to immediately aim for AGI. What does it buy them that they don't already have? They can take the (subjective) time to think over the situation, and perhaps decide that FAI would be the best way to move forward.
In order for FAI to win pre-WBE, FAI has to get more resources than AGI (e.g., more, smarter researchers, computing power), but because FAI is much harder than AGI, it needs a large advantage. The "race for WBE" is better because it's a fairer one and you may only need to win by a small margin.
If FAI is much harder, WBE race has more potential for winning than pre-WBE race, but still low probability (getting more resources than all AI efforts is unlikely, and by the time the WBE race even begins, a lot is already lost).
Also, if someone (who isn't necessarily an FAI group to start with) dominates WBE, they have no strong reason to immediately aim for AGI. What does it buy them that they don't already have? They can take the (subjective) time to think over the situation, and perhaps decide that FAI would be the best way to move forward.
No strong reason but natural stupidity. This argues for developing enough theory pre-WBE to make deliberate delay in developing AGI respectable/likely to get traction.
The workshop report gave "a roughly 14% chance of win if de novo AI (Friendly AI or not) came first" so the two of us seem to be much more pessimistic than average. Do you think we should be updating in their direction, or vice versa? (Unless the workshop is counting "partial wins" like an AI that fills the universe with orgasmium, or what I called "Instrumentally Friendly AI"?)
I don't understand the reasoning that neuromorphic AI is likely to come before WBE (page 4). Sure, we'll have neuron emulations before WBE, but neuromorphic AI should be about as much harder than neurons as non-neuromorphic AI is than transistors. Our understanding of how the brain works will help some, but we've got a long ways to go and it is not a straightforward problem.
Neurons are already optimized to make intelligence out of; transistors are not. Also, we already have an example of how to put them together the right way, which we can draw inspiration from in our own designs. Some subsystems of the brain are simpler than others, so we will very likely be able to simulate those before we have WBE.
Here is a short new publication from the Singularity Institute, on the 2-day workshop that followed Singularity Summit 2011.
Note the new publication design. We are currently porting our earlier publications to this template, too.