Do Earths with slower economic growth have a better chance at FAI?

I was raised as a good and proper child of the Enlightenment who grew up reading The Incredible Bread Machine and A Step Farther Out, taking for granted that economic growth was a huge in-practice component of human utility (plausibly the majority component if you asked yourself what was the major difference between the 21st century and the Middle Ages) and that the "Small is Beautiful" / "Sustainable Growth" crowds were living in impossible dreamworlds that rejected quantitative thinking in favor of protesting against nuclear power plants.

And so far as I know, such a view would still be an excellent first-order approximation if we were going to carry on into the future by steady technological progress:  Economic growth = good.

But suppose my main-line projection is correct and the "probability of an OK outcome" / "astronomical benefit" scenario essentially comes down to a race between Friendly AI and unFriendly AI.  So far as I can tell, the most likely reason we wouldn't get Friendly AI is the total serial research depth required to develop and implement a strong-enough theory of stable self-improvement with a possible side order of failing to solve the goal transfer problem.  Relative to UFAI, FAI work seems like it would be mathier and more insight-based, where UFAI can more easily cobble together lots of pieces.  This means that UFAI parallelizes better than FAI.  UFAI also probably benefits from brute-force computing power more than FAI.  Both of these imply, so far as I can tell, that slower economic growth is good news for FAI; it lengthens the deadline to UFAI and gives us more time to get the job done.  I have sometimes thought half-jokingly and half-anthropically that I ought to try to find investment scenarios based on a continued Great Stagnation and an indefinite Great Recession where the whole developed world slowly goes the way of Spain, because these scenarios would account for a majority of surviving Everett branches.

Roughly, it seems to me like higher economic growth speeds up time and this is not a good thing.  I wish I had more time, not less, in which to work on FAI; I would prefer worlds in which this research can proceed at a relatively less frenzied pace and still succeed, worlds in which the default timelines to UFAI terminate in 2055 instead of 2035.

I have various cute ideas for things which could improve a country's economic growth.  The chance of these things eventuating seems small, the chance that they eventuate because I write about them seems tiny, and they would be good mainly for entertainment, links from econblogs, and possibly marginally impressing some people.  I was thinking about collecting them into a post called "The Nice Things We Can't Have" based on my prediction that various forces will block, e.g., the all-robotic all-electric car grid which could be relatively trivial to build using present-day technology - that we are too far into the Great Stagnation and the bureaucratic maturity of developed countries to get nice things anymore.  However I have a certain inhibition against trying things that would make everyone worse off if they actually succeeded, even if the probability of success is tiny.  And it's not completely impossible that we'll see some actual experiments with small nation-states in the next few decades, that some of the people doing those experiments will have read Less Wrong, or that successful experiments will spread (if the US ever legalizes robotic cars or tries a city with an all-robotic fleet, it'll be because China or Dubai or New Zealand tried it first).  Other EAs (effective altruists) care much more strongly about economic growth directly and are trying to increase it directly.  (An extremely understandable position which would typically be taken by good and virtuous people).

Throwing out remote, contrived scenarios where something accomplishes the opposite of its intended effect is cheap and meaningless (vide "But what if MIRI accomplishes the opposite of its purpose due to blah") but in this case I feel impelled to ask because my mainline visualization has the Great Stagnation being good news.  I certainly wish that economic growth would align with FAI because then my virtues would align and my optimal policies have fewer downsides, but I am also aware that wishing does not make something more likely (or less likely) in reality.

To head off some obvious types of bad reasoning in advance:  Yes, higher economic growth frees up resources for effective altruism and thereby increases resources going to FAI, but it also increases resources going to the AI field generally which is mostly pushing UFAI, and the problem arguendo is that UFAI parallelizes more easily.

Similarly, a planet with generally higher economic growth might develop intelligence amplification (IA) technology earlier.  But this general advancement of science will also accelerate UFAI, so you might just be decreasing the amount of FAI research that gets done before IA and decreasing the amount of time available after IA before UFAI.  Similarly to the more mundane idea that increased economic growth will produce more geniuses some of whom can work on FAI; there'd also be more geniuses working on UFAI, and UFAI probably parallelizes better and requires less serial depth of research.  If you concentrate on some single good effect on blah and neglect the corresponding speeding-up of UFAI timelines, you will obviously be able to generate spurious arguments for economic growth having a positive effect on the balance.

So I pose the question:  "Is slower economic growth good news?" or "Do you think Everett branches with 4% or 1% RGDP growth have a better chance of getting FAI before UFAI"?  So far as I can tell, my current mainline guesses imply, "Everett branches with slower economic growth contain more serial depth of cognitive causality and have more effective time left on the clock before they end due to UFAI, which favors FAI research over UFAI research".

This seems like a good parameter to have a grasp on for any number of reasons, and I can't recall it previously being debated in the x-risk / EA community.

EDIT:  To be clear, the idea is not that trying to deliberately slow world economic growth would be a maximally effective use of EA resources and better than current top targets; this seems likely to have very small marginal effects, and many such courses are risky.  The question is whether a good and virtuous person ought to avoid, or alternatively seize, any opportunities which come their way to help out on world economic growth.

EDIT 2:  Carl Shulman's opinion can be found on the Facebook discussion here.

174 comments, sorted by
magical algorithm
Highlighting new comments since Today at 12:56 AM
Select new highlight date
Moderation Guidelines: Reign of Terror - I delete anything I judge to be annoying or counterproductiveexpand_more

To stay with the lingo (also, is "arguendo" your new catchphrase?): There are worlds in which slower economic growth is good news, and worlds in which it's not. As to which of these contribute more probability mass, that's hard -- because the actual measure would be technological growth, for which economic growth can be a proxy.

However, I find it hard to weigh scenarios such as "because of stagnant and insufficient growth, more resources are devoted to exploiting the remaining inefficiencies using more advanced tech" versus "the worldwide economic upswing caused a flurry of research activities".

R&D, especially foundational work, is such a small part of worldwide GDP that any old effect can dominate it. For example, a "cold war"-ish scenario between China and the US would slow economic growth -- but strongly speedup research in high-tech dual-use technologies.

While we often think "Google" when we think tech research, we should mostly think DoD in terms of resources spent -- state actors traditionally dwarf even multinational corporations in research investments, and whether their investements are spurned or spurred by a slowdown in growth (depending on the non-specified cause of said slowdown) is anyone's guess.

R&D, especially foundational work, is such a small part of worldwide GDP that any old effect can dominate it.

(Note: I agree with this point.)

In binary choices trading depends entirely on just how much you realize the techniques and what broker you work! I for myself leafing through a lot of tactics but the most effective strategies was included with the publication http://bestoptionstrategies.net encourage to use. Broblemy with brokers is not rare, and they have to take proven only. It absolutely was there that collected tested. Many thanks for your attention, I am hoping to simply help in your problem)))

For example, a "cold war"-ish scenario between China and the US would slow economic growth -- but strongly speedup research in high-tech dual-use technologies.

Yes - I think we'd be in much better shape with high growth and total peace than the other way around. Corporations seem rather more likely to be satisfied with tool AI (or at any rate AI with a fixed cognitive algorithm, even if it can learn facts) than, say, a nation at war.

There are worlds in which slower economic growth is good news, and worlds in which it's not.

Indeed. The question of "would X be better" usually is shorthand for "would X be better, all else being equal", and since in this case X is an integrated quantity over basically all human activity it's impossible for all else to be equal. To make the question well defined you have to specify what other influences go into the change in economic growth. Even in the restricted question where we look at various ways that charity and activism might increase economic growth, it looks likely that different charities and different policy changes would have different effects on FAI development.

So how would working to decrease US military spending rank as an effective altruist goal then? I'd guess most pro-economic-growth EAs are also in favor of it.

Related questions:

1) Do Earths with dumber politicians have a better chance at FAI?

2) Do Earths with anti-intellectual culture have a better chance at FAI?

3) Do Earths with less missionary rationalism have a better chance at FAI?

4) How much time should we spend pondering questions like (1)-(3)?

5) How much time should we spend pondering questions like (4)?

1) Do Earths with dumber politicians have a better chance at FAI?

How much dumber? If we can make politicians marginally dumber in a way that slows down economic growth, or better yet decreases science funding while leaving economic growth intact, without this causing any other marginal change in stupid decisions relevant to FAI vs. UFAI, then sure. I can't think of any particular marginal changes I expect because I already expect almost all such decisions to be made incorrectly, but I worry that this is only a failure of my imagination on my part - that with even dumber politicians, things could always become unboundedly worse in ways I didn't even conceive.

2) Do Earths with anti-intellectual culture have a better chance at FAI?

This seems like essentially the same question as above.

Do Earths with less missionary rationalism have a better chance at FAI?

No. Missionary rationalists are a tiny fraction of world population who contribute most of FAI research and support.

So far as I can tell, the most likely reason we wouldn't get Friendly AI is the total serial research depth required to develop and implement a strong-enough theory of stable self-improvement with a possible side order of failing to solve the goal transfer problem.

I'm curious that you seem to think the former problem is harder or less likely to be solved than the latter. I've been thinking the opposite, and one reason is that the latter problem seems more philosophical and the former more technical, and humanity seems to have a lot of technical talent that we can eventually recruit to do FAI research, but much less untapped philosophical talent.

Also as another side note, I don't think we should be focusing purely on the "we come up with a value-stable architecture and then the FAI will make a billion self-modifications within the same general architecture" scenario. Another possibility might be that we don't solve the stable self-improvement problem at all, but instead solve the value transfer problem in a general enough way that the FAI we build immediately creates an entirely new architecture for the next generation FAI and transfer its values to its creation using our solution, and this happens just a few times. (The FAI doesn't try to make a billion self-modifications to itself because, just like us, it knows that it doesn't know how to safely do that.) (Cousin_it make a similar comment earlier.)

In all I can see three arguments for prioritizing the value transfer problem over the stable self-improvement problem: 1) the former seems harder so we need to get started on it earlier; 2) we know the former definitely needs to be solved whereas the latter may not need to be; 3) the former involves work that's less useful for building UFAI.

(On the main topic of the post, I've been assuming, without having put too much thought into it, that slower economic growth is good for eventually getting FAI. Now after reading the discussions here and on Facebook I realize that I haven't put enough thought into it and perhaps should be less certain about it than I was.)

On the main topic of the post, I've been assuming, without having put too much thought into it, that slower economic growth is good for eventually getting FAI.

Here's an attempt to verbalize why I think this, which is a bit different from Eliezer's argument (which I also buy to some extent). First I think UFAI is much easier than FAI and we are putting more resources into the former than the latter. To put this into numbers for clarity, let's say UFAI takes 1000 units of work, and FAI takes 2000 units of work, and we're currently putting 10 units of work into UFAI per year, and only 1 unit of work per year into FAI. If we had a completely stagnant economy, with 0% growth, we'd have 100 years to do something about this, or for something to happen to change this, before it's too late. If the economy was instead growing at 5% per year, and this increased both UFAI and FAI work by 5% per year, the window of time "for something to happen" shrinks to about 35 years. The economic growth might increase the probability per year of "something happening" but it doesn't seem like it would be enough to compensate for the shortened timeline.

Also: Many likely reasons for something to happen about this center around, in appropriate generality, the rationalist!EA movement. This movement is growing at a higher exponent than current economic growth.

I think this is the strongest single argument that economic growth might currently be bad. However even then what matters is the elasticity of movement growth rates with economic growth rates. I don't know how we can measure this; I expect it's positive and less than one, but I'm rather more confident about that lower bound than the upper bound.

I don't have an answer for the question, but I note that the hypothetical raises the possibility of an anthropic explanation for twenty-first century recessions. So if you believe that the Fed is run by idiots who should have , consider the possibility that in branches where the Fed did in fact , the world now consists of computronium.

I find this especially compelling in light of Japan's two "lost decades" combined with all the robotics research for which Japan is famous. Obviously the anthropic hypothesis requires the most stagnation in nations which are good at robots and AI.

I don't have an answer for the question

I hope we can all agree that in discussions on LW this should by no means be regarded as a bad thing.

Can we put a lid on this conflation of subjective probability with objective quantum branching please? A deterministic fair coin does not split the world, and neither would a deterministic economic cycle. Or are we taking seriously the possibility that the course of the economy is largely driven by quantum randomness?

EDIT: actually I just noticed that small quantum fluctuations from long ago can result in large differences between branches today. At that point I'm confused about what the anthropics implies we should see, so please excuse my overconfidence above.

Can we put a lid on this conflation of subjective probability with objective quantum branching please? A deterministic fair coin does not split the world, and neither would a deterministic economic cycle. Or are we taking seriously the possibility that the course of the economy is largely driven by quantum randomness?

I'm certainly taking it seriously, and am somewhat surprised that you're not. Some ways small-sized effects (most likely to "depend" of quantum randomness) can eventually have large-scale impacts:

  • DNA Mutations
  • Which sperm gets to the egg
  • The weather
  • Soft errors from cosmic rays or thermal radiation

Or are we taking seriously the possibility that the course of the economy is largely driven by quantum randomness?

Isn't everything?

This comment was banned, which looked to me like a probable accident with a moderator click, so I unbanned it. If I am in error can whichever mod PM me after rebanning it?

Naturally if this was an accident, it must have been a quantum random one.

Whether or not quantum randomness drives the course of the economy it's still a really good idea to stop conflating subjective probability and the corresponding notion of possible worlds with quantum/inflationary/whatever many world theories. Rolf's comment doesn't actually do this: I read him as speaking entirely about the anthropic issue. Eliezer, on the other hand, totally is conflating them in the original post.

I understand that there are reasons to think anthropic issues play an essential role in the assignment of subjective probabilities, especially at a decision theoretic level. But given a) subjective uncertainty over whether or not many-worlds is correct, b) our ignorance of how the Born probability rule figures into the relationship and c) the way anthropics skews anticipated experiences I am really suspicious that anyone here is able to answer the question:

"Do you think Everett branches with 4% or 1% RGDP growth have a better chance of getting FAI before UFAI"?

People are actually answering

"Do you think possible worlds with 4% or 1% RGDP growth have a better chance of getting FAI before UFAI"?

which is not obviously the same thing.

You don't need quantum many worlds for this kind of speculation: e.g. a spatially infinite universe would also do the trick.

As I said:

Whether or not quantum randomness drives the course of the economy it's still a really good idea to stop conflating subjective probability and the corresponding notion of possible worlds with quantum/inflationary/whatever many world theories.

I agree, subjective uncertainty isn't the same as quantum uncertainty!

On the other hand, there have been rumors that coinflips are not deterministic. See here.

My comment was not intended in full seriousness. :)

It is not at all clear to me that this hypothesis shouldn't be taken seriously. It's not clear to me that it should be, either!

It also explains why the dot com boom had to burst,

why Charles Babbage never built his Analytical Engine,

why Archimedes was killed, and Antikythera mechanism drowned in the sea,

why most children in our culture hate maths, and why internet is mostly used for chatting, games, and porn.

It unfortunately also explains

  • why Alan Turing never published his work on the theory of computation
  • why all the projects in the 1950s aimed at making general-purpose computers got cancelled for complex political reasons no one understood
  • why that big earthquake killed everyone at the Dartmouth Conference, tragically wiping out almost the entire nascent field of AI
  • why all attempts at constructing integrated circuits mysteriously failed
  • why progress abruptly stopped following Moore's law in the early 1980s
  • why no one has ever been able to make computer systems capable of beating grandmasters at chess, questioning Jeopardy answers, searching huge databases of information, etc.

All of which are true in other possible worlds, which for all we know may have a greater amplitude than ours. That we are alive does not give us any information on how probable we are, because we can't observe the reference class. For all we know, we're one of those worlds that skate very, very close to the edge of disaster, and the two recessions of the aughts are the only thing that have kept us alive; but those recessions were actually extremely unlikely, and the "mainline" branches of humanity, the most probable ones, are alive because the Cuban War of 1963 set the economy back to steam and horses. (To be sure, they have their problems, but UFAI isn't among them.)

Note that, if you take many-worlds seriously, then in branches where UFAI is developed, there will still be some probability of survival due to five cosmic rays with exactly the right energies hitting the central CPU at just the right times and places, causing SkyNet to divide by zero instead of three. But the ones who survive due to that event won't be very probable humans. :)

If most copies of me died in the shooting but I survived, I should expect to find that I survived for only one reason, not for multiple independent reasons. Perhaps the killer's gun jammed at the crucial moment, or perhaps I found a good place to hide, but not both.

On the other hand, if you are being shot at repeatedly and survive a long time, you should expect there to be lots of reasons (or one reason with very broad scope -- maybe everyone's guns were sabotaged in a single operation, or maybe they've been told to let you live, or a god is looking out for you). And it's only in that sort of situation that anthropic "explanations" would be in any way sensible.

It's always true enough to say "well, of course I find myself still alive because if I weren't I wouldn't be contemplating the fact that I'm still alive". But most of the time this is really uninteresting. Perhaps it always is.

The examples given in this thread seem to me to call out for anthropic explanations to much the same extent as does the fact that I'm over 40 years old and not dead yet.

This just prompted me to try to set a subjective probability that quantum immortality works, so e.g. if I remember concluding that it was 5% likely at 35 and find myself still alive at 95, I will believe in quantum immortality (going by SSA tables).

I'm currently finding this subjective probability too creepy to actually calculate.

I suggest giving some thought first to exactly what "believing in quantum immortality" really amounts to.

To me, it means expecting to experience the highest-weighted factorization of the hamiltonian that contains a conscious instantiation of me, no matter how worse-than-death that branch may be.

I think you should analyse further. Expecting conditional on still being alive? Surely you expect that even without "quantum immortality". Expecting to find yourself still alive, and experience that? Again, what exactly do you mean by that? What does it mean to expect to find yourself still alive? (Presumably not that others will expect to find you still alive in any useful sense, because with that definition you don't get q.i.)

I expect there are Everett branches in which you live to 120 as a result of a lot of good luck (or, depending on what state you're in, bad luck). Almost equivalently, I expect there's a small but nonzero probability that you live to 120 as a result of a lot of luck. { If you live to 120 / In those branches where you live to 120 } you will probably have experienced a lot of surprising things that enabled your survival. None of this is in any way dependent on quantum mechanics, still less on the many-worlds interpretation.

It seems to me that "believing in quantum immortality" is a matter of one's own values and interpretive choices, much more than of any actual beliefs about how the world is. But I may be missing something.

I should perhaps be more clear that I'm not distinguishing between "MWI and functionalism are true" and "quantum immortality works." That is, if "I" consciously experience dying, and my consciouness ceases, but "I" go on experiencing things in other everett branches, I'm counting that as QI.

Expecting conditional on still being alive? Surely you expect that even without "quantum immortality"...What does it mean to expect to find yourself still alive?

I'm currently making observations consistent with my own existence. If I stop making that kind of observation, I consider that no longer being alive.

Going again with the example of a 35 year old: Conditional on having been born, I have a 96% chance of still being alive. So whatever my prior on QI, that's far less than a decibel of evidence in favor of it. Still, ceteris paribus, it's more likely than it was at age 5.

Sure. But I'm not sure I made the point I was trying to make as clearly as I hoped, so I'll try again.

Imagine two possible worlds. In one of them, QM works basically as currently believed, and the way it does this is exactly as described by MWI. In the other, there is at every time a single kinda-classical-ish state of the world, with Copenhagen-style collapses or something happening as required.

In either universe it is possible that you will find yourself still alive at 120 (or much more) despite having had plenty of opportunities to be killed off by accident, illness, etc. In either universe, the probability of this is very low (which in the former case means most of the measure of where we are now ends up with you dead earlier, and in the latter means whatever exactly probability means in a non-MWI world). In either universe, every observation you make will show yourself alive, however improbable that may seem.

How does observing yourself still alive at 150 count as evidence for MWI, given all that?

What you mustn't say (so it seems to me): "The probability of finding myself alive is very low on collapse theories and high on MWI, so seeing myself still alive at 150 is evidence for MWI over collapse theories". If you mean the probability conditional on you making the observation at age 150, it's 1 in both cases. If you mean the probability not conditional on that, it's tiny in both cases. (Assuming arguendo that Pr(nanotech etc. makes lots of people live to be very old by then) is negligible.) The same applies if you try to go halfway and take the probability simply conditional on you making the observation: MWI or no MWI, only a tiny fraction of observations you make will be at age 150.

In either universe it is possible that you will find yourself still alive at 120

In the MWI-universe, it is probable at near unity that I will find myself still alive at 120. In the objective collapse universe, there's only a small fraction of a percent chance that I'll find myself alive at 120. In the objective collapse universe, every observation I make will show myself alive--but there's only a fraction of a percent of a chance that I'll make an observation that shows my age as 120.

If you mean the probability conditional on you making the observation at age 150, it's 1 in both cases.

The probability of my making the observation "I am 150 years old," given objective collapse, is one of those probabilities so small it's dominated by "stark raving mad" type scenarios. Nobody you've ever known has made that observation; neither has anybody they know. How can this not be evidence?

What's the observation you're going to make that has probability near-1 on MWI and probabilty near-0 on collapse -- and probability given what?

"I'm alive at 120, here and now" -- that has small probability either way. (On most branches of the wavefunction that include your present self, no version of you gets to say that. Ignoring, as usual, irrelevant details involving positive singularities, very large universes, etc.)

"90 years from now I'll still be alive" (supposing arguendo that you're 30 now) -- that has small probability either way.

"I'm alive at 120, conditional on my still being alive at 120" -- that obviously has probability 1 either way.

"On some branch of the wavefunction I'm still alive at 120" -- sure, that's true on MWI and (more or less by definition) false on a collapse interpretation; but it's not something you can observe. It corresponds exactly to "With nonzero probability I'm still alive at 120", which is true on collapse.

"90 years from now I'll still be alive" (supposing arguendo that you're 30 now) -- that has small probability either way.

This is the closest one. However, that's not an observation, it's a prediction. The observation is "90 years ago, I was 30." That's an observation that almost certainly won't be made in a collapse-based world; but will be made somewhere in an MWI world.

"I'm alive at 120, here and now" -- that has small probability either way. (On most branches of the wavefunction that include your present self, no version of you gets to say that.)

"small probability either way" only applies if I want to locate myself precisely, within a branch as well as within a possible world. If I only care about locating myself in one possible world or the other, the observation has a large probability in MWI.

It seems to me that "believing in quantum immortality" is a matter of one's own values and interpretive choices, much more than of any actual beliefs about how the world is.

You are correct.

If Charles Babbage had built his analytic engine, then that would seem to me to have gotten programming started long earlier, such that FAI work would in turn start much sooner, and so we'd have no hardware overhang to worry about. Imagine if this conversation were taking place with 1970's technology.

See here and here.

This position seems unlikely to me at face value. It relies on a very long list of claims, and given the apparently massive improbability of the conjunction, there is no way this consideration is going to be the biggest impact of economic progress:

  1. The most important determinant of future welfare is whether you get FAI or UFAI (this presupposes a relatively detailed model of how AI works, of what the danger looks like, etc.)
  2. This will happen quite soon, and relevant AI work is already underway.
  3. The main determinant of FAI vs. UFAI is whether an appropriate theoretical framework for goal-stability is in place.
  4. As compared to UFAI work, the main difficulty for developing such a framework for goal-stability is the serial depth of the problem.
  5. A 1% boost in economic activity this year has a non-negligible effect on the degree of parallelization of relevant AI work.

I don't see how you can defend giving any of those points more than 1/2 probability, and I would give the conjunction less than 1% probability. Moreover, even in this scenario, the negative effect from economic progress is quite small. (Perhaps a 1% increase in sustained economic productivity makes the future 0.1% worse if this story is true, and each year of a 1% increase in economic productivity makes the future 0.002% worse?)

So on balance it seems to me like this would say that a 1% increase in economic productivity would make the future 0.00002% worse? That is a ridiculously tiny effect; even if you couldn't see any particular reason that economic progress helped or hurt I think your prior should expect other effects to dominate, and you can use other considerations to get a handle on the sign. I guess you think that this is an underestimate, but I would be interested to know where you disagree.

I would guess that the positive effect from decreasing the cumulative risk of war alone are several orders of magnitude higher than that. I know you think that a world war that killed nearly everyone might be positive rather than negative, but in that case slowing down economic progress would still have positive effects that are orders of magnitude larger than the effect on FAI parallelization.

Even without doing any calculations, it is extraordinarily hard to imagine that the difference between "world at war" and "world at peace" is less than the difference between "world with slightly more parallelization in AI work" and "world with slightly less parallelization;" almost everyone would disagree with you, and as far as I can tell their reasons seem better than yours. Similarly, it is hard to imagine that the interaction between generational turnover and economic activity wouldn't swamp this by several orders of magnitude.

General remark: At some point I need to write a post about how I'm worried that there's an "unpacking fallacy" or "conjunction fallacy fallacy" practiced by people who have heard about the conjunction fallacy but don't realize how easy it is to take any event, including events which have already happened, and make it look very improbable by turning one pathway to it into a large series of conjunctions. E.g. I could produce a long list of things which allegedly have to happen for a moon landing to occur, some of which turned out to not be necessary but would look plausible if added to the list ante facto, with no disjunctive paths to the same destination, and thereby make it look impossible. Generally this manifests when somebody writes a list of alleged conjunctive necessities, and I look over the list and some of the items seem unnecessary (my model doesn't go through them at all), obvious disjunctive paths have been omitted, the person has assigned sub-50% probability to things that I see as mainline 90% probabilities, and conditional probabilities when you assume the theory was right about 1-N would be significantly higher for N+1. Most of all, if you imagine taking the negation of the assertion and unpacking it into a long list of conjunctive probabilities, it would look worse - there should be a name for the problem of showing that X has weak arguments but not considering that ~X has even weaker arguments. Or on a meta level, since it is very easy to make things look more conjunctive, we should perhaps not be prejudiced against things which somebody has helpfully unpacked for us into a big conjunction, when the core argument still seems pretty simple on some level.

When I look over this list, my reaction is that:

(1) is a mainline assumption with odds of 5:1 or 10:1 - of course future intergalactic civilization bottlenecks through the goals of a self-improving agency, how would you get to an intergalactic civilization without that happening? If this accounts for much of our disagreement then we're thinking about entirely different scenarios, and I'm not sure how to update from your beliefs about mostly scenario B to my beliefs about mostly scenario A. It makes more sense to call (1) into question if we're really asking about global vs. local, but then we get into the issue of whether global scenarios are mostly automatic losses anyway. If (1) is really about whether we should be taking into account a big chunk of survivable global scenarios then this goes back to a previous persistent disagreement.

(2) I don't see the relevance - why does a long time horizon vs. a short time horizon matter? 80 years would not make me relax and say that we had enough serial depth, though it would certainly be good news ceteris paribus, there's no obvious threshold to cross.

Listing (3) and (4) as separate items was what originally made my brain shout "unpacking fallacy!" There are several subproblems involved in FAI vs. UFAI, of which the two obvious top items are the entire system being conducive to goal stability through self-improvement which may require deducing global properties to which all subsystems must be conducive, and the goal loading problem. These both seem insight-heavy which will require serial time to solve. The key hypothesis is just that there are insight-heavy problems in FAI which don't parallelize well relative to the wide space of cobbled-together designs which might succeed for UFAI. Odds here are less extreme than for (1) but still in the range of 2:1-4:1. The combined 3-4 issue is the main weak point, but the case for "FAI would parallelize better than UFAI" is even weaker.

(5) makes no sense to ask as a conditionally independent question separate from (1); if (1) is true then the only astronomical effects of modern-day economic growth are whatever effects that growth has on AI work, and to determine if economic growth is qualitatively good or bad, we ask about the sign of the effect neglecting its magnitude. I suppose if the effect were trivial enough then we could just increase the planet's growth rate by 5% for sheer fun and giggles and it would have no effect on AI work, but this seems very unlikely; a wealthier planet will ceteris paribus have more AI researchers. Odds of 10:1 or better.

On net, this says that in my visualization the big question is just "Does UFAI parallelize better than FAI, or does FAI parallelize better than UFAI?" and we find that the case for the second clause is weaker than the first; or equivalently "Does UFAI inherently require serial time more than FAI requires serial time?" is weaker than "Does FAI inherently require serial time more than UFAI requires serial time?" This seems like a reasonable epistemic state to me.

The resulting shove at the balance of the sign of the effect of economic growth would have to be counterbalanced by some sort of stronger shove in the direction of modern-day economic growth having astronomical benefits. And the case for e.g. "More econ growth means friendlier international relations and so they endorse ideal Y which leads them to agree with me on policy Z" seems even more implausible when unpacked into a series of conjunctions. Lots of wealthy people and relatively friendly nations right now are not endorsing policy Z.

To summarize and simplify the whole idea, the notion is:

Right now my estimate of the sign of the astronomical effect of modern-day economic growth is dominated by a 2-node conjunction of, "Modern-day econ growth has a positive effect on resources into both FAI and UFAI" and "The case for FAI parallelizing better than UFAI is weaker than the converse case". For this to be not true requires mainly that somebody else demonstrate an effect or set of effects in the opposite direction which has better net properties after its own conjunctions are taken into account. The main weakness in the argument and lingering hope that econ growth is good, isn't that the original argument is very conjunctive, but rather it's that faster econ growth seems like it should have a bunch of nice effects on nice things and so the disjunction of other econ effects might conceivably swing the sign the other way. But it would be nice to have at least one plausible such good effect without dropping our standards so low that we could as easily list a dozen equally (im)plausible bad effects.

Even without doing any calculations, it is extraordinarily hard to imagine that the difference between "world at war" and "world at peace" is less than the difference between "world with slightly more parallelization in AI work" and "world with slightly less parallelization;"

With small enough values of 'slightly' obviously the former will have a greater effect, the question is the sign of that effect; also it's not obvious to me that moderately lower amounts of econ growth lead to world wars, and war seems qualitatively different in many respects from poverty. I also have to ask if you are possibly maybe being distracted by the travails of one planet as a terminal value, rather than considering that planet's instrumental role in future galaxies.

At some point I need to write a post about how I'm worried that there's an "unpacking fallacy" or "conjunction fallacy fallacy" practiced by people who have heard about the conjunction fallacy but don't realize how easy it is to take any event, including events which have already happened, and make it look very improbable by turning one pathway to it into a large series of conjunctions.

Related: There's a small literature on what Tversky called "support theory," which discusses packing and unpacking effects: Tversky & Koehler (1994); Ayton (1997); Rottenstreich & Tversky (1997); Macchi et al. (1997); Fox & Tversky (1998); Brenner & Koehler (1999); Chen et al. (2001); Boven & Epley (2003); Brenner et al. (2005); Bligin & Brenner (2008).

Luke asked me to look into this literature for a few hours. Here's what I found.

The original paper (Tversky and Koehler 1994) is about disjunctions, and how unpacking them raises people’s estimate of the probaility. So for example, asking people to estimate the probability someone died of “heart disease, cancer, or other natural causes” yields a higher probability estimate than if you just ask about “natural causes.”

They consider the hypothesis this might be because they take the researcher’s apparent emphasis as evidence that’s it’s more likely, but they tested & disconfirmed this hypothesis by telling people to take the last digit of their phone number and estimate the percentage of couples that have that many children. Percentages sum to greater than 1.

Finally, they check whether experts are vulnerable to this bias by doing an experiment similar to the first experiment, but using physicians at Stanford University as the subjects and asking them about a hypothetical case of a woman admitted to an emergency room. They confirmed that yes, experts are vulnerable to this mistake too.

This phenomenon is known as “subadditivity.” A subsequent study (RottenStreich and Tversky 1997) found that subadditivity can even occur when dealing with explicit conjunctions. Macci et al. (1999) found evidence of superadditivity: ask some people how probable it is that the freezing point of alcohol is below that of gasoline, other people how probable it is that the freezing point of gasoline is below that of alcohol, average answers sum to less than 1.

Other studies try to refine the mathematical model of how people make judgements in these kinds of cases, but the experiments I’ve described are the most striking empirical results, I think. One experiment that talks about unpacking conjunctions (rather than disjunctions, like the experiments I’ve described so far) is Boven and Epley (2003, particularly their first experiment, where they ask people how much an oil refinery should be punished for pollution. This pollution is described either as leading to an increase in “asthma, lung cancer, throat cancer, or all varieties of respiratory diseases,” or just as leading to an increase in “all varieties of respiratory diseases.” In the first condition, people want to punish refinery more. But, in spite of being notably different from previous unpacking experiments, still not what Eliezer was talking about.

Below are some other messy notes I took:

http://commonsenseatheism.com/wp-content/uploads/2013/10/Fox-Tversky-A-belief-based-account-of-decision-under-uncertainty.pdf Uses support theory to develop account of decision under uncertainty.

http://commonsenseatheism.com/wp-content/uploads/2013/10/Brenner-Koehler-Subjective-probability-of-disjunctive-hypotheses-local-weight-models-for-decomposition-and-evidential-support.pdf Something about local weights; didn't look at this one much.

http://commonsenseatheism.com/wp-content/uploads/2013/10/Chen-et-al-The-relation-between-probability-and-evidence-judgment-an-extension-of-support-theory.pdf Tweaking math behind support theory to allow for superadditivity.

http://commonsenseatheism.com/wp-content/uploads/2013/10/Brenner-et-al-Modeling-patterns-of-probability-calibration-with-random-support-theory.pdf Introduces notion of random support theory.

http://bear.warrington.ufl.edu/brenner/papers/bilgin-brenner-jesp08.pdf Unpacking effects weaker when dealing with near future as opposed to far future.

Other articles debating how to explain basic support theory results: http://bcs.siu.edu/facultypages/young/JDMStuff/Sloman%20(2004)%20unpacking.pdf http://aris.ss.uci.edu/~lnarens/Submitted/problattice11.pdf http://eclectic.ss.uci.edu/~drwhite/pw/NarensNewfound.pdf

What this shows is that people are inconsistent in a certain way. If you ask them the same question in two different ways (packed vs. unpacked) you get different answers. Is there any indication of which is the better way to ask the question, or whether asking it some other way is better still? Without an answer to this question, it's unclear to me whether we should talk about an "unpacking fallacy" or a "failure to unpack fallacy".

Here's a handy example discussion of related conjunction issues from the Project Cyclops report:

We have outlined the development of technologically competent life on Earth as a succession of steps to each of [which] we must assign an a priori probability less than unity. The probability of the entire sequence occurring is the product of the individual (conditional) probabilities. As we study the chain of events in greater detail we may become aware of more and more apparently independent or only slightly correlated steps. As this happens, the a priori probability of the entire sequence approaches zero, and we are apt to conclude that, although life indeed exists here, the probability of its occurrence elsewhere is vanishingly small.

The trouble with this reasoning is that it neglects alternate routes that converge to the same (or almost the same) end result. We are reminded of the old proof that everyone has only an infinitesimal chance of existing. One must assign a fairly small probability to one's parents and all one's grandparents and (great)^n-grandparents having met and mated. Also one must assign a probability on the order of 2^-46 to the exact pairing of chromosomes arising from any particular mating. When the probabilities of all these independent events that led to a particular person are multiplied, the result quickly approaches zero. This is all true. Yet here we all are. The [explanation] is that, if an entirely different set of matings and fertilizations had occurred, none of "us" would exist, but a statistically indistinguishable generation would have been born, and life would have gone on much the same.

Regarding the "unpacking fallacy": I don't think you've pointed to a fallacy here. You have pointed to a particular causal pathway which seems to be quite specific, and I've claimed that this particular causal pathway has a tiny expected effect by virtue of its unlikeliness. The negation of this sequence of events simply can't be unpacked as a conjunction in any natural way, it really is fundamentally a disjunction. You might point out that the competing arguments are weak, but they can be much stronger in the cases were they aren't predicated on detailed stories about the future.

As you say, even events that actually happened can also be made to look quite unlikely. But those events were, for the most part, unlikely ex ante. This is like saying "This argument can suggest that any lottery number probably wouldn't win the lottery, even the lottery numbers that actually won!"

If you had a track record of successful predictions, or if anyone who embraced this view had a track record of successful predictions, maybe you could say "all of these successful predictions could be unpacked, so you shouldn't be so skeptical of unpackable arguments." But I don't know of anyone with a reasonably good predictive record who takes this view, and most smart people seem to find it ridiculous.

  1. I don't understand your argument here. Yes, future civilization builds AI. It doesn't follow that the value of the future is first determined by what type of AI they build (they also build nanotech, but the value of the future isn't determined by the type of nanotech they build, and you haven't offered a substantial argument that discriminates between the cases). There could be any number of important events beforehand or afterwards; there could be any number of other important characteristics surrounding how they build AI which influence whether the outcome is positive or negative.

  2. Do you think the main effects of economic progress in 1600 were on the degree of parallelization in AI work? 1800? The magnitude of the direct effects of economic progress on AI work depends on how close the economic progress is to the AI work; as the time involved gets larger, indirect effects come to dominate.

  3. You have a specific view, that there is a set of problems which need to be solved in order to make AI friendly, and that these problems have some kind of principled relationship to the problems that seem important to you now. This is as opposed to e.g. "there are two random approaches to AI, one of which leads to good outcomes and one of which leads to bad outcomes," or "there are many approaches to AI, and you have to think about it in advance to figure out which lead to good outcomes" or "there is a specific problem that you can't have solved by the time you get to AI if you want to to have a positive outcome" or an incredible variety of alternative models. The "parallelization is bad" argument doesn't apply to most of these models, and in some you have "parallelization is good."

  4. Even granting that your picture of AI vs. FAI is correct, and there are these particular theoretical problems that need to be solved, it is completely unclear that more people working in the field makes things worse. I don't know why you think this follows from 3 or can be sensibly lumped with 3, and you don't provide an argument. Suppose I said "The most important thing about dam safety is whether you have a good theoretical understanding the dam before building it" and you said "Yes, and if you increase the number of people working on the dam you are less likely to understand it by the time it gets built, because someone will stumble across an ad hoc way to build a dam." This seems ridiculous both a priori and based on the empirical evidence. There are many possible models for the way that important problems in AI get solved, and you seem to be assuming a particular one.

  5. Suppose that I airdrop in a million knowledge workers this year and they leave next year, corresponding to an exogenous boost in productivity this year. You are claiming that this obviously increases the degree of parallelization on relevant AI work. This isn't obvious, unless a big part of the relevant work is being done today (which seems unlikely, casually?)

I agree that I've only argued that your argument has a tiny impact; it could still dominate if there was literally nothing else going on. But even granting 1-5 there seem to be other big effects from economic growth.

The case in favor of growth seems to be pretty straightforward; I linked to a blog post in the last comment. Let me try to make the point more clearly:

Increasing economic activity speeds up a lot of things. Speeding up everything is neutral, so the important point is the difference between what it speeds up and what it doesn't speed up. Most things people are actually trying to do get sped up, while a bunch of random things (aging and disease, natural disasters, mood changes) don't get sped up. Lots of other things get sped up but significantly less than 1-for-1, because they have some inputs that get sped up and some that don't (accidents of all kinds, conflicts of all kinds, resource depletion). Given that things people are trying to do get sped up, and the things that happen which they aren't trying to do get sped up less, we should expect the effect to be to positive, as long as people are trying to do good things.

What's a specific relevant example of something people are trying to speed up / not speed up besides AGI (= UFAI) and FAI? You pick out aging, disease, and natural disasters as not-sped-up but these seem very loosely coupled to astronomical benefits.

Increasing capital stocks, improving manufacturing, improving education, improving methodologies for discourse, figuring out important considerations. Making charity more efficient, ending poverty. Improving collective decision-making and governance. All of the social sciences. All of the hard sciences. Math and philosophy and computer science. Everything that everyone is working on, everywhere in the world.

I picked out conflict, accidents, and resource depletion as not being sped up 1-for-1, i.e. such that a 1% boost in economic activity corresponds to a <1% boost in those processes. Most people would say that war and accidents account for many bad things that happen. War is basically defined by people making decisions that are unusually misaligned with aggregate welfare. Accidents are basically defined by people not getting what they want. I could have lumped in terrorism, and then accounted for basically all of the ways that we can see things going really badly in the present day.

You have a particular story about how a bad thing might happen in the future. Maybe that's enough to conclude the future will be entirely unlike the present. But it seems like (1) that's a really brittle way to reason, however much you want to accuse its detractors of the "unpacking fallacy," and most smart people take this view, and (2) even granting almost all of your assumptions, it's pretty easy to think of scenarios where war, terrorism, or accidents are inputs into AI going badly, or where better education, more social stability, or better decision-making are inputs into AI going well. People promoting these positive changes are also working against forces that wouldn't be accelerated, like people growing old and dying and thereby throwing away their accumulated human capital, or infrastructure being stressed to keep people alive, etc. etc.

Increasing capital stocks, improving manufacturing, improving education, improving methodologies for discourse, figuring out important considerations. Making charity more efficient, ending poverty. Improving collective decision-making and governance. All of the social sciences. All of the hard sciences. Math and philosophy and computer science. Everything that everyone is working on, everywhere in the world.

How is an increased capital stock supposed to improve our x-risk / astronomical benefit profile except by being an input into something else? Yes, computer science benefits, that's putatively the problem. We need certain types of math for FAI but does math benefit more from increased capital stocks compared to, say, computing power? Which of these other things are supposed to save the world faster than computer science destroys it, and how? How the heck would terrorism be a plausible input into AI going badly? Terrorists are not going to be the most-funded organizations with the smartest researchers working on AGI (= UFAI) as opposed to MIT, Google or Goldman Sachs.

Does your argument primarily reduce to "If there's no local FOOM then economic growth is a good thing, and I believe much less than you do in local FOOM"? Or do you also think that in local FOOM scenarios higher economic growth now expectedly results in a better local FOOM? And if so is there at least one plausible specific scenario that we can sketch out now for how that works, as opposed to general hopes that a higher economic growth exponent has vague nice effects which will outweigh the shortening of time until the local FOOM with a correspondingly reduced opportunity to get FAI research done in time? When you sketch out a specific scenario, this makes it possible to point out fragile links which conjunctively decrease the probability of that scenario, and often these fragile links generalize, which is why it's a bad idea to keep things vague and not sketch out any concrete scenarios for fear of the conjunction fallacy.

It seems to me that a lot of your reply, going by the mention of things like terrorism and poverty, must be either prioritizing near-term benefits over the astronomical future, or else being predicated on a very different model from local FOOM. We already have a known persistent disagreement on local FOOM. This is an important modular part of the disagreement on which other MIRIfolk do not all line up on one side or another. Thus I would like to know how much we disagree about expected goodness of higher econ growth exponents given local FOOM, and whether there's a big left over factor where "Paul Christiano thinks you're just being silly even assuming that a FOOM is local", especially if this factor is not further traceable to a persistent disagreement about competence of elites. It would then be helpful to sketch out a concrete scenario corresponding to this disagreement to see if it looks even more fragile and conjunctive.

(Note that e.g. Wei Dai also thought it was obviously true that faster econ growth exponents had a negative-sign effect on FAI, though, like me, this debate made him question (but not yet reject) the 'obvious' conclusion.)

(Note that e.g. Wei Dai also thought it was obviously true that faster econ growth exponents had a negative-sign effect on FAI, though, like me, this debate made him question (but not yet reject) the 'obvious' conclusion.)

I'm confused by the logic of this sentence (in particular how the 'though' and 'like me' fit together). Are you saying that you and Wei both at first accepted that faster econ growth meant less chance of FAI, but then were both caused to doubt this conclusion by the fact that others debated the claim?

This was one of those cases where precisely stating the question helps you get to the answer. Thanks for the confirmation!

Even given a very fast local foom (to which I do assign a pretty small probability, especially as we make the situation more detailed and conclude that fewer things are relevant), I would still expect higher education and better discourse to improve the probability that people handle the situation well. It's weird to cash this out as a concrete scenario, because that just doesn't seem like how reasonable reasoning works.

But trying anyway: someone is deciding whether to run an AI or delay, and they correctly choose to delay. Someone is arguing that research direction X is safer than research direction Y, and others are more likely to respond selectively to correct arguments. Someone is more likely to notice there is a problem with a particular approach and they should do something differently, etc. etc.

Similarly, I expect war or external stressors to make things worse, but it seems silly to try and break this down as very specific situations. In general, people are making decisions about what to do, and if they have big alternative motivations (like winning a war, or avoiding social collapse, or what have you), I expect them to make decisions that are less aligned with aggregate welfare. They choose to run a less safe AI, they pursue a research direction that is less safe, etc. Similarly, I expect competent behavior by policy-makers to improve the situation across a broad distribution of scenarios, and I think that is less likely given other pressing issues. We nationalize AI projects, we effectively encourage coordination of AI researchers, we fund more safety-conscious research, etc. Similarly, I expect that an improved understanding of forecasting and decision-making would improve outcomes, and improved understanding of social sciences would play a small role in this. And so on.

But at any rate, my main question is how you can be so confident of local foom that you think this tiny effect given local foom scenarios dominates the effect given business as usual? I don't understand where you are coming from there. The secondary objection is to your epistemic framework. I have no idea how you would have thought about the future if you lived in 1800 or even 1900; it seems almost certain that this framework reasoning would have led you to crazy conclusions, and I'm afraid that the same thing is true in 2000. You just shouldn't expect to be able to think of detailed situations that determine the whole value of the universe, unless you are in an anomalous situation, but that doesn't mean that your actions have no effect and that you should condition on being in an anomalous situation.

Even given a very fast local foom (to which I do assign a pretty small probability, especially as we make the situation more detailed and conclude that fewer things are relevant), I would still expect higher education and better discourse to improve the probability that people handle the situation well. It's weird to cash this out as a concrete scenario, because that just doesn't seem like how reasonable reasoning works.

But trying anyway: someone is deciding whether to run an AI or delay, and they correctly choose to delay. Someone is arguing that research direction X is safer than research direction Y, and others are more likely to respond selectively to correct arguments. Someone is more likely to notice there is a problem with a particular approach and they should do something differently, etc. etc.

How did this happen as a result of economic growth having a marginally greater exponent? Doesn't that just take us to this point faster and give less time for serial thought, less time for deep theories, less time for the EA movement to spread faster than the exponent on economic growth, etcetera? This decision would ceteris paribus need to be made at some particular cumulative level of scientific development, which will involve relatively more parallel work and relatively less serial work if the exponent of econ growth is higher. How does that help it be made correctly?

Exposing (and potentially answering) questions like this is very much the point of making the scenario concrete, and I have always held rather firmly on meta-level epistemic grounds that visualizing things out concretely is almost always a good idea in math, science, futurology and anywhere. You don't have to make all your predictions based on that example but you have to generate at least one concrete example and question it. I have espoused this principle widely and held to it myself in many cases apart from this particular dispute.

But at any rate, my main question is how you can be so confident of local foom that you think this tiny effect given local foom scenarios dominates the effect given business as usual?

Procedurally, we're not likely to resolve that particular persistent disagreement in this comment thread which is why I want to factor it out.

My secondary objection is to your epistemic framework. I have no idea how you would have thought about the future if you lived in 1800 or even 1900; it seems almost certain that this framework reasoning would have led you to crazy conclusions, and I'm afraid that the same thing is true in 2000.

I could make analogies about smart-people-will-then-decide and don't-worry-the-elite-wouldn't-be-that-stupid reasoning to various historical projections that failed, but I don't think we can get very much mileage out of nonspecifically arguing which of us would have been more wrong about 2000 if we had tried to project it out while living in 1800. I mean, obviously a major reason I don't trust your style of reasoning is that I think it wouldn't have worked historically, not that I think your reasoning mode would have worked well historically but I've decided to reject it because I'm stubborn. (If I were to be more specific, when I listen to your projections of future events they don't sound very much like recollections of past events as I have read about them in history books, where jaw-dropping stupidity usually plays a much stronger role.)

I think an important thing to keep in mind throughout is that we're not asking whether this present world would be stronger and wiser if it were economically poorer. I think it's much better to frame the question as whether we would be in a marginally better or worse position with respect to FAI today if we had the present level of economic development but the past century from 1913-2013 had taken ten fewer years to get there so that the current date were 2003. This seems a lot more subtle.

past events as I have read about them in history books, where jaw-dropping stupidity usually plays a much stronger role.

How sure are you that this isn't hindsight bias, that if various involved historical figures had been smarter they would have understood the situation and not done things that look unbelievably stupid looking back?

Do you have particular historical events in mind?

We are discussing the relative value of two different things: the stuff people do intentionally (and the byproducts thereof), and everything else.

In the case of the negative scenarios I outlined this is hopefully clear: wars aren't sped up 1-for-1, so there will be fewer wars between here and any relevant technological milestones. And similarly for other stressors, etc.

Regarding education: Suppose you made everything 1% more efficient. The amount of education a person gets over their life is 1% higher (because you didn't increase the pace of aging / turnover between people, which is the thing people were struggling against, and so people do better at getting what they want).

Other cases seem to be similar: some things are a wash, but more things get better than worse, because systematically people are pushing on the positive direction.

Procedurally, we're not likely to resolve that particular persistent disagreement in this comment thread which is why I want to factor it out.

This discussion was useful for getting a more precise sense of what exactly it is you assign high probability to.

I wish you two had the time for a full-blown adversarial collaboration on this topic, or perhaps on some sub-problem within the topic, with Carl Shulman as moderator.

At some point I need to write a post about how I'm worried that there's an "unpacking fallacy" or "conjunction fallacy fallacy" practiced by people who have heard about the conjunction fallacy...

Please do this. I really, really want to read that post. Also I think writing it would save you time, since you could then link to it instead of re-explaining it in comments. (I think this is the third time I've seen you say something about that post, and I don't read everything you write.)

If there's anything I can do to help make this happen (such as digging through your old comments for previous explanations of this point, copyediting, or collecting a petition of people who want to see the post to provide motivation), please please please let me know.

If there's anything I can do to help make this happen (such as digging through your old comments for previous explanations of this point, copyediting, or collecting a petition of people who want to see the post to provide motivation), please please please let me know.

My experience has been that asking people "let me know if I can help" doesn't result in requests for help. I'd suggest just going ahead and compiling a list of relevant comments (like this one) and sending them along.

(If Eliezer doesn't end up writing the post, well, you now have a bunch of comments you could use to get started on a post yourself.)

The "normal view" is expressed by GiveWell here. Eliezer's post above can be seen as a counterpoint to that. GiveWell does acknowledge that "One of the most compelling cases for a way in which development and technology can cause harm revolves around global catastrophic risks..."

Any thoughts about what sort of society optimizes for insight into difficult problems?

I have a few thoughts.... Naturally first question is what does "optimise for insight" mean.

  1. A society which values leisure and prosperity, eg the current Scandinavians...? Evidence; They punch well above their weight economically, produce world class stuff (Volvo, Nokia, Ericsson, Bang&Olufsen spring to mind), but working pace from my experience could be described as "leisurely". Possibly best "insights/manhour"