DanielW's Shortform

DanielW

This is a special post for quick takes by DanielW. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

It is brought up quite often, but I have not (yet!) heard a satisfactory answer to the question: "why are so many leading people in advocating AI risk also investing in developing LLMs and AI-dependent ecosystem?"

It seems to me, the position (in simple form) holds:

Divesting from a technology discourages it's development/advancement n aggregate (investing in a technology encourages it)
Therefore, if the development/advancement of a technology is, in aggregate, undesirable, one should be divest from it and minimize any investment (not withstanding benefits that are greater than the risks of development)

This has been the approach to things like fossil fuels, which i believe is correct. Yes, individual use has minimal impact, and what matters is the aggregate but individual divestment impacts the aggregate and makes development less profitable and alternatives more desirable. Of course, sometimes fossil fuel usage is the best viable option for some technologies and the goal isn't to eliminate it completely but rather to drive up the costs and decrease the profits to the industry so their use will be restricted to areas where the benefits are far in excess than the risk and development will be reduced to manageable level.

A lot of "pop" opposition to AI does take this approach (e.g. lambasting and refusing to invest or do business with companies/products that come from AI models) but a lot of the crowd here and in "rationalist" spheres seem to view this more as childish and actually are investing in people doing more with LLMs while inviting on their danger.

Just to give one example, take @gwern (among others) promoting bounties for producing AI content. I am not going to talk about his business, (my understanding, which may be incorrect, is his position is simply along the line of "There is no contradiction between calling for a temporary pause on the very largest giant-AI experiments, and [developing] your own AI company.")

This seems like the opposite approach I would want to take if my goal was to stop companies from developing AI systems. Obviously, the more profitable developing these systems is, the more firms will want to invest in the industry and work to expand the systems. And obviously on the otherside, the less profit firms see in these developments, the more their investment and usage of AI systems is going to be restricted to only the clearest use-cases where the benefit outweighs the cost; people and firms obviously invest more in alternatives instead and view investing in large long term investments with more skepticism the less they see an industry as profitable.

1DanielW2mo

Also, I should say, that when you plug in the numbers you get big negatives. Let's say i estimate at the current probilibility distribution of AI development has a net expected utility of say -1,000,000,000,000 ( say, my p(doom) is ~1/8, i think 8 billion humans will die and i value each human at 1,000). If i think my actions (including positive feedback!) are contributing 1/100,000,000th of AI development, then my expectated utility from the AI development I contribute should be ~ -10,000 (about the same as the value I would place on 10 human lives). So unless I think i am saving 10 lives (or equivelent) in contributing 1 hundredth of a millionth of AI development, I should find my actions negative. Current total investment in AI is under 2 trillion. Being conservative (a dollar now means more than a dollar later!) you should value contributing $2,000 to AI development as being roughly as negative as killing a stranger (under a simplified framework, that just looks at dollars).

2Nisan2mo

Now we're getting somewhere. You're assuming that providing one billionth of this year's investment in AI means decreasing the expected utility from human extinction by one billionth. But that's not right. If you invest in AI, that means the AI companies will choose to raise less money from other investors. The net effect is that AI companies will raise a little more money than if you hadn't invested (but less than the amount you invest), get a better interest rate, and spend less effort on fundraising this year. And this will have some effect on the probability of an extinction-level AI catastrophe this generation, but the effect isn't linear in the amount you invest. In a parallel thread, your theory of impact was that divestment has an effect on regulation. That's also nonlinear. Investing in AI is something I'm likely to do. (Arguably >10% of the S&P 500 by market cap use marginal investment to try to build frontier models.) Here's how I justify it: The amount of stock I can buy won't move the stock price perceptibly. If I estimated the price movement it would be tiny. That would translate into AI companies expending a tiny bit less effort on selling stock this year to fund datacenters, which accelerates timelines a little bit, which increases existential risk a very tiny amount. On the other hand, if I invest my savings in a broad-market index fund that includes AI, that'll increase my savings and I'll have more time to spend doing AI safety research, calling my Congressperson, etc., which are more directly impactful. Plus, I have lots of personal uses for more money. I'm basically satisfied with where this thread ended up. (I just wanted to convey that it's in general reasonable to do things that seem individually worth it, even if they seem to be promoting AI in some way.) I want to flag that I won't have the energy to find a crux in our disagreement over the Unslop contest or whether I should buy index funds. Though if you have a good argument about why n

You seem to be confusing some things, probably because I was not as precise with definitions of terms as I should have been.

In a parallel thread, your theory of impact was that divestment has an effect on regulation. That's also nonlinear.

Divestment was an example of a common behavior activists engage in (which I would broadly speaking endorse). The impact from divestment on a industry is different from the negative impact of funding, advertising them, and paying an industry. They not direct inversions of each other, divesting is a very different activity.... (read more)