imagine (maybe all of a sudden) we're able to create barely superhuman-level AIs aligned to whatever values we want at a barely subhuman-level operation cost

we might decide to have anyone able to buy AI agents aligned with their values

or we might (generally) think this way to give access to that tech would be bad, but many companies are already incentivized to do that individually and can't all cooperate not to (and they actually reached this point gradually, previously selling near human-level AIs)

then it seems like everyone/most people would start to run such an AI and give it access to all their resources--at which point that AI can decide what to do, whether that's investing in some companies and then paysing themselves periodically or invest in running more copies of themselves, etc. deciding when to use those resources for the human to consume vs reinvesting them

maybe people would wish for everyone to run AI systems with "aggregated human values" instead of their personal values, but given others aren't doing that, they won't either

now, intelligence isn't static anymore--presumably, the more money you have, the more intelligence you have, and the more intelligence the more money.

so let's say we suddenly have this tech and everyone is instantiating one such agent (which will make decisions about number and type of agents) that has access to all their resources

what happens?

maximally optimist scenario: solving coordination is not too late and gets done easily and at a low cost. utopia

optimist scenario: we don't substantially improve coordination, but our current coordination level is good enough for an Okay Outcome

pessimist scenario: agents are incentived to create subagents with other goals for instrumentally convergent purposes. defecting is better than cooperating individually, but defecting-defecting still leads to extremely bad outcomes (just not as bad as if you had cooperated in a population of cooperators). those subagents quickly take over and kill all humans (those who cooperated are killed slightly sooner). or, not requiring misaligned AIs, maybe the aestivation hypothesis is true but we won't coordinate to delay energy consumption or wars will use all surplus leaving nothing for humans to consume

I'm not confident we're in an optimist scenario. being able to download one's values and then load them in an AI system (and having initial conditions where that's all that happens) might not be sufficient for good outcomes

this is evidence for the importance of coordinating on how AGI systems get used, and that distributing that wealth/intelligence directly might not be the way to go. rather, it might be better to keep that intelligence concentrated and have some value/decision aggregation mechanism to decide what to do with it (rather than distributing it and later not being able to pool it back together if that's needed, which seems plausible to be)

a similar reasoning can apply for poverty alleviation: if you want to donate money to a group of people (say residents of a poor country) and if you think they haven't solved their coordination problem, then maybe instead of distributing that money and let them try to coordinate to put back (part of) that money in a shared pool for collective goods, you can just directly put that money in such a pool--the problem about figuring out the shared goal remains but it at least arguably solves the problem of pooling that money (ex.: to fund research for a remedy to a disease affecting that population)