Also very rough response
- I think the debate would probably benefit from better specification of what is meant by "misalignment" or "solving alignment"
-- I do not think the convincing versions of gradual disempowerment either rely on misalignment or result power concentration among humans for relatively common meaning of alignment roughly at the level "does what the developer wants and approves, resolving conflicts between their wants in a way which is not egregiously bad". If "aligned" means something at the level "implements coherent extrapolated volition of humanity" or "solves AI safety" than yes.
- Economic
-- the counter-argument seems to be roughly in the class "everyone owns index funds" and "state taxes AIs"
--- count-counter arguments are:
----- difficulty of indexing economy undergoing radical technological transiton (as explained in an excellent post by Beren we reference)
----- problems with stability of property rights: people in the US or UK often perceive them as very stable, but they depend on state enforcing them -> state becomes a more load-bearing component of the system
----- taxation: same -> state becomes a more load-bearing component of the system
----- in many cases some income can be nominally collected in the name of humans, but they may have very little say in the process or how is it used (for some intuition, consider His Majesty Revenue & Customs. HMRC is direct descendant of a chain of org collecting customs from ˜13th century; in the beginning, His Majesty had a lot of say in what these are and also could actually use the revenue; now, not really)
- Cultural. If humans remain economically empowered (in the sense of having much more money than AI), I think they will likely remain culturally empowered.
-- this takes a bit too much econ perspective on culture; cultural evolution is somewhat coupled with economy, but is an independent system with different feedback loops
-- in particular it is important to understand that while in most econ thinking preferences of consumers are exogenous, culture is largely what sets the preferences; to some extent culture is what the consumers are made of -> having overwhelming cultural production power means setting consumer preference
--- for some intuitions, consider current examples
---- right-wing US twitter discourse is often influenced by anonymous accounts run by citizens of India and Pakistan; people running these accounts often have close to zero econ power, and their main source of income is the money they get for posts
----- yet they are able to influence what eg Elon Musk thinks, despite the >10ˆ7 wealth difference
----- Even AI-AI culture, if it promotes bad outcomes for humans and humans can understand this, will be indirectly selected against as humans (who have money) prefer interacting with AI systems that have good consequences for their well-being. seems to prove too much. Again, consider Musk. He is the world's wealthiest person, yet it is the case that his mind is often inhabited by ideas that are bad for him, his well-being, and have overall bad consequences.
State
-- unclear to me: why would you expect "formal power" to keep translating to real power (For some intuitions: United Kingdom. Quite many things in the country are done in the name of His Majesty The King)
-- we assume institutional AIs will be aligned to institutions and institutional interests, not their nominal human representatives or principals
-- I think the model of the world where superagents like states or large corporations have "dozens of people controlling these entities" is really not how the world works. Often the person nominally in charge is more a servant of the entity aligned to it rather than "principal".
--- “While politicians might ostensibly make the decisions, they may increasingly look to AI systems for advice on what legislation to pass, how to actually write the legislation, and what the law even is. While humans would nominally maintain sovereignty, much of the implementation of the law might come from AI systems.” / ll seems good, if AI is well-aligned? Imo, it would be bad to not hand off control to aligned AIs that would be more competent and better motivated that us
---- I think you should be really clear who are the AIs aligned to. Either eg US governmental AIs are aligned to US government and state in general, in which case the dynamic leads to a state with no human principals with any real power, and humans will just rubber-stamp.
---- Or the governmental AIs are aligned to specific humans, such as US president. This would imply very large changes of power relative to current state, transitioning from republic to personal dictatorship. Both US state and US citizens would fight this
(may respond to some of the rough thoughts later, they explore interesting directions)
Overall, i think this considerations favours economic power concentration among the humans who are legally allowed to invest in the most promising opportunities and have AI advisors to help them
And, conversely, this would would decrease the economic influence of other humans and AIs
Epistemic status: very rough! Spent a couple of days reading the Gradual Disempowerment paper and thinking about my view on it. Won’t spend longer on this, so am sharing rough notes as is
Summary
I think most people who have read this far shouldn’t read on! The rest of this post is rough “thinking out loud” notes.
I discuss:
Things I agree with
Points of disagreement
Steelman of GD
Ok, stepping back, am i convinced by my own steelman?
Steelman of GD even with the alignment problem solved?
Could all this go through if AI alignment is solved? Let me try to argue for that conclusion.
Stepping back, is this convincing?
Especially rough thoughts