This is a special post for quick takes by noma. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Some obvious takes on AI alignment that we may forget sometimes. I will just write out obvious things that are very obvious.
Let's first write out (have chatgpt write for me) a simple impossibility theorem for AI alignment.
This is obvious
However, it becomes immediately clear upon seeing the 2nd assumption, it seems naturally that instead of trying to align any single one AI, one should be thinking about mechanism design; the same way we align humans with each other that is. How do we align humans with each other? reputation, law, and repeated interaction, treaties, deterrence, verification, trade dependence, institutions, competition, defectors, whistleblowers etc.
E.g., you can imagine that you train one set of AI's to be defectors while another set of AI's are trained to maximally pursue utility. There may be AI's operating as "States" and AI's that behave like "Cops", etc.
If there are people working on agent foundations, then the problem of aligning agents with each other fundamentally mathematically equivalent between AI's and AI's and AI's and humans. This is again an obvious statement. If you take seriously the realms of mathematics and the realms of decision theory, and you write out 'here is an agent with some utility' and then continue forwards with your reasoning, and you assign no specialness to humans or AI's anywhere; then there is no reason to treat humans as different from AI's.
This implies that if humans can be aligned with each other, then AI's can be aligned with humans. Sometimes I think, it can be useful to write out the obvious. Hope it wasn't too obvious however.