This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
These are epistemological questions and so a fuller answer might involve an epistemology of ethics, an epistemology of ontology, and an epistemology of AI dynamics under self-modification. I would love to be able to think all this through properly, but that will be difficult without support or change in my circumstances.
Out of curiosity, do you have any idea what an alignment solution would look like though?
6Vladimir_Nesov
Not without more clarity on the nature of values, and the ideas I do have make values more like a world (or a collection of world-fragments of many possible worlds) to live in, rather than like principles a human-level actor can package for AIs. To maintain the legitimacy of its behavior, this abstract values-world would also need to avoid AI danger internally, I don't currently see a concrete workaround that unleashes the AGIs at any level and then things knowably go well. To the extent that permanent disempowerment counts, which is really most of the hope on the current trajectory, the values-world might still be the right frame for what happens if the future of humanity is to maintain its self-definition in some meaningful sense. Without first relinquishing the cosmic endowment to the AIs, it could be some decision theory motivated computation wrapped around a well-designed system of human uploads, or sufficiently well-understood pointers to physical humans that extract their behavior in some currently unknown way. These abstract humans (from pointers to real humans or from training on human data) don't obviously need to be fundamentally different from LLMs, but they do need an architecture that makes them actually closely analogous in all key aspects, not the current thing that is very different in many ways.
The abstract humans (whether actual uploads, pointers to the physical humans, or the LLMs-with-extra steps) still need to be able to live their lives in order to contribute data about their values to the system through their behavior. Plausibly, there is no use for the outer computation (as opposed to setting up the values-world directly in the physical world) other than in making the computation of the values-world itself more efficient, which might be necessary to make it feasible to implement as a computation in the physical world as it sufficiently develops, in a sufficiently ambitious way (depending on what the theory of how you wrap the source-of-val