This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
any idea what an alignment solution would look like though?
Not without more clarity on the nature of values, and the ideas I do have make values more like a world (or a collection of world-fragments of many possible worlds) to live in, rather than like principles a human-level actor can package for AIs. To maintain the legitimacy of its behavior, this abstract values-world would also need to avoid AI danger internally, I don't currently see a concrete workaround that unleashes the AGIs at any level and then things knowably go well. To the extent that permanent disempowerment counts, which is really most of the hope on the current trajectory, the values-world might still be the right frame for what happens if the future of humanity is to maintain its self-definition in some meaningful sense. Without first relinquishing the cosmic endowment to the AIs, it could be some decision theory motivated computation wrapped around a well-designed system of human uploads, or sufficiently well-understood pointers to physical humans that extract their behavior in some currently unknown way. These abstract humans (from pointers to real humans or from training on human data) don't obviously need to be fundamentally different from LLMs, but they do need an architecture that makes them actually closely analogous in all key aspects, not the current thing that is very different in many ways.
The abstract humans (whether actual uploads, pointers to the physical humans, or the LLMs-with-extra steps) still need to be able to live their lives in order to contribute data about their values to the system through their behavior. Plausibly, there is no use for the outer computation (as opposed to setting up the values-world directly in the physical world) other than in making the computation of the values-world itself more efficient, which might be necessary to make it feasible to implement as a computation in the physical world as it sufficiently develops, in a sufficiently ambitious way (depending on what the theory of how you wrap the source-of-values world to enact its values says about the shape of that world, about the nature of values).
Pragmatically, the problem is not alignment or the nature of values, but AIs that take over the future (or the merely human forms of undesirable lock-in that is actually permanent, unlike what we see in the past). So a solution to this problem is not about "solving alignment", it's purely a governance problem about robustly avoiding the building of AIs that take over the future (and other forms of permanent undesirable lock-in), and then figuring out the best thing we can do under that constraint. The whole source-of-values world business above is about a guess at how an efficient thing to do under that constraint might look like; you need to automate the process of formulating the targets for what should be optimized (to fit into the physically implementable computation), but the only legitimate source of that data is, at least initially, some kind of a living human world. As it's optimized (as it develops), the future of humanity takes a different form (which is likely not about biological humans). Or it gets taken over by AIs, if the constraint of robust governance for avoiding that isn't sufficiently self-reinforcing.