Computationally tractable mathematical models of alignment are bound to be biased and blind to certain aspects of human values

No single mathematical model of human values that has orders of magnitude fewer degrees of freedom than an actual human will adequately capture the complexity of value because humans are complex systems and, therefore, cannot be reduced to a much simpler model.

If the model is sufficiently complex to robustly capture human values, such as whole-brain emulation, then the ethical concerns and S-risks of actually using these models for alignment appear because the model itself may suffer.

Many mathematical theories of human cognition or frameworks for computing (inferring) human values are considered as the basis for alignment, as well as process theories of alignment that implicitly rely on a particular mathematical theory even if it doesn’t infer (humans’ or AIs’) values explicitly, such as the shard theory (RL-based), or the latest Beren Millidge’s computational anatomy of human values, or cooperative inverse reinforcement learning, or Bayesian models and approaches, or various linguistic process theories of alignment that I expect to become very hot this year due to the astonishing success of LLMs. However, since all these theories are collapsing the complexity of humans (or else they are equivalent to full human simulations), they are all bound to be incomplete. Moreover, all these theories are bound to be biased (this is a form of inductive bias if you wish), that is, to be relatively blind to specific kinds of human values, or to specific aspects of human nature that we can see as somehow related to (or producing) “values”.

In other words, human values are not only complex in the sense that they are very elaborate. Crucially, human values are also not capturable within a single mathematical framework or ontology for describing them.

From “solving the alignment problem” to engineering the alignment process

The main implication of the above thesis is that we should abandon the frame that we should “solve” the alignment problem and seek a smart theory that will “crack” this problem.

I feel that a fair amount of unproductive debates and unproductive...