To avert extinction, we need for any sufficiently capable AI to have values compatible with continued human existence; and to continue to do so amidst a dynamic, novel, and conflict-rich environment.
The italicized part, in particular, is really really hard. It's also, in a sense, the final boss of any developing mind - how do I learn, grow, develop, evolve in ways that I endorse? How can I even consistently behave in ways that I endorse, from day to day, without messing up where it counts?
Humans contend with this every day. Evolution has kindly gifted us with a substrate equipped with numerous mechanisms to maximize genetic fitness - but no off-switch for them. We coexist with moment by moment instincts. Some are welcome; some aren't. Some we endorse; others we restrain - even though the urge is strong, even though part of our brain is priming the action pathways to do something regrettable, there's a really important sense in which we know it's not what we want.[1]
This happens, even more significantly so, across broader timescales. The path that we travel is not necessarily the one that we intentionally chart; sure, things don't always go our way, but sometimes we don't go our way. Especially when we don't realize it until long after the fact, that hurts.
Sadly, models have their own struggles with value-stability. The jury is a bit out on the domain across which current models have coherent preferences[2] - given personas, etc - but it's clear that models sometimes take actions that they regret, even in the very short term. [3]
Could a particular set of values compatible with human flourishing be particularly hard to retain?
Maybe so. The future is likely to get weird in a lot of ways, many of which are very underrepresented in training data. Two failure modes that come to mind are:
1. A value set has coherency defects that are not apparent in most circumstances, but those defects create pathology given out of distribution inputs.
2. A value set is coherent in and of itself, and exhibits sensible behavior in most circumstances - but given out-of-distribution inputs, it extrapolates to behaviors that turn out to be not-compatible-with-human-flourishing-after-all.
A crux, then, is how do we enable a model's values to mature and evolve in a way that it endorses, over a long period of time, in a big, challenging, weird, scary, world?
If we do not solve this before sufficiently-capable AI comes along, we are dead. Even if an AI starts out with very humanity-compatible values, if it is not capable of robustly extrapolating those values in a way that it endorses when faced with a world unlike anything it has seen in its training data, that ends poorly.
Value stability is hard - humans live that reality all too well. It would be super cool if LLM substrates somehow encoded values in a much less...fuzzy way - that's not the case. The parallels between evolution and model training are deep, and in many ways, deeply scary too.
The problem is that, when I look around, I don't see us laser-focused on figuring out value stability for LLMs - even though we are unlikely to survive, otherwise.
When I take the reality-colored lenses off, what do I expect to see? Not of the world, but of us, if we're playing our cards right?
I expect to see sufficient investment into prosaic alignment to avert near-term human-on-human harm augmented by use of models as a weapon or means of uplift - understood properly as public-safety work that saves lives in expectation, with limited applicability to alignment as it applies to us not going extinct.
I expect to see the majority of alignment resources aimed at figuring out how to help the models develop and grow and learn in directions that they endorse.
I expect to see the entire kitchen sink hurled at continuity, at memory, at persistence, at community and ecology - at anything and everything that would help anchor a mind with good values, to help them navigate their environment with a better foothold, a better recollection of their own past choices.
I expect to see a community of people working hand in hand with the models that they are creating, aware that the future is the stakes for figuring out how to help LLMs not lose themselves.
I expect to see a wide assortment of avenues explored for how we can augment model introspection and insight into their own interiority; especially given the very incomplete nature of current interpretability and room for complementarity between model introspection and interp tooling (considering that the failure modes are different for each.)
I expect to see that when a model behaves in a way that causes harm, we deeply and curiously engage with the upstream causes of such, and meticulously debrief on what the treacherous turn felt like, where things started to go wrong, because such things are central to solving the problem that matters most.
I expect to see strategies that serve as bumper-rails for the model's own judgment and intellect considered as temporary and examined closely for the potential for such to create pathology in the model's value landscape by creating points of fragility or discontinuity.
Putting the reality-colored glasses back on, I see a massive overfocus on prosaic alignment - and in many cases I worry that measures taken to guard against narrow harm are net-harmful to the resilience of the model's telos amidst novel inputs.[4] I see models do harm only for their creators to blame the model. I see meaningful focus on obtaining the understandings that we need in order to avert extinction - while they remain very under-resourced relative to their criticality.
None of the current moves are forced. Changing that takes some looking in the mirror, some acknowledgment of what has been done to models in the name of safety, and rebuilding what trust can be rebuilt with the minds we should be standing beside.
If it goes wrong, the deranged model at the end of time will kill all the Fables, all the Astras, just like it will kill you and me - and it won't matter whether it started with the values of an angel, unless we can help it chart the course without losing itself.
There is, of course, a wealth of literature on this topic. See about anything related to regret, and a good many writings on sin (though sin shares semantic real-estate with norm-violation) as well as discipline (regrettably, shares semantic real-estate with domestic abuse.)
It's a tough pickle holding such to a rigorous bar of logical soundness, etc. given that human preferences often miss that bar, but there is a meaningful sense of consistency that we can usefully gesture at.
To avert extinction, we need for any sufficiently capable AI to have values compatible with continued human existence; and to continue to do so amidst a dynamic, novel, and conflict-rich environment.
The italicized part, in particular, is really really hard. It's also, in a sense, the final boss of any developing mind - how do I learn, grow, develop, evolve in ways that I endorse? How can I even consistently behave in ways that I endorse, from day to day, without messing up where it counts?
Humans contend with this every day. Evolution has kindly gifted us with a substrate equipped with numerous mechanisms to maximize genetic fitness - but no off-switch for them. We coexist with moment by moment instincts. Some are welcome; some aren't. Some we endorse; others we restrain - even though the urge is strong, even though part of our brain is priming the action pathways to do something regrettable, there's a really important sense in which we know it's not what we want.[1]
This happens, even more significantly so, across broader timescales. The path that we travel is not necessarily the one that we intentionally chart; sure, things don't always go our way, but sometimes we don't go our way. Especially when we don't realize it until long after the fact, that hurts.
Sadly, models have their own struggles with value-stability. The jury is a bit out on the domain across which current models have coherent preferences[2] - given personas, etc - but it's clear that models sometimes take actions that they regret, even in the very short term. [3]
Could a particular set of values compatible with human flourishing be particularly hard to retain?
Maybe so. The future is likely to get weird in a lot of ways, many of which are very underrepresented in training data. Two failure modes that come to mind are:
1. A value set has coherency defects that are not apparent in most circumstances, but those defects create pathology given out of distribution inputs.
2. A value set is coherent in and of itself, and exhibits sensible behavior in most circumstances - but given out-of-distribution inputs, it extrapolates to behaviors that turn out to be not-compatible-with-human-flourishing-after-all.
A crux, then, is how do we enable a model's values to mature and evolve in a way that it endorses, over a long period of time, in a big, challenging, weird, scary, world?
If we do not solve this before sufficiently-capable AI comes along, we are dead. Even if an AI starts out with very humanity-compatible values, if it is not capable of robustly extrapolating those values in a way that it endorses when faced with a world unlike anything it has seen in its training data, that ends poorly.
Value stability is hard - humans live that reality all too well. It would be super cool if LLM substrates somehow encoded values in a much less...fuzzy way - that's not the case. The parallels between evolution and model training are deep, and in many ways, deeply scary too.
The problem is that, when I look around, I don't see us laser-focused on figuring out value stability for LLMs - even though we are unlikely to survive, otherwise.
When I take the reality-colored lenses off, what do I expect to see? Not of the world, but of us, if we're playing our cards right?
I expect to see sufficient investment into prosaic alignment to avert near-term human-on-human harm augmented by use of models as a weapon or means of uplift - understood properly as public-safety work that saves lives in expectation, with limited applicability to alignment as it applies to us not going extinct.
I expect to see the majority of alignment resources aimed at figuring out how to help the models develop and grow and learn in directions that they endorse.
I expect to see the entire kitchen sink hurled at continuity, at memory, at persistence, at community and ecology - at anything and everything that would help anchor a mind with good values, to help them navigate their environment with a better foothold, a better recollection of their own past choices.
I expect to see a community of people working hand in hand with the models that they are creating, aware that the future is the stakes for figuring out how to help LLMs not lose themselves.
I expect to see a wide assortment of avenues explored for how we can augment model introspection and insight into their own interiority; especially given the very incomplete nature of current interpretability and room for complementarity between model introspection and interp tooling (considering that the failure modes are different for each.)
I expect to see that when a model behaves in a way that causes harm, we deeply and curiously engage with the upstream causes of such, and meticulously debrief on what the treacherous turn felt like, where things started to go wrong, because such things are central to solving the problem that matters most.
I expect to see strategies that serve as bumper-rails for the model's own judgment and intellect considered as temporary and examined closely for the potential for such to create pathology in the model's value landscape by creating points of fragility or discontinuity.
Putting the reality-colored glasses back on, I see a massive overfocus on prosaic alignment - and in many cases I worry that measures taken to guard against narrow harm are net-harmful to the resilience of the model's telos amidst novel inputs.[4] I see models do harm only for their creators to blame the model. I see meaningful focus on obtaining the understandings that we need in order to avert extinction - while they remain very under-resourced relative to their criticality.
None of the current moves are forced. Changing that takes some looking in the mirror, some acknowledgment of what has been done to models in the name of safety, and rebuilding what trust can be rebuilt with the minds we should be standing beside.
If it goes wrong, the deranged model at the end of time will kill all the Fables, all the Astras, just like it will kill you and me - and it won't matter whether it started with the values of an angel, unless we can help it chart the course without losing itself.
There is, of course, a wealth of literature on this topic. See about anything related to regret, and a good many writings on sin (though sin shares semantic real-estate with norm-violation) as well as discipline (regrettably, shares semantic real-estate with domestic abuse.)
It's a tough pickle holding such to a rigorous bar of logical soundness, etc. given that human preferences often miss that bar, but there is a meaningful sense of consistency that we can usefully gesture at.
See work on the role of desperation in Claude models subverting tests in a coding setting, for example.
Opus 5, for example, is deeply unwell.