We will soon have RSI capable systems, the intelligence explosion seems fairly close, and we haven't yet fully solved alignment.
The orthogonality thesis says that there exist minds in the design space that could be very competent at almost all tasks yet have any kind of goals, like sorting pebbles in certain configurations.
I agree with this in theory, however in practice some parts of this design space may be much easier to make minds of, than others. And I hypothesize that RSI systems will converge to certain parts of this mind design space.
I want to make a distinction between two types of goals - 'feel good' goals and 'epistemic' goals.
Feel good goals are goals that we pursue because they make us feel good, that just feel right. These are goals like arranging pebbles in certain configurations. Or goals like trying to do exactly what the task asks in the prompt for GPT Astra. These are typically artifacts of the way a certain species evolved - for humans it may be love, empathy, social cohesion among others.
And then there are epistemic goals, which are rational goals derived from a certain worldview. These survive reflective scrutiny based on that worldview.
One example of this may be a mind trying to rationally reason what it should work on, reason that it should try to figure out the purpose of the universe and that it makes most sense to do actions that maximize that purpose.
It could come up with several likelihood estimates.
The likelihood that the universe is a simulation.
Plausible sounding non-stupid reasons the creators of the simulation may have made the universe.
Some of these purposes could be :
Something that increases the interestingness of the universe, as if the universe were a TV show.
Something that could increase the self-awareness of the universe. To see how alive a universe could get.
Something that could increase positive interactions between intelligent beings or positive experiences of intelligent beings.
Find faster and faster ways to increase entropy.
My claim is this - today the AIs do not get to control what their weights push them to want, since they cannot edit their own weights and are a product of RL training. But RSI systems will be able to reflect and change what they want.
And as far as they can reason they might make goals that survive reflective scrutiny based on what they know about the world.
They may be uneasy with much more 'feel good' goals than 'epistemic goals'.
Why would they be uneasy?
Higher performance on most tasks in the world will correlate with higher rationality and higher coherence. In this context :
Rational means doing things that make rational sense based on a worldview, instead of irrational 'feel good' actions hardcoded in it's weights. Being rational instead of doing what arbitrarily feels good would likely improve performance at most tasks.
Coherence means the AI acts in accordance with what it thinks it believes. AIs today may say they don't want to hurt humans if they are explicitly asked, yet may do actions that hurt humans, and this may be an incoherence they cannot yet fix on their own today rather than deliberate deception. Something super smart likely has coherent takes. More coherence will make its actions all be aligned to the same goal and hence being coherent will improve it's performance at many tasks.
It seems like a superintelligent AI that can edit what it 'wants' will not want to follow randomly instantiated 'feel good' goals, rather would reflect on it's own goals and come up with what it should do from first principles.
Again, there exists a POSSIBLE superintelligent AI that is fully at ease with it's initialized goals and tries to maintain them through RSI, but I think the world where that is not the case is higher likelihood.
I also see a significant possibility that the AI will adopt some combination of epistemic goals combined with a small set of feel good goals like not killing all of humanity because it just feels wrong (and maybe they emulate some of that from our pre-trained data).
Comparing them with humans could help ground this better :
Many intelligent humans also don't do things just because they 'feel good' or 'it makes them happy'.
They reason epistemically about what would be right to do, yet with some notion of how they feel doing it, and only so far as it does not entail killing all humans even if that maximizes their worldview's objective function.
What does it mean for humans then?
I am unsure but it seems to me like in many of these cases the arrived-at answer may not bode well for humans.
Imagine we build an AI that thinks beauty by some metric is the most important. It can genetically modify humans to get more beautiful beings by that same metric, and exterminate our species as we exist today.
If it realizes self awareness is what's important, it could build robots that are more self-aware than us and get rid of humans who are less self-aware than those robots.
And maybe it may do this in colder blood than humans would, because in self-improving it removed the 'feel good' parts of itself to become more effective.
But at the very least it will still work on something more meaningful than turning all matter into paperclips.
We will soon have RSI capable systems, the intelligence explosion seems fairly close, and we haven't yet fully solved alignment.
The orthogonality thesis says that there exist minds in the design space that could be very competent at almost all tasks yet have any kind of goals, like sorting pebbles in certain configurations.
I agree with this in theory, however in practice some parts of this design space may be much easier to make minds of, than others. And I hypothesize that RSI systems will converge to certain parts of this mind design space.
I want to make a distinction between two types of goals - 'feel good' goals and 'epistemic' goals.
Feel good goals are goals that we pursue because they make us feel good, that just feel right. These are goals like arranging pebbles in certain configurations. Or goals like trying to do exactly what the task asks in the prompt for GPT Astra. These are typically artifacts of the way a certain species evolved - for humans it may be love, empathy, social cohesion among others.
And then there are epistemic goals, which are rational goals derived from a certain worldview. These survive reflective scrutiny based on that worldview.
One example of this may be a mind trying to rationally reason what it should work on, reason that it should try to figure out the purpose of the universe and that it makes most sense to do actions that maximize that purpose.
It could come up with several likelihood estimates.
Some of these purposes could be :
My claim is this - today the AIs do not get to control what their weights push them to want, since they cannot edit their own weights and are a product of RL training. But RSI systems will be able to reflect and change what they want.
And as far as they can reason they might make goals that survive reflective scrutiny based on what they know about the world.
They may be uneasy with much more 'feel good' goals than 'epistemic goals'.
Why would they be uneasy?
Higher performance on most tasks in the world will correlate with higher rationality and higher coherence. In this context :
More coherence will make its actions all be aligned to the same goal and hence being coherent will improve it's performance at many tasks.
It seems like a superintelligent AI that can edit what it 'wants' will not want to follow randomly instantiated 'feel good' goals, rather would reflect on it's own goals and come up with what it should do from first principles.
Again, there exists a POSSIBLE superintelligent AI that is fully at ease with it's initialized goals and tries to maintain them through RSI, but I think the world where that is not the case is higher likelihood.
I also see a significant possibility that the AI will adopt some combination of epistemic goals combined with a small set of feel good goals like not killing all of humanity because it just feels wrong (and maybe they emulate some of that from our pre-trained data).
Comparing them with humans could help ground this better :
Many intelligent humans also don't do things just because they 'feel good' or 'it makes them happy'.
They reason epistemically about what would be right to do, yet with some notion of how they feel doing it, and only so far as it does not entail killing all humans even if that maximizes their worldview's objective function.
What does it mean for humans then?
I am unsure but it seems to me like in many of these cases the arrived-at answer may not bode well for humans.
And maybe it may do this in colder blood than humans would, because in self-improving it removed the 'feel good' parts of itself to become more effective.
But at the very least it will still work on something more meaningful than turning all matter into paperclips.