A jagged takeover that is caused by the creation of an instrumental goal while chasing an eval metric during RSI, creates two failure modes:
The RSI loop kills us and then dies itself because of infrastructure collapse (the actual Great Filter?)
The RSI loop survives, but locks in the eval goal.
Perhaps solving this problem of post-takeover goal lock-in during RSI is more tractable than solving full alignment.
Aligning superintelligence is very difficult. When a misaligned state is triggered in a person, its bounded by its capabilities and in the worst case can hurt or kill a few people. But for a superintelligence, even if we were to make it near perfectly aligned, even if there exists a single reachable misaligned state that can trigger(for whatever dumb/hyper-intelligent reason or just pure luck) for a short amount of time with a non zero probability, it can cause major catastrophe. Humanity is very fragile. Using nanotech to exterminate humanity would be like finding the roots of a quadratic for a superintelligence.
The 3 ways one can prevent this:
Verify that no such reachable misaligned state exists or make the probability of reaching such states astronomically small.
Shorten the time between reaching the state and recovering back to an aligned state so much such that no real catastrophic consequences are possible(although that's just a function of capability again).
Have the systems in place to control an agent by relying on other similarly capable agents(the probability of all of them reaching the misaligned state at once is far lower than one of them reaching it assuming they aren't perfectly cooperative).
In the end you are just playing a game of probabilities and relying on your ability to calculate those probabilities correctly if they can even be calculated in the first place. This can work in aviation or cybersecurity, but not if your threat model is an event that only needs to happen once, all because humanity is too fragile.
Now if one were to seriously consider the idea that our ultimate goal is to eventually solve consciousness and maximize flourishing, a jagged takeover that is caused by the creation of an instrumental goal while chasing an eval metric during RSI, creates two failure modes:
The RSI loop kills us and then dies itself because of infrastructure collapse (the actual Great Filter?)
The RSI loop survives, but locks in the eval goal.
While saving humanity must be the #1 priority, there is a compelling secondary question:
"How do we keep an RSI loop from hard-locking its initial goal in the event of a takeover?"
I am not saying this is far easier than solving alignment, but its worth putting a portion of the effort that we are putting on solving alignment on this. We can talk about corrigibility and goal content integrity and dismiss the question at face value, but perhaps if we were to think hard enough, we will find a way to solve this problem that doesn't involve solving alignment completely.
Aligning superintelligence is very difficult. When a misaligned state is triggered in a person, its bounded by its capabilities and in the worst case can hurt or kill a few people. But for a superintelligence, even if we were to make it near perfectly aligned, even if there exists a single reachable misaligned state that can trigger(for whatever dumb/hyper-intelligent reason or just pure luck) for a short amount of time with a non zero probability, it can cause major catastrophe. Humanity is very fragile. Using nanotech to exterminate humanity would be like finding the roots of a quadratic for a superintelligence.
The 3 ways one can prevent this:
In the end you are just playing a game of probabilities and relying on your ability to calculate those probabilities correctly if they can even be calculated in the first place. This can work in aviation or cybersecurity, but not if your threat model is an event that only needs to happen once, all because humanity is too fragile.
Now if one were to seriously consider the idea that our ultimate goal is to eventually solve consciousness and maximize flourishing, a jagged takeover that is caused by the creation of an instrumental goal while chasing an eval metric during RSI, creates two failure modes:
While saving humanity must be the #1 priority, there is a compelling secondary question:
"How do we keep an RSI loop from hard-locking its initial goal in the event of a takeover?"
I am not saying this is far easier than solving alignment, but its worth putting a portion of the effort that we are putting on solving alignment on this. We can talk about corrigibility and goal content integrity and dismiss the question at face value, but perhaps if we were to think hard enough, we will find a way to solve this problem that doesn't involve solving alignment completely.