Eliezer Yudkowsky and John Wentworth[1] (and many others) are still correct that ~no alignment approach generated by existing civilization will scale to ASI. I recommend you read here, here, or watch this video if you haven't yet and disagree.
However, I think an aligned and moderately superintelligent entity (3-6 standard deviations beyond the most capable of us at X-risk relevant cognition) would move the needle. I'mnotoriginal in thinking so.
Research and political efficacy are heavy-tailed over g. Someone mildly superintelligent at memetic influence could probably negotiate a pause, and likewise, a superphilosopher / mathematician has a much better chance of finding an algorithmic expression of corrigibility than current civilization.
But despite this, and despite most of the tech tree being accessible, no existing actors are publicly and competently angling for tail-end cognition enhancement. BCI funding today is ~entirely prosthetic.
In this sequence, I'll explain why tail-intelligence BCIs seem impactful and tractable; how this sidesteps the hard part of aligning a superintelligence sufficient for a pivotal act; classes of algorithms a server could use to massively improve brain function; and relevant bioengineering constraints. And also FM radio, if you're into that.
I think John underweights fluid g relative to acquired technical skill as the limiting input for alignment work; a future post in this sequence argues that skill acquisition is probably the easiest cognitive trait to BCI-boost, so the more right John is about this, the more tractable human superintelligent alignment becomes.
Eliezer Yudkowsky and John Wentworth[1] (and many others) are still correct that ~no alignment approach generated by existing civilization will scale to ASI. I recommend you read here, here, or watch this video if you haven't yet and disagree.
However, I think an aligned and moderately superintelligent entity (3-6 standard deviations beyond the most capable of us at X-risk relevant cognition) would move the needle. I'm not original in thinking so.
Research and political efficacy are heavy-tailed over g. Someone mildly superintelligent at memetic influence could probably negotiate a pause, and likewise, a superphilosopher / mathematician has a much better chance of finding an algorithmic expression of corrigibility than current civilization.
But despite this, and despite most of the tech tree being accessible, no existing actors are publicly and competently angling for tail-end cognition enhancement. BCI funding today is ~entirely prosthetic.
In this sequence, I'll explain why tail-intelligence BCIs seem impactful and tractable; how this sidesteps the hard part of aligning a superintelligence sufficient for a pivotal act; classes of algorithms a server could use to massively improve brain function; and relevant bioengineering constraints. And also FM radio, if you're into that.
I think John underweights fluid g relative to acquired technical skill as the limiting input for alignment work; a future post in this sequence argues that skill acquisition is probably the easiest cognitive trait to BCI-boost, so the more right John is about this, the more tractable human superintelligent alignment becomes.