This is the hardest challenge I foresee with BCI intelligence augmentation.
The simplest architectural prior for improving human intelligence is to simulate what a brain would do given more compute. This is usually more time ("what conclusion would I come to given 100 years of thought"), but more brain mass seems more tractable given current technology.
Consider that models of long-term cognition get very sparse feedback. You can't finetune an LLM on your reflective trajectory across a century because that would take an entire lifetime. With high-resolution neural connections (and a robust understanding of how neurons learn), we could close a mass-based feedback loop in milliseconds-to-minutes.
Given better genetic engineering tech, we might give people a virus which 10x'ed their neuron count without terrible physiological side effects. We very probably can't do that this decade because genetic edits are hard.
But we do have a lot of GPUs.
If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we've effectively boosted someone's brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.
It's like extending brain maturation beyond 2-to-3-decade typical timelines. Existing biological circuits specify some signal similar to a gradient error, and the computronium updates commensurately, just as any section of the universally-learning parts of the brain do.
An incorrect learning rule almost certainly doesn't cause competent agentic misalignment, but it won't make a superintelligent human either. Locating the right rule seems very hard; theredoesnotseemto be consensus on how local updates are implemented.
This is the hardest challenge I foresee with BCI intelligence augmentation.
The simplest architectural prior for improving human intelligence is to simulate what a brain would do given more compute. This is usually more time ("what conclusion would I come to given 100 years of thought"), but more brain mass seems more tractable given current technology.
Consider that models of long-term cognition get very sparse feedback. You can't finetune an LLM on your reflective trajectory across a century because that would take an entire lifetime. With high-resolution neural connections (and a robust understanding of how neurons learn), we could close a mass-based feedback loop in milliseconds-to-minutes.
Given better genetic engineering tech, we might give people a virus which 10x'ed their neuron count without terrible physiological side effects. We very probably can't do that this decade because genetic edits are hard.
But we do have a lot of GPUs.
If we train an AI to send similar patterns as lots of extra (developmentally integrated) neurons would, we've effectively boosted someone's brain mass. I strongly suspect that the relevant algorithms for training such an AI are compactly specifiable, since it seems like neurons learn from simple local firing statistics.
It's like extending brain maturation beyond 2-to-3-decade typical timelines. Existing biological circuits specify some signal similar to a gradient error, and the computronium updates commensurately, just as any section of the universally-learning parts of the brain do.
An incorrect learning rule almost certainly doesn't cause competent agentic misalignment, but it won't make a superintelligent human either. Locating the right rule seems very hard; there does not seem to be consensus on how local updates are implemented.