This is a special post for quick takes by Andrii Vasylenko. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The human brain is complicated. It's made of spaghetti code that depends on being in a narrow range of environmental circumstances to function at all well.
As such, I think that alignment efforts focused on reverse-engineering the brain (e.g.) are unlikely to work; however it is that people often turn out reliably altruistic, it probably doesn't generalize well.
Once I read this and learned about the Wiliams syndrome, my first idea was that it originates from the brain's failure to remove any unused connections, which negatively affects its ability to learn. People affected with such a syndrome tend to become more altruistic, not less.
Additionally, I wonder how the brain is supposed to be reverse-engineered and tested. Cannell's proposal to generate many independent brainlike AIs incapable of telepathic communication would at least have a fair chance to detect the misaligned brains.
Finally, I suspect that the way for people to turn reliably altruistic fails to generalise for reasons different from misconfiguring the brain:
- Unlike the humans, who are only to internalize caring about other humans of similar capabilities, the ASI is to care about the humans who have become as useless to it as pets[1] are to humans living in cities.
- Alongside caring about each other, the humans internalize various ideologies which sometimes cause them to enslave or wipe out other groups of humans not as an act of revenge.
1
1
1
1I was making a more general point that the brain's "alignment" is fragile and circumstance-dependent (training environment, capability level, architecture, etc).
I'm pessimistic about the effectiveness of proposals which aren't robust this sense. E.g. a full solution to the diamond maximizer problem would fit this criterion, while e.g. the proposal you linked wouldn't.