This is a special post for quick takes by Andrii Vasylenko. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The human brain is complicated. It's made of spaghetti code that depends on being in a narrow range of environmental circumstances to function at all well.
As such, I think that alignment efforts focused on reverse-engineering the brain (e.g.) are unlikely to work; however it is that people often turn out reliably altruistic, it probably doesn't generalize well.
I agree. I think if we had a clear understanding how exactly lots of people end up altruistic and what tricks the brain uses to do so we would be in a better spot. Per unit of time I don't think there are a lot of things that taught me as much about the brain as reading Steven Byrnes Brain-Like AGI safety sequence. Certainly more than trying to think about the diamond maximizer problem directly (or reading what other people came up with in the process of doing so). Understanding yourself and your allies is important. The more you know about what a diamond is, the easier it is for you to not fuck up catastrophically on the margin. I am not arguing a lot of people should be working on that.
Once I read this and learned about the Wiliams syndrome, my first idea was that it originates from the brain's failure to remove any unused connections, which negatively affects its ability to learn. People affected with such a syndrome tend to become more altruistic, not less.
Additionally, I wonder how the brain is supposed to be reverse-engineered and tested. Cannell's proposal to generate many independent brainlike AIs incapable of telepathic communication would at least have a fair chance to detect the misaligned brains.
Finally, I suspect that the way for people to turn reliably altruistic fails to generalise for reasons different from misconfiguring the brain:
- Unlike the humans, who are only to internalize caring about other humans of similar capabilities, the ASI is to care about the humans who have become as useless to it as pets[1] are to humans living in cities.
- Alongside caring about each other, the humans internalize various ideologies which sometimes cause them to enslave or wipe out other groups of humans not as an act of revenge.
1
1
1
1I was making a more general point that the brain's "alignment" is fragile and circumstance-dependent (training environment, capability level, architecture, etc).
I'm pessimistic about the effectiveness of proposals which aren't robust this sense. E.g. a full solution to the diamond maximizer problem would fit this criterion, while e.g. the proposal you linked wouldn't.