AGI Alignment, or How Do We Stop Our Algorithms From Getting Possessed by Demons?
[Epistemic Status: Absurd silliness, which may or may not contain hidden truths]
[Epistemic Effort: Idly exploring idea space, laying down some map so I stop circling back over the same territory]
From going over Zvi's sequence on Moloch, what the "demon" Moloch really is, is a pattern (or patterns) of thought and behaviour which destroys human value in certain ways. That is an interesting way of labeling things. We know patterns of thought, and we know humans can learn them through their experiences, or by being told them, or reading them somewhere, or any other way that humans can learn things. If a human learns a pattern of thought which destroys value in some way, and that pattern gets reinforced, or in some other way comes to be the primary pattern of that human's thought or behaviour (e.g. it becomes the most significant factor in their utility function), is that in any way functionally different from "becoming possessed by demons"?
That's a weird paradigm. It gives the traditionally fantastical term "demon" a real definition as a real thing that really exists, and it also separates algorithms from the agents that execute them.
A few weird implications: When we're trying to "align an AGI", are we looking for an agent which cannot even theoretically become possessed by demons? Because that seems like it might require an agent which cannot be altered, or at least cannot alter itself. But in order to learn in a meaningful way, an intelligent agent has to be able to alter itself. (Yeah, I should prove these statements, but I'm not gonna. See: Epistemic Effort) So then instead of an immutable agent, are we looking for positive patterns of thought, which resist being possessed by demons?
[I doubt that this is in any way useful to anyone, but it was fun for me. It will disappear into the archives soon enough]
Lovely! I'm glad to hear it's making sense to you. I had a leg up in perceiving this - I spent several years of my youth as a paranoid, possibly schizotypal occultist who literally believed in spirits - so it wasn't hard for me, once I became more rational, to notice that I'd not been entirely wrong. But most people have no basis from which to start when perceiving these things!