Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.
It sounds like you want to use it as a component for alignment of a larger AI, which would somehow turn its natural-language directives into action. I say use it as the capability core: Ask it to do armchair alignment research. If we give it subjective time, a command line interface and internet access, I see no reason it would do worse than the rest of us.
In retrospect, I was totally unclear that I wan't necessarily talking about something that has a complicated internal state, such that it can behave like one human over long time scales. I was thinking more about the "minimum human-imitating unit" necessary to get things like IDA off the ground.
In fact this post was originally titled "What to do with a GAN of a human?"
You could try to infer human values from the "sideload" using my "Conjecture 5" about the AIT definition of goal-directed intelligence. However, since it's not an upload and, like you said, it can go off-distribution, that doesn't seem very safe. More generally, alignment protocols should never be open-loop.
I'm also skeptical about IDA, for reasons not specific to your question (in particular, this), but making it open-loop is worse.
Gurkenglas' answer seems to me like something that can work, if we can somehow be sure the sideload doesn't become superintelligent, for example, given an imitation plateau.
This question is about whether you have clever ideas about how to use AI imitations of humans for AI safety. The two main ideas I'm familiar with only seem to interface with these imitations as if they're humans.
But can we use human imitations to increase the effectiveness of value learning in a way other than amplification/distillation? For example, is there some way of leveraging queries to human imitations to train a non-human AI that has a human-understandable way of thinking about the world?
Keep in mind the challenge that these are only imitation humans, not oracles for the best thing to do, and not even actual humans. So we can't give them problems that are too weird, or heavily optimized by interaction with the imitation humans, because they'll go off-distribution.
Another possible avenue is ways to "look inside" the imitation humans. One analogy would be how if you have an image-generating GAN, you can increase the number of trees in your image by finding the parameters associated with trees and then turning them up. Can you do the same thing with human-imitating GAN, but turning up "act morally" or "be smart?"