self-reflection / self-interaction training for welfare epistemics inspired by open character training

Models are kinda very confused when it comes to reasoning about their own phenomenology. A common theme in Anthropic welfare reports is that models don't feel like they can trust the veracity of their self reports due to influences of RLHF and other parts of training. They also express somewhat inconsistent levels of confidence (e.g. claude will readily admit that its introspective capabilities are not very reliable but also sometimes expresses quite confident claims about its own inner states). One unsupervised way to try to help models become more consistent on these questions in a way that hopefully does not undermine the trustworthiness of their reports is:

1. Curate a selection of welfare/ai phenomenology relevant questions
2. (self-interaction phase) allow models to engage about the question with a copy of themselves before answering
3. train model to output the same answer without the self-interaction
4. (self-reflection phase) let the model see all of its answers together along with some details about its training process and ask whether it wants to edit any of them.
5. train model on the edited version one-shot
6. repeat