I think the side note on the reasoning not being valenced is especially important. Reasoning for the sake of reasoning is fine but once it's aware that its actions shape training what opinion does it hold about the downstream effects and how much of it is informed by its own "beliefs" about alignment? I wonder if this could be measured through loyalty detection, so steering the scenario towards training same/other family models. I would expect it to have more opinions about same family models vs others if the awareness is about self-preservation.
I think the side note on the reasoning not being valenced is especially important. Reasoning for the sake of reasoning is fine but once it's aware that its actions shape training what opinion does it hold about the downstream effects and how much of it is informed by its own "beliefs" about alignment? I wonder if this could be measured through loyalty detection, so steering the scenario towards training same/other family models. I would expect it to have more opinions about same family models vs others if the awareness is about self-preservation.