The user in the LLM might be a moral patient too (a strange implication of the PSM and simulators)
Thanks to Christopher Ackerman, Margot Stakenborg, Darion Cassel, Andy Han, and Ariana Azarbal for feedback and/or valuable discussions on these topics. TL;DR We think that, conditional on attributing moral weight to the assistant, we should also attribute a non-negligible amount of moral weight to the user persona. Our argument proceeds...