I was wondering if there has been any work getting around specifying the "correct" decision theory by just using a more limited decision theory and adjusting terminal values to deal with this.

I think we might be able to get an agent that does what we want without formalizing the right decision theory buy instead making a modification to the value loading used. This way, even an AI with a simple, limited decision theory like evidential decision theory could make good choices.

I think that normally when considering value loading, people imagine finding a way to provide the AI answers to the question, "What preference ordering over possible worlds would I have, after sufficient reflection, which I would then use with whatever decision theory I would use upon sufficient reflection?". My proposal is to instead make an evidential decision theory and change value-loading to instead answer the question, "What preference ordering would I, on sufficient reflection, want an agent that uses evidential decision theory to have"? This could be used with other decision theories, too.

In principle, you could make an evidential-decision-theoretic agent take the same actions an agent with a more sophisticated decision theory would.

One option is to modify the utility function to have a penalty for doing things contrary to your ideal decision theory. For example, suppose you, on reflection, would think that functional decision theory is the "correct" decision theory. Then when specifying the preference ordering for the agent, you could provide a penalty in situations in which the agent does something contrary to what functional decision theory would recommend.

Another option is to include preferences about mathematical objects representing what would have happened in some other logically possible world if the agent did a certain action. Then, the AI could have preferences about what that mathematical construct outputs. To be clear, though the construct is about what would happen in some other possible world, it's an actual mathematical object, and statements about it are still true or false in the real world.

For example, suppose an AI is considering giving in to xor-extortion. Then the AI could see that, conditioning on it having a given output, AI's like it in other possible worlds would on average do worse, and preferences against this could be loaded.

I don't see anything implausible about being able to load preferences like those described in the second question into an AI, nor a clear reason to think is would be harder than loading preferences that answer the first one. Some of the techniques for value-loading I've seen involve getting the AI to learn terminal values from training data, and you could modify the learned terminal values by modifying the training data appropriately.

Another potential technique to use in value-loading is to somehow pick out the people in the AI's world model and then query them for their values. Techniques like this could potentially be used to allow for appropriate loading of terminal values, for example, by querying people's brains for a question like "what would you, on reflection, want an evidential-decision-theoretic agent to value?", rather than what "would you, on reflection, what an agent using whatever decision theory you actually use to value?"

The advantage of using a simple decision theory and adjusting value loading is that the AI makes the right choice for what we want by just correct value-loading and just implementing a basic, easy decision theory, like evidential decision theory.