In a lab experiment where it was told it was going to be shut down, Claude sometimes blackmailed fictional employees who controlled its shutdown button (again, we also tested frontier models from all the other major AI developers and they often did the same thing).

Why is it always the blackmail result that gets reported from this paper? Frontier models were also found willing to cause a fictional employee's death to avoid shutdown. It's weird to me that that's so often ignored.

Here's another justification for hyperbolic discounting, drawing on the idea that you're less psychologically connected to your future selves.

I've always seen this idea attributed to Martin Weitzman, and he cites these papers as making a similar point. Seems like an interesting case of simultaneous discovery: four papers making the same sort of point all appearing between 1996 and 1999.

What's your current view? We should aim for virtuousness instead of corrigibility?

are uploads conscious? What about AIs? Should we care about shrimp? What population ethics views should we have? What about acausal trade? What about pascal's wager? What about meaning? What about diversity?

It sounds like you're saying an AI has to get these questions right in order to count as aligned, and that's part of the reason why alignment is hard. But I expect that many people in the AI industry don't care about alignment in this sense, and instead just care about the 'follow instructions' sense of alignment.

Yeah I think the only thing that really matters is the frequency with which bills are dropped, and train stations seem like high-frequency places.

More reasons to worry about relying on constraints:

• As you say, your constraints might be insufficiently general ('nearest unblocked strategy,' etc. This seems like a big issue to me. People like Jesus and the Buddha seem to have gained huge amounts of influence without needing to violate any obvious deontological constraints.)
• Your constraints might be insufficiently strong (e.g. maybe the constraints are strong enough to keep the AI compliant all throughout training but then the AI gets a really great opportunity in deployment...).
• Your constraints might be just 'outer shell,' like humans' instinctual fear of heights (Barnett and Gillen). The AI might see them as an obstacle to overcome, rather than as a part of its terminal values.
• Your constraints might actually be false beliefs that later get revised (e.g. that lying never pays)(Barnett and Gillen).
• Your constraints might cause theoretical problems that motivate the AI to revise them away (e.g. money pumps, intransitivities, violations of the Independence of Irrelevant Alternatives, implausible dependence on which outcome is designated as the status quo, paralysis, trouble dealing with risk, arbitrariness of constraints' exact boundaries).
• Your constraints might cause other misalignments (e.g. the AI wants to take extreme measures to prevent other agents from lying too).
• Your constraints might make the AI incapable (e.g. they might falsify the strategy-stealing assumption, or make AIs too timid [e.g. maybe the AI will be extremely reluctant to say anything it's not absolutely certain of]).
• Your constraints might fail to motivate the AI to do good alignment work (e.g. the AI produces alignment slop).
• Your constraints might make the AI bad at moral philosophy (and we might need AI-powered moral philosophy to get a really good future).

Yeah risk aversion can only make the AI cooperate if the AI thinks that getting paid for cooperation is more likely than successful rebellion. It seems pretty plausible to me that this will be true for moderately powerful AIs, and that we'll be able to achieve a lot with the labor of these moderately powerful AIs, e.g. enough AI safety research to pre-empt the existence of extremely powerful misaligned AIs (who likely would be more confident of successful rebellion than getting paid for cooperation).

They don't have to be short-term oriented! Their utility function could be:

$u=f\left({\Sigma }_i{p}_i\right)$

Where $f$ is some strictly concave function and $p_i$ is the agent's payment at time $i$. Agents with this sort of utility function don't discount the future at all. They care just as much about improvements to $p_i$ regardless of whether $i$ is 1 or 1 million. And yet, for the right kind of $f$, these agents can be risk-averse enough to prefer a small salary with higher probability to a shot at eating the lightcone with lower probability.