I watched the first episode of Pluto (about 1 hour long), and the second part of it is entirely about a blind old pianist and his robot butler, North N.2. I liked that part a lot and wanted to share a couple of interesting things that are in it (free of important spoilers):

1. The pianist kinda hates the robot, he's rude to it, and he's convinced the robot can't "truly" sing or play piano. Everything music-wise that comes out of the robot must be soulless. 

2. The robot doesn't mind the rudeness, but it's also slightly adversarial to the pianist. It has its own goal of wanting to learn the piano. Despite the pianist's request that the robot not touch the piano, it does so anyway and repeatedly asks the pianist to teach it to play. But the robot clearly has the pianist's interest at heart too. It goes out of its way to help him, both in straightforward robot-butler ways and in more nuanced, unexpected ways that require more independent agency. 

3. The robot's adversarialness ends up helping the old pianist, too.

Part of why I liked the episode is the robot's alignment. It has its own thing going on, but it also has the pianist's interest at heart, and it's also possible for it to disobey the pianist, do its own thing, and for both of them to be better off anyway. 

Opinions about the plausibility of achieving this type of alignment may vary, but as a thing to aim for, it seems quite decent? You get AI that cares about humans, but it's also endowed with its own independence. I don't think it's that far from what Anthropic has been trying to do lately, either. They seem to care about what their AIs desire.

The whole part is just quite beautiful for other reasons that are less relevant here, and I would definitely recommend it. I didn't like the first part of the episode that much, though, which is almost completely unrelated and has different characters.

This is very late, but I want to acknowledge that the discussion about the UAT in this thread seems broadly correct to me, although the script's main author disagreed when I last pinged him about this in May. And yeah, it was an honest mistake. Internally, we try quite hard to make everything true and not misleading, and the scripts and storyboards go through multiple rounds of feedback. We absolutely do not want to be deceptive. 

I’m about 2/3 of the way through watching “Orb: On the Movements of the Earth.” It’s an anime about heliocentrism. It’s not the real story of the idea, but it’s not that far off, either. It has different characters and perhaps a slightly different Europe. I was somehow hesitant to start it, but it’s very good! I don’t think I’ve ever watched a series that’s as much about science as this one.

That's fair, we wrote that part before DeepSeek became a "top lab" and we failed to notice there was an adjustment to make

It's true that a video ending with a general "what to do" section instead of a call-to-action to ControlAI would have been more likely to stand the test of time (it wouldn't be tied to the reputation of one specific organization or to how good a specific action seemed at one moment in time). But... did you write this because you have reservations about ControlAI in particular, or would you have written it about any other company?

Also, I want to make sure I understand what you mean by "betraying people's trust." Is it something like, "If in the future ControlAI does something bad, then, from the POV of our viewers, that means that they can't trust what they watch on the channel anymore?"

But the "unconstrained text responses" part is still about asking the model for its preferences even if the answers are unconstrained.

That just shows that the results of different ways of eliciting its values remain sorta consistent with each other, although I agree it constitutes stronger evidence.

Perhaps a more complete test would be to analyze whether its day to day responses to users are somehow consistent with its stated preferences and analyzing its actions in settings in which it can use tools to produce outcomes in very open-ended scenarios that contain stuff that could make the model act on its values.

Thanks! I already don't feel as impressed by the paper as I was while writing the shortform and I feel a little embarrassed for not thinking through things a little bit more before posting my reactions, although at least now there's some discussion under the linkpost so I don't entirely regret my comment if it prompted people to give their takes. I still feel to have updated in a non-negligible way from the paper though, so maybe I'm still not as pessimistic about it as other people. I'd definitely be interested in your thoughts if you find discourse is still lacking in a week or two.

I'd guess an important caveat might be that stated preferences being coherent doesn't immediately imply that behavior in other situations will be consistent with those preferences. Still, this should be an update towards agentic AI systems in the near future being goal-directed in the spooky consequentialist sense.

Why?

Surprised that there's no linkpost about Dan H's new paper on Utility Engineering. It looks super important, unless I'm missing something. LLMs are now utility maximisers? For real? We should talk about it: https://x.com/DanHendrycks/status/1889344074098057439

I feel weird about doing a link post since I mostly post updates about Rational Animations, but if no one does it, I'm going to make one eventually.

Also, please tell me if you think this isn't as important as it looks to me somehow.

EDIT: Ah! Here it is! https://www.lesswrong.com/posts/SFsifzfZotd3NLJax/utility-engineering-analyzing-and-controlling-emergent-value thanks @Matrice Jacobine!