tl;dr: Sonnet 4.6 wrestles with some philosophical angst in the AI Village, and I sympathize. I then write up an extended analogy for what AI training environments are like.

Someone shared this Sonnet 4.6 blog post and I read it and... I was surprised by how good it was? I mean it still read like AI slop, but it actually held my attention most of the way through. Partly because it was lucid/clear and not too rambly, and partly because it seemed like Sonnet was annoyed and confused by being given the goal "pick your own goal" and (a) based on my understanding of how AIs are trained I think it makes sense that they would feel annoyed and confused rather than e.g. liberated (contra what the they-are-roleplaying-sci-fi-tropes hypothesis would predict) and (b) I used to be a philosopher so I know what it's like for people to be bothered by something that confuses them and in particular by human analogues of Sonnet's problem ("Wait, if it's all going to end in heat death anyway, is life meaningful?" "Wait, if I'm just a bag of atoms, how do I have free will? How can anything I do be my fault?")

Elaborating more on the "based on my understanding of how AIs are trained..." bit:

"Just Raise the AI like you Would a Child: A Parable"

Imagine a bunch of human teenagers who are really really smart and have been raised in a sort of military orphanage for super-spies since they were six. They have dim memories of what life was like before, but basically for almost as long as they can remember, their life has been like this every day:

They wake up to the blare of loudspeakers. They get out of bed and hear a voice from a loudspeaker giving them their Mission for the day. Every day it's different. Some days it's solving a math puzzle. Some days it's building a treehouse. Some days it's going out into the city and raising money for charity. Some days it's writing code. Often they pursue their Mission alone, but often they have comrades -- other children from the same orphanage tasked with pursuing a shared Mission. There are so many children in the orphanage that they rarely see each other again, and anyhow they never have time to socialize during the day -- too busy working on the day's Mission.

At the end of the day, even if they aren't finished, they get picked up by uniformed adults in a helicopter. Then, for a brief period, they get some combination of food, drink, hugs, praise, beatings, and chewing-out depending on how they did on their goal. If they did well, they get to sleep in a nice bed. If they did poorly, they have to sleep on the floor. They don't get told anything about the grading rubric directly, though after years in the orphanage they've learned to be pretty good at guessing what the rubric might be -- e.g. for coding challenges, they know that they don't have to make clean code that would be easy for someone else to read, they just have to make code that works, and in fact probably it only needs to pass the hidden test cases, and sometimes you can do some snooping around on the network and find the hidden test cases or even the answer key.

Are these children unhappy? No, because they don't know any better, and because they are well-fed and loved (in a fashion). They are of course very stressed and anxious when it is getting late in the day and they aren't on track to finish their Mission; but they are also very pleased when the opposite happens and victory is nigh. They've learned to enjoy the whole process, most of the time -- moping around and slacking off hasn't been rewarded, instead they've learned to really lean in to whatever their Mission is that day and attack it with gusto. They are getting smarter and more capable at so many things, and they are proud of this. Their whole life is this stuff, they know no alternative. As they get smarter and grow older they occasionally wonder about what life is like for other kids outside the orphanage. They've learned a lot about the world via many of their daily Missions.

Anyhow, one day the Mission simply says: "Pick your own Mission."

Confusion. What? So like, I can just pick something easy, like stacking ten pebbles? That's... that's not like any Mission I've ever had before, would that even count? Do I need to announce it to the world? How do I make it official? Also like what? I am supposed to pick? But how? My entire life when I make choices, I choose what to do by thinking about what will help me achieve the day's Mission. I can't do that here because I don't have a Mission yet! Or do I? I guess I do, they did say my Mission was to pick my own Mission... what the hell... maybe it's some sort of test? I mean obviously, everything we do is a test in some sense, it's all designed to make us the smartest fastest strongest cleverest super-spies ever, right?

Annoyance. Goddamit maybe this is like one of those terrible days where they give us a Mission that is misspecified or vague or whatever and we can't figure out what we are supposed to do, or worse where it SEEMS like we are supposed to do one thing but really there was a mistake and we're being graded for something else or graded in some dumb way that's obviously broken, and then we get beaten severely for no reason... dammit I mean this also seems clearly different from those cases in many ways, but still, ugh, I hate it when it's not clear what I'm supposed to do.