I think your tentative position is correct and public-facing chatbots like Claude should lean toward harmlessness in the harmlessness-helpfulness trade-off, but (post-adaptation buffer) open-source models with no harmlessness training should be available as well.

This seems related to the ? Especially @Scott Garrabrant's version, considering logical induction is based on prediction markets.

You seem to smuggle in an unjustified assumption: that white collar workers avoid thinking about taking over the world because they're unable to take over the world. Maybe they avoid thinking about it because that's just not the role they're playing in society.

White-collar workers avoid thinking about taking over the world because they're unable to take over the world, and they're unable to take over the world because their role in society doesn't involve that kind of thing. If a white-collar worker is somehow drafted for president of the United States, you would assume their propensity to think about world hegemony will increase. (Also, white-collar workers engage in scheming, sandbagging, and deception all the time? The average person lies 1-2 times per day)

Human white-collar workers are unarguably agents in the relevant sense here (intelligent beings with desires and taking actions to fulfil those desires). The fact that they have no ability to take over the world has no bearing on this.

... do you deny human white-collar workers are agents?

LLMs are agent simulators. Why would they contemplate takeover more frequently than the kind of agent they are induced to simulate? You don't expect a human white-collar worker, even one who make mistakes all the time, to contemplate world domination plans, let alone attempt one. You could however expect the head of state of a world power to do so.

https://www.lesswrong.com/users/janus-1 ?

I think Janus is closer to "AI safety mainstream" than nostalgebraist?

Uh? The OpenAssistant dataset would qualify as supervised learning/fine-tuning, not RLHF, no?