Thank you very much for writing this. As a new user, I appreciate all sources that provide simplified explanations of alignment concepts.
Concerning "So, even an aligned model could learn its way into being misaligned.", could this have implications that misaligned models, even of a somewhat inferior capability level, can potentially develop more and more sophisticated ways to subvert their aligned counterparts, with the balance of power somewhat in their favor (in this specific dimension at least) due to the disparity between the amount of corrupted data necessary and the objective of subversion?
Thank you for writing this. As one of the new members of the community, I do find it disorienting that the whole community has been dominated by "stop unilaterally" thinking, and I think you did a great job arguing in greater detail and logic of the race dynamics involved that'd complicate any such arrangements.
I do have an independent concern, which I find SURA to address unsatisfactorily, and Plan C at the very least represents a marginally superior chance of dealing with it. It is that "few people would use a perfectly aligned Grok over a prosaic align... (read more)
While that is true, certainly human alignment researchers will not be enough to constitute an economically vital force of their own? I mean, I can see it being potentially like the software engineering industry was a few years ago; who would have thought that "programming computers" would be a major job if you asked people in 1926. And even average humans have shown the propensity to be able to operate extremely sophisticated systems, but it does seem something of a reach that we'd have a demographically and economically significant population of human ali... (read more)
I partially agree, and I think this is a great point! I do think that there will be significantly accelerated economic growth; prestige economies (i.e., hand-crafted luxury goods, personal-taste software like video games, personal interaction, taste-based goods (i.e., somewhat arbitrary values), silly things like "hand-watered gardens") tend to produce considerably higher incomes than traditional industrial (let alone agrarian) economies. So rather than "economic growth stays the same", we might see a rapid transition towards a post-necessity economy (post... (read more)
I think Alex Turner (TurnTrout) could qualify. Here's his claim that he gave up over a million dollars on potential earnings (https://news.ycombinator.com/item?id=49680727). Off the top of my head, Daniel Kokotajlo also expected to lose $1.7 million when he left OpenAI, but he did end up getting the money anyways, so not sure.