peterbarnett's Shortform

peterbarnett

peterbarnett's Shortform — LessWrong

Comment Permalink

I am very concerned about breakthroughs in continual/online/autonomous learning because this is obviously a necessary capability for an AI to be superhuman. At the same time, I think that this might make a bunch of alignment problems more obvious, as these problems only really arise when the AI is able to learn new things. This might result in a wake up of some AI researchers at least.

Or, this could just be wishful thinking, and continual learning might allow an AI to autonomously improve without human intervention and then kill everyone.

Showing 3 of 4 replies (Click to show all)

Bronson Schoen10mo74

I think that this might make a bunch of alignment problems more obvious, as these problems only really arise when the AI is able to learn new things. This might result in a wake up of some AI researchers at least.

My best guess is that to the extent these things can be even vaugely associated with task completion / reward hacking / etc, they’re going to be normalized pretty quickly and trained against like other visible misalignment. I think coding agents are an instructive example here.

I also worry we’re getting used to the current era of “the models ar... (read more)

2Seth Herd10mo

I tried to elaborate on that intuition in A country of alien idiots in a datacenter: AI progress and public alarm. When combined with increased deployment from increased deployment and job replacement, I think it will probably wake up the public more than AI researchers. And I think some researchers will just go along with public opinion by accident, since that's the path of least resistance. I also agree that the alarm might not outweigh the increased rate of progress. I also want to note that it doesn't take a breakthrough. Continuous learning already exists (fine-tuning and RAG designed for that purpose, and context engineering). These aren't good enough yet to make major waves or allow really long-term progress, but continued progress will steadily push them to new deployments. Slow steady progress tilts the balance toward the alarm being more beneficial. But maybe not enough.

3gustaf10mo

See also this comment by Vladimir_Nesov + discussion from 4 days ago that starts with [...]

See in context