red_light's Shortform

red_light

red_light's Shortform — LessWrong

Comment Permalink

https://rikiyat.github.io/blog/weight-space-programming/

Based only on raw weights, Astra 6 is able to tell what a model was trained to do and directly edit, without environmental feedback, those model weights to drastically increase performance on various tasks.

~~RSI probably immediately falls out of these capabilities, and we do not know if OpenAI has RL'd for this.~~
RSI does not fall out of these capabilities, but we have no reason to believe that these were well elicited.^[1] If these capabilities can be improved through RL, then I think we have reason for concern.

^{^}
These capabilities seem to be emerging on the frontier, so they might be elicited: https://x.com/alexkrentsel/status/2117376683724660898?a=zL0qV4at-5PBm8oi_HY&s=46

Showing 3 of 39 replies (Click to show all)

1Alex Gibson6h

"Aligning a model to humans" is a capability. They're currently a little good at it, and so one could imagine them getting exponentially better at it. "surgically removing or altering safety restrictions." - What even is the theory here. Things have gone terribly wrong by the time models have access to their own weights. There's lots of examples of capabilities that have shown up in weak form for years which haven't foomed, so idk what you mean by extremely high base rate.

1Haiku6h

Prosaically, yes, but they might not want to do so. Even if they wanted to, alignment might get harder faster than AI gets better at it, due to a series of hard thresholds, including capabilities paradigm changes. More straightforwardly, superalignment is a fundamentally different thing, and AI has shown no measurable proficiency at that. (Arguably, neither have humans.) [...] Yes. [...] I didn't mean to imply hyperbolic growth. In fact, some capabilities increases have been linear rather than exponential. The vast majority have been exponential, though, on whatever chosen metric. If you have an example of any cognitive task that AI could do a little bit / poorly in 2024, and that it can still only do a little bit / poorly in 2026, I would be interested to know that. I'm not aware of any such example, and I don't want to spread misinformation if examples do exist.

Alex Gibson6h10

You're not aware of a cognitive task that models are poor at?

See in context