x

LESSWRONG
LW

jknowak

Top postsTop post

jknowak

Subscribe

Message

business data analyst / ex insurance broker
upskilling in ML Safety following a plan based on Gabe Mukobi's guide
volunteer at EA Poland
based in Poznań, Poland

5

4

4y

jknowak

Top postsTop post

Activation Engineering Theories of Impact

Below I summarize the thoughts of other people on what's the Theory of Impact for Activation Engineering. I mostly base it on the "discussion" parts of the papers and the answers under @Chris_Leong's post What's the theory of impact for activation vectors? Alex Turner's posts on controlling a maze-solving policy network, and a paper on steering GPT-2-XL by adding an activation vector, introduced activation engineering, a set of "techniques which steer models by modifying their activations". > As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime. Over a year later and there's a Slack server to coordinate research projects and propose new ideas or open problems in this area. This is, to my knowledge, the closest we have to a practical implementation of retargetting the search. ToIs Low-tax value alignment of LLMs The original "activation addition" paper claims that > activation engineering can flexibly retarget LLM behavior without damaging general performance. We speculate that this involves changing the model’s currently-active (mixture of) goals. Suitably developed, the activation engineering approach could enable safety progress while incurring a very low ‘alignment tax’ Alex Turner claims (as far as I understand) that that steering vectors can significantly enhance a model's performance on key aspects of safety, such as truthfulness, reducing the tendency to hallucinate or generate false information, minimizing sycophancy or overly agreeable responses, discouraging power-seeking behavior, and mitigating myopic tendencies. Activation vectors don't work in isolation; they can be effectively combined with existing techniques like prompting and fine-tuning, leading to even greater improvements. This means that activation vectors may represent a valuable addition to the growing arsenal of tools available to AI researchers and developers striving to create more aligned and beneficial

6Jul 18, 2024

Top postsTop post

jknowak

Subscribe

Message

business data analyst / ex insurance broker
upskilling in ML Safety following a plan based on Gabe Mukobi's guide
volunteer at EA Poland
based in Poznań, Poland

5

4

4y

[Linkpost] "Exploiting" the In-Kind Donation Bias

> [...] It's not just a disaster-relief problem. It is also visible when corporations do CSR days, or how people put up things for charity auctions. The question is: why does giving stuff feel so much better than giving cash? > > [...] > > This might sound like I’m...

Mar 171

[Linkpost] Let's Start Eventmaxxing

> We should organize more events. You should try organizing some if you are not doing it already. > > [...] > > I came to a conclusion that well-run events are too rare in our lives and that there should be more of them. Events add real value, most...

Mar 170

[Linkpost] Many Faces Of Problem Decomposition

> I love thinking tools and frameworks, and encountered many of them by working in design, finance, consulting, IT, and non-profits. I got excited when I saw many of them fit a similar pattern. This is a linkpost for the first post on my Substack, that I wrote in December...

Mar 171

Activation Engineering Theories of Impact

Below I summarize the thoughts of other people on what's the Theory of Impact for Activation Engineering. I mostly base it on the "discussion" parts of the papers and the answers under @Chris_Leong's post What's the theory of impact for activation vectors? Alex Turner's posts on controlling a maze-solving policy...

Jul 18, 20246

Top postsTop post

Activation Engineering Theories of Impact

Below I summarize the thoughts of other people on what's the Theory of Impact for Activation Engineering. I mostly base it on the "discussion" parts of the papers and the answers under @Chris_Leong's post What's the theory of impact for activation vectors? Alex Turner's posts on controlling a maze-solving policy network, and a paper on steering GPT-2-XL by adding an activation vector, introduced activation engineering, a set of "techniques which steer models by modifying their activations". > As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime. Over a year later and there's a Slack server to coordinate research projects and propose new ideas or open problems in this area. This is, to my knowledge, the closest we have to a practical implementation of retargetting the search. ToIs Low-tax value alignment of LLMs The original "activation addition" paper claims that > activation engineering can flexibly retarget LLM behavior without damaging general performance. We speculate that this involves changing the model’s currently-active (mixture of) goals. Suitably developed, the activation engineering approach could enable safety progress while incurring a very low ‘alignment tax’ Alex Turner claims (as far as I understand) that that steering vectors can significantly enhance a model's performance on key aspects of safety, such as truthfulness, reducing the tendency to hallucinate or generate false information, minimizing sycophancy or overly agreeable responses, discouraging power-seeking behavior, and mitigating myopic tendencies. Activation vectors don't work in isolation; they can be effectively combined with existing techniques like prompting and fine-tuning, leading to even greater improvements. This means that activation vectors may represent a valuable addition to the growing arsenal of tools available to AI researchers and developers striving to create more aligned and beneficial

6Jul 18, 2024

[Linkpost] Many Faces Of Problem Decomposition

1Mar 17

[Linkpost] "Exploiting" the In-Kind Donation Bias

1Mar 17

[Linkpost] Let's Start Eventmaxxing

0Mar 17