TL;DR: A potential source of risk from frontier models comes from bad actors purposely training them towards harmful ends or circumventing safety guards: so-called “harmful fine-tuning attacks (HFTAs)”. We summarize a set of immunization conditions that defenses against HFTAs should satisfy. This work was done as part of AI Safety...
This is a short high-level description of our work from AI Safety Camp and continued research on the training-time domain authorization research program, a conceptual introduction, and its implications for AI Safety. TL;DR: No matter how safe models are at inference-time, if they can be easily trained (or learn) to...
[This is a post summarizing the motivation for an AISC 2024 project: If you are interested in participating you can apply here: https://aisafety.camp/ (project 25: Asymmetric control in LLMs: model editing and steering that resists control for unalignment)] Tl;dr: Techniques for AI alignment could equally be used to create misaligned...
TL;DR: A potential source of risk from frontier models comes from bad actors purposely training them towards harmful ends or circumventing safety guards: so-called “harmful fine-tuning attacks (HFTAs)”. We summarize a set of immunization conditions that defenses against HFTAs should satisfy.
This work was done as part of AI Safety Camp (AISC).
The purpose of this post is to extend our discussion of training-time domain authorization (TTDA) with a special case of TTDA that is perhaps most relevant to AI alignment and AI safety: figuring out how to prevent training towards harmful (or illegal) ends. We further scope this down to the setting of natural language generation in current safety-gaurded large language models in order to construct a... (read 3381 more words →)
Thanks for reaching out this is all great feedback.
That we will defenitly address. I will dm you for the vaccine implementation as we are currently working on this as well and to see what would be useful for code sharing since we are wee bit aways from having shareable replication of the whole paper.
Some answers
Thanks for pointing this out - I think its a critical point.
Im not imagining anything in particular (and ya in that paper we do very much do "baby" sized attacks)
Generally ya this is a problem we need to work out: what is the relationship between a defence strength and the budget an attacker would take to overcome this AND for large groups that have budget for training from scratch would defences here even make an impact.
I think your right in that the large budget groups who can just train from scratch would just not be impacted by defences of this nature. While that seems to really poo-poo this whole endevour, I think... (read more)
This is a short high-level description of our work from AI Safety Camp and continued research on the training-time domain authorization research program, a conceptual introduction, and its implications for AI Safety.
TL;DR: No matter how safe models are at inference-time, if they can be easily trained (or learn) to be unsafe then they are fundementally still unsafe. We have some ways of potentially mitigating this.
Training-time domain authorization (TTDA) essentially means that we are looking for a method that makes training a neural model towards some set of behaviours (the domain) either impossible, hard, or expensive (for example in terms of the compute budget of some imagined attacker). Another framing that might fit... (read 1857 more words →)
I think people underestimate formal study of research methods like reading texts / taking a course on research methodology for improving research abilities.
There are many concepts within control, experimental design, validitiy, and reliability like construct validity or conclusion validity that you would learn from a research methods textbook that are super helpful for improving the quality of research. I think many researchers implicitly learn these things without ever knowing what they exactly are but that is usually through trial and errors (peer review rejections and embarrasment) which can be avoided by looking into research methods texts.
Of course this doesn't help with the discovery aspect of research which I think your article is good at outlining but at some point research questions need to be investigated and understanding research design makes it really obvious the kinds of work you need to do in order to have a high quality investigation.
Thanks for the pointer! Yes RL has a lot of research of this kind - as an empirical research I just get stuck sometimes in translation
For my own clarity: What is the difference between mathematical approaches to alignment and other technical approaches like mechanistic interpretability work?
I imagine the focus is on in principal arguments or proofs regarding the capabilities of a given system rather than empirical or behavioural analysis but you mention RL so just wanted to get some colour on this.
Any clarification here would be helpful!
[This is a post summarizing the motivation for an AISC 2024 project: If you are interested in participating you can apply here: https://aisafety.camp/ (project 25: Asymmetric control in LLMs: model editing and steering that resists control for unalignment)]
Tl;dr: Techniques for AI alignment could equally be used to create misaligned models in many cases. We should attempt to develop methods that might only work for alignment and not misalignment to reduce the risk of externalities related to technical alignment research.
The problem
A recent paper led by the Center for AI Safety entitled “Representation Engineering: A Top-Down Approach to AI Transparency” demonstrates a number of comprehensive control techniques in the domain of machine ethics and AI safety.... (read 475 more words →)
If someone did this - it would be nice to collect preference data over answers that are helpful to alignment and not helpful to alignment… that could be a dataset that is interesting for a variety of reasons like analyzing current models abilities to help with alignment, gaps in being helpful w.r.t alignment and of course providing a mechanism for making models better at alignment… a model like this could also maybe work as a specialized type of Constitutional AI to collect feedback from the models preferences that are more “alignment-aware” so to speak… none of this of course is a solution to alignment as the OP points out but interesting nonetheless.
I’d be interested in participating in this project if other folks set something up…
Im struggling to understand how this is is different from “we will build aligned ai to align ai”. specifically: Can someone explain to me how human-like and AGI are different? Can someone explain to me why human-like AI avoids typical x-risk scenarios (given those human-likes could say clone themselves, speed up themselves and rewrite their own software and easily become unbounded)? Why isnt an emulated cognitive system a real cognitive system… i don’t understand how you can emulate a human-like intelligence and it not be the same as fully human-like.
currently my reading of this is we will build human-like AI because humans are bounded so it will be too, those bounds are:... (read more)
What are your thoughts on prompt tuning as a mechanism for discovering optimal simulation strategies?
I know you mention condition generation as something to touch on in future posts but I’d be eager to hear about where you think prompt tuning comes in considering continuous prompts are differentiable and so can be learned/optimized for specific simulation behaviour.
Hey there,
I was just wondering how you deal with hallucination and faithfulness issues of large language models from a technical perspective? The user experience perspective seems clear - you can give users control and consent over what Elicit is suggesting and so on.
However we know LLMs are prone to issues of faithfulness and factuality (Pagnoni et al. 2021 as one example for abstractive summarization) and this seems like it would be a big issue for research where factual correctness is very important. In a biomedical scienario, if a user of Elicit gets an output that presents a figure wrongly extracted (say from a proceeding sentence, or hallucinated as the highest log likelihood... (read more)
Thanks for giving this great work - I certainly agree with you on the limits of unlearning as it’s currently conceived for safety. I do wonder if a paradigm of “preventing learning” is a way to get around these limits.