This is the third blog post for Boaz Barak’s AI Safety Seminar at Harvard University. I intended to condense the lecture into an easily readable format as much as possible.

Author Intro:

Hello to everyone reading this! I am Ege, a Junior at Harvard, studying Statistics and Physics with an intended master’s in Computer Science. My main research interests span improving the reasoning capabilities of models while making said reasoning more explicit and trustworthy, exploring in-context learning capabilities of models and more. I am taking the course to gain a better view of the industry opinion on what is considered trustworthy and safe in the context of Artificial Intelligence, and methods for moving towards that goalpost. If you would like to learn more about me, feel free to visit egecakar.com. For contact, feel free to reach out at ecakar[at]college•harvard•edu.

Outline

We begin with a short summary of the pre-reading, as well as links for the reader.

We then continue with Prof. Barak’s lecture on adversarial robustness and security, and different defense techniques.

Later, we continue with a guest lecture from Anthropic’s Nicholas Carlini, who talks about his research.

Afterwards, we move on to a guest lecture by Keri Warr, most of which will not be included per the speaker’s request.

Lastly, we end with the student experiment by Ely Hahami, Emira Ibrahimović and Lavik Jain.

Pre-readings

Are aligned neural networks adversarially aligned?

Arxiv

While I could summarize this myself, I believe the authors of the paper would do a better job, and present you with the abstract:

“Large language models are now tuned to align with the goals of their creators, namely to be “helpful and harmless.” These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial...

Posts

Wikitag Contributions

Comments