This work was supported by UChicago XLab.

Today, we are announcing our first major release of the XLab AI Security Guide: a set of online resources and coding exercises covering canonical papers on jailbreaks, fine-tuning attacks, and proposed methods to defend AI systems from misuse.

Each page on the course contains a readable blog-style overview of a paper and often a notebook that guides users through a small replication of the core insight the paper makes. Researchers and students can use this guide as a structured course to learn AI security step-by-step or as a reference, focusing on specific sections relevant to their research. When completed chronologically, sections build on each other and become more advanced as students pick up conceptual insights and technical skills.

Why Create AI Security Resources?

While many safety-relevant papers have been documented as readable blog posts on LessWrong or formatted as pedagogically useful replications in ARENA, limited resources exist for high-quality AI security papers.

One illustrative example is the paper Universal and Transferable Adversarial Attacks on Aligned Language Models. This paper introduces the Greedy Coordinate Gradient (GCG) algorithm, which jailbreaks LLMs through an optimized sequence of tokens appended to the end of a malicious request. Interestingly, these adversarial suffixes (which appear to be nonsense) transfer across models and different malicious requests. The mechanism that causes these bizarre token sequences to predictably misalign a wide variety of models remains unknown.

This work has been covered by the New York Times and has racked up thousands of citations, but unfortunately, there are no high-quality blog posts or instructional coding exercises to understand GCG. Consequently, if students or early-career researchers are interested in further diving into work like GCG, it’s difficult to know where to start.

Given the extensive ecosystem of companies and universities pursuing high-quality research on AI Security, we wanted to parse through it all, find what is relevant, and document it in a readable way. We consider this to be an impactful lever to pull, because it allows us to spread a huge volume of safety-relevant research without having to do the work of making novel discoveries. As for the format of our notebooks, we think that replicating ...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments