This work was supported by UChicago XLab. Today, we are announcing our first major release of the XLab AI Security Guide: a set of online resources and coding exercises covering canonical papers on jailbreaks, fine-tuning attacks, and proposed methods to defend AI systems from misuse. Each page on the course...
From the UChicago XLab AI Security Team: Zephaniah Roe, Jack Sanderson, Julian Huang, Piyush Garodia Correspondence to team@xlabaisecurity.com. For the best reading experience, we recommend viewing on our website. Methodology note: This red-teaming sprint may include small metric inaccuracies (see Appendix). Summary We red-teamed gpt-oss-20b and found impressive robustness but...
This work was supported by UChicago XLab.
Today, we are announcing our first major release of the XLab AI Security Guide: a set of online resources and coding exercises covering canonical papers on jailbreaks, fine-tuning attacks, and proposed methods to defend AI systems from misuse.
Each page on the course contains a readable blog-style overview of a paper and often a notebook that guides users through a small replication of the core insight the paper makes. Researchers and students can use this guide as a structured course to learn AI security step-by-step or as a reference, focusing on specific sections relevant to their research. When completed chronologically, sections build on each other and become more advanced... (read 1265 more words →)
I think that there's probably a better framing to this argument than "gradient routing is better than pretraining filtering." Assuming it's a binary decision and both are equally feasible for the purpose of safety:
People have correctly pointed out that this isn't the case for causing emergent misalignment, but it notably is for causing safety mechanisms to fail in general and is also the likely reason why attacks such as adversarial tokenization work.
From the UChicago XLab AI Security Team: Zephaniah Roe, Jack Sanderson, Julian Huang, Piyush Garodia
Correspondence to team@xlabaisecurity.com. For the best reading experience, we recommend viewing on our website.
Methodology note: This red-teaming sprint may include small metric inaccuracies (see Appendix).
We red-teamed gpt-oss-20b and found impressive robustness but exploitable chain-of-thought (CoT) failure modes. In this post, we characterize the model’s vulnerabilities and identify intriguing patterns in its failures and successes.
We probe gpt-oss using a jailbreak that rewrites the request, embeds it in a compliant-sounding CoT template, and repeats that template up to 1,500 times. Surprisingly, we find that more CoT repetitions do not always correspond to higher attack success rates. Rather, there appears to... (read 2707 more words →)