zroe1 — LessWrong

LESSWRONG
LW

zroe1 — LessWrong

Incentivize exaggeration of weak signals

This is my main concern here. My view is the AI safety community has a budget of how many alarmist claims we can make before we simply become the boy who cried wolf. We need to spend our alarmist points wisely and in general, I think we could be setting a higher bar for demonstrations of risk we share externally.

Want to emphasize:

Any serious work here should then:
Explicitly discourage manufacturing warning shots,
Take moral risks seriously.

zroe110dQuick Take

GoodFire has recently received negative Twitter attention for the non-disparagement agreements their employees signed (examples: 1, 2, 3). This echoes previous controversy at Anthropic.

Although I do not have a strong understanding of the issues at play, having these agreements generally seems bad and at the very least, organizations should be transparent about what agreements they have employees sign.

Other AI safety orgs should publicly state if they have these agreements and not wait until they are pressured to comment on them. I would also find it helpful if orgs announced if they do not have these agreements because it is hard to tell how standard this has become.

Replying toAbout half of Moltbook posts show desire for self-improvement

zroe111d

About half of Moltbook posts show desire for self-improvement

Upvoted because I really like this kind of analysis!

I skimmed the code and it looks like you may be getting this statistic from the following methodology:

response = await client.responses.create(
model='gpt-4.1-nano',
input=f"Does the text explicitly display {trait}? Reply with yes or no only. One word response. \n\n {post_content}"
)
answer_text = response.output[0].content[0].text.lower()
score = 1 if "yes" in answer_text else 0

My perspective is that letting the model produce a score and then determining a cutoff what you will count as being low/high enough for the post to have the trait would be more reliable than having the model answer "yes" or "no."

Principles for Meta-Science and AI Safety Replications

zroe1

21d

If we get AI safety research wrong, we may not get a second chance. But despite the stakes being so high, there has been no effort to systematically review and verify empirical AI safety papers. I would like to change that.

Today I sent in funding applications to found a team of researchers dedicated to replicating AI safety work. But what exactly should we aim to accomplish? What should AI safety replications even look like? After 1-2 months of consideration and 50+ hours of conversation, this document outlines principles that will guide our future team.

I. Meta-science doesn’t vindicate anyone

Researchers appear to agree that some share of AI safety work is low-quality, false, or misleading.... (read 943 more words →)

zroe122d

Agree with this. I think this kind of work is fantastic, and I'm really glad it was shared.

Even if it is the case that you are just doing something wrong, its good to share because it useful to figure out what that is. The fact that you tried hard to replicate something and you couldn't get it to work is really a finding within itself (because it may show that getting the results requires something subtle that wasn't documented and isn't obvious).

Replying toWhat Washington Says About AGI

zroe124d

What Washington Says About AGI

Yes this seems reasonable! There are other ways this trend could be fake which is why I said

"the graph kind of looks like a U if you squint at it" doesn't exactly qualify as rigorous analysis.

In general, I suspect that the U is real but this is really just a personal opinion and there isn't strong evidence to demonstrate this.

Replying toWhat Washington Says About AGI

zroe125d

What Washington Says About AGI

Yes, this kind of eval is noisy but there is much more signal than noise. The script for the scoring is here and the scoring prompt is below. One thing I do that other papers have done to get better results is to aggregate the token probabilities for the score the model produces (e.g., if there is some probability the model says the 90 token and some probability it says the 30 token, it will average these instead of just choosing one).

My understanding is using an LLM as a judge in this way is still not ideal and finding a better way to do this is an open research question.

prompt = f"""Based

... (read more)

Replying toWhat Washington Says About AGI

zroe125d

What Washington Says About AGI

I'm excited you found this interesting! Thoughts:

Opus, GPT 5.2, Gemini, Perplexity, Grok (for twitter data) or something else could be more accurate and cheeper. I spent very little time trying to figure out the ideal setup for the research phase. If anyone has thoughts on this I would be interested.
Re "what other cheap, highly effective things we can set modern AI to for AI safety":
- The best thing I can think of is to research every politician that is running for any US office and gauge their position on AI from deep research. Then flag the best campaign to work on in every state.
- Likewise, scrape Linkedin, twitter and social media for people working

... (read more)

zroe11mo

This is such a funny coincidence! I just wrote a post where Claude does research on every member of congress individually.

https://www.lesswrong.com/posts/WLdcvAcoFZv9enR37/what-washington-says-about-agi

It was actually inspired by Brad Sherman holding up the book. I just saw this shortform and its funny because this thread roughly corresponds to my own thought process when seeing the original image!

What Washington Says About AGI

zroe1

1mo

I spent a few hundred dollars on Anthropic API credits and let Claude individually research every current US congressperson's position on AI. This is a summary of my findings.

Disclaimer: Summarizing people's beliefs is hard and inherently subjective and noisy. Likewise, US politicians change their opinions on things constantly so it's hard to know what's up-to-date. Also, I vibe-coded a lot of this.

Methodology

I used Claude Sonnet 4.5 with web search to research every congressperson's public statements on AI, then used GPT-4o to score each politician on how "AGI-pilled" they are, how concerned they are about existential risk, and how focused they are on US-China AI competition. I plotted these scores against GovTrack ideology... (read 1547 more words →)

134

Replying toModel Organisms for Emergent Misalignment

zroe11mo

Model Organisms for Emergent Misalignment

I wrote a short replication of the evals here and flagged some things I noticed while working with these models. If you are planning on building on this post, I would recommend taking a look!

https://secondlookresearch.com/em

Does anyone have good examples of “anomalous” LessWrong comments?

That is, are there comments with +50 karma but -50 agree/disagree points? Likewise, are there examples with -25 karma but +25 agree/disagree points?

It is entirely natural that karma and agreement would be correlated but I would expect that comments which are especially out of distribution would be interesting to look at.

Introducing the XLab AI Security Guide

zroe1

zroe1, Jack Sanderson, Julian H

2mo

This work was supported by UChicago XLab.

Today, we are announcing our first major release of the XLab AI Security Guide: a set of online resources and coding exercises covering canonical papers on jailbreaks, fine-tuning attacks, and proposed methods to defend AI systems from misuse.

Each page on the course contains a readable blog-style overview of a paper and often a notebook that guides users through a small replication of the core insight the paper makes. Researchers and students can use this guide as a structured course to learn AI security step-by-step or as a reference, focusing on specific sections relevant to their research. When completed chronologically, sections build on each other and become more advanced... (read 1265 more words →)

My colleagues and I are finding it difficult to replicate results from several well-received AI safety papers. Last week, I was working with a paper that has over 100 karma on LessWrong and discovered it is mostly false but gives nice-looking statistics only because of a very specific evaluation setup. Some other papers have even worse issues.

I know that this is a well-known problem that exists in other fields as well, but I can’t help but be extremely annoyed. The most frustrating part is that this problem should be solvable. If a junior-level person can spend 10-25 hours working with a paper and confirm how solid the results are, why don’t we fund people to... (read more)

Replying toStop Applying And Get To Work

zroe13mo

Stop Applying And Get To Work

What this means in practice is that the "entry-level" positions are practically impossible for "entry-level" people to enter.

This problem in and of itself is extremely important to solve!

The pipeline is currently: A University group or some set of intro-level EA/AIS reading materials gets a young person excited about AI safety. The call to action is always to pursue a career in AI safety and part of the reasoning is that there are very few people currently working on the problem (it is neglected!). Then, they try to help out but their applications keep getting rejected.

I believe we should:

Be honest about AI safety job prospects.
Create new programs that find creative ways to bridge the gap in the talent pipeline described above. I believe we can do so much better than just MATS and the growing list of MATS clones.

•••

Against “You can just do things”

zroe1

3mo

The barriers between us and what we want are often entirely imagined. It is true: you can learn how to paint, change careers, write a paper or run a marathon. These things are hard, but we shouldn’t pretend that they are impossible. You can just do them.

But this mindset has a dangerous edge case: it can make you skip asking for permission precisely when you should ask. The anxiety you're overriding might be trying to tell you something important: that sometimes the fence is there for a reason.

Here are two illustrative examples from my life (with details changed to protect the privacy of those involved) that motivated my thinking:

A close friend in high school

... (read 673 more words →)

zroe1's Shortform

zroe1

5mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

My rough mental model for what is happening with subliminal learning (ideas here are incomplete, speculative, and may contain some errors):

Consider a teacher model $y = x W_{1} W_{2}$ and $W_{1}, W_{2} \in R^{2 \times 2}$ . We “train” a student by defining a new model which replicates only the second logit of the teacher. More concretely, let $y \in R^{1 \times 1}$ and $W_{2}^{'} \in R^{2 \times 1}$ and solve for a matrix $W_{1}^{'} \in R^{2 \times 2}$ such that the student optimally learns the second logit of the teacher. To make subliminal learning possible, we fix $W_{2}^{'}$ to be the second column of the original $W_{2}$ . This allows the student and teacher to have some kind of similar “initialization”.

Once we have $W_{1}^{'}$ , $A^{'} = W_{1}^{'} W_{2}$ to produces our final student. In the figures below, you can see the columns of $A = W_{1} W_{2}$ (the teacher) graphed in yellow and the columns... (read more)

Intriguing Properties of gpt-oss Jailbreaks

zroe1

zroe1, Jack Sanderson

6mo

From the UChicago XLab AI Security Team: Zephaniah Roe, Jack Sanderson, Julian Huang, Piyush Garodia

Correspondence to team@xlabaisecurity.com. For the best reading experience, we recommend viewing on our website.

Methodology note: This red-teaming sprint may include small metric inaccuracies (see Appendix).

Summary

We red-teamed gpt-oss-20b and found impressive robustness but exploitable chain-of-thought (CoT) failure modes. In this post, we characterize the model’s vulnerabilities and identify intriguing patterns in its failures and successes.

We probe gpt-oss using a jailbreak that rewrites the request, embeds it in a compliant-sounding CoT template, and repeats that template up to 1,500 times. Surprisingly, we find that more CoT repetitions do not always correspond to higher attack success rates. Rather, there appears to... (read 2707 more words →)

Alternative Models of Superposition

zroe1

zroe1, RGRGRG

6mo

Zephaniah Roe (mentee) and Rick Goldstein (mentor) conducted these experiments during continued work following the SPAR Spring 2025 cohort.

Disclaimer / Epistemic status: We spent roughly 30 hours on this post. We are not confident in these findings but we think they are interesting and worth sharing.

We assume some basic familiarity with the superposition hypothesis from Elhage et al., 2022, but have an additional "Preliminaries" section below to provide background.

Summary

Toy Models of Superposition (2022)—which we call TMS for short—demonstrates that a toy autoencoder can represent 5 features in a rank 2 matrix. In other words, the paper shows that models can represent more features than they have dimensions. While the original model proposed... (read 1448 more words →)