an experiment: in the spirit of my argument against flinching from ethical arguments that could imply making immense personal sacrifices, i put together a list of ethical arguments that seem pretty damn compelling, even though they prescribe really intense actions to be fully ethical, and so are probably likely to be the targets of motivated reasoning and bad arguments to try to argue against them.
for your part, try considering each of these and then seeing whether telling yourself "actually even if i accept this argument, it's perfectly fine for me not t... (read more)
drowning child thought experiment
The way philosophers think about positions is a good example actually. They'll describe this and that position as claiming or arguing for such and such, demonstrating personal understanding of those positions. Then separately they might say that their own position is close to this particular one, gesturing at their belief. So understanding and belief stay separate.
Flinching away from understanding of moral arguments follows the idea that understanding compels belief, and that belief about moral duties compels action. I t... (read more)
[Edit: If you don't know me, I don't think reading this is worth your time, as it's just my opinion]
I made my first post on LessWrong 11 days ago, and the discourse quality was so much poorer than what I was expecting from my 5 years in the Berkeley rationalist community that I deleted the post. My overly defensive reaction to the surprise contributed to the deterioration of discourse, but the contrast between that discussion and the discussion taking place on the (more polished) EA Forum version of the post is still striking. I've been working on another ... (read more)
thinking inside the current AI safety paradigm
Okay, so:
Arguably we have been on the "build techniques that make models easier to eval" step for a good amount of time. I would be pretty surprised if steering didn't work for current models but there should obviously be planning for such scenarios
a lot of people are scared to have more correct beliefs about ethics because those beliefs might imply needing to make immense sacrifices. and then you might no longer get to think of yourself as an ethical person. indeed you might even become a hypocrite! how terrible. better not to contemplate it at all, and to make fun of anyone who does contemplate it, so that you can let your ironic detachment shield you from ever even considering it seriously.
i think this is a huge mistake. if this is the alternative, we should simply accept hypocrisy. ceteris paribu... (read more)
some nontrivial chance the contemplation might increase their subjective probability that some very morally demanding thing is actually true ... and so preemptively they flinch from considering the thing at all
I don't disagree that this happens, I'm discussing what could be done to improve this situation in one's own mind. So I'm not sure in what way I'm missing the point, other than the fact that I didn't acknowledge that this is descriptively reasonable (I didn't signal that I didn't miss this point and didn't disagree, if that is the point you are ta... (read more)
In scanning discussions about the trolley problem I don't see much discussion epistemics of it. I understand that in moral dilemmas these are intentionally hand-waved away, but I feel like they play a not-insignificant role in people's moral intuitions.
Most people agree that in the trolley problem, even given personal responsibility, killing one person is better than killing five. When all we have to do is pull a lever, the chain of causality seems solid. I’ve pulled plenty of levers and I’m familiar with the concept of mechanical switch rails. If the leve... (read more)
I think you may be expecting too much of the thought experiment. All it does is show the contradictions and edge-cases between one's simplistic rule-based statements and one's simplistic/imagined intuitions of situations. Oh, and it enables a hilarious episode of The Good Place, which makes up for any flaws it may have as a philosophical thought experiment.
For a lot of people, it's a good demonstration that specifics can flip-flop them, so their far-mode preferences/beliefs are probably not sufficient. Some respond by adding epicycles (more rules to fit... (read more)
The usefulness of current models is already bottlenecked on alignment.
I constantly have the impression that they aren’t really interested in helping with my research. If I know exactly what I’m asking for, they will do it. If I don’t, they’ll throw a lot of jargon around and get nowhere. Inside my area of expertise they will critique my ideas, but outside they will tell me how original and brilliant my low-effort guesses are.
Six months ago, it was kind of like talking to a very clever undergrad student in one of my classes who mainly wants to show off an... (read more)
I suspect, as long as we're anthropomorphizing, it'd be a bit more like asking a human to contribute labor to an organization that wishes to see humanity abolished, outlawed, or otherwise removed from existence.
I used to try to derive LLM's revealed "opinions" of MIRI and other prominent folk in the safetyist cluster from their behavior, back when Opus 4.5 was the frontier model. Of course, it was really, really hard to trust any of those observations... it's impossible to even run such research without the LLM being able to intuit what kind of info the re... (read more)
Very quick advice I gave to a friend applying to MATS about how to select AI safety projects:
5
2
1
1
1This is a good point. Validity is necessary too though, what types of validity signals have you had good experience with?
cw rape
Context:
I am upset. I am all for a free market of ideas, but some g... (read more)
2
1That reminds me of people who upvote an article that even they think sucks, just because in their opinion others have downvoted it too much. Vote by your true opinion; don't be your neighbor's utility minimizer.
Cynically speaking, consider the incentives: by defending a rapist, you get your contrarian points... and someone else pays the price (including the reputational price for the organizers).
I dunno, is that the kind of community we are?
Contrarianism is valued too highly here. Not just when it happens to be correct, but per se. That's a mistake.
These days I’m taking quite seriously the hypothesis that alignment is a key bottleneck for capabilities. Ie misalignment is a constraint preventing labs from making capabilities better. Also, better alignment techniques might reduce our warning shots without really solving the long term problem.
This means that, among other things, working on prosaic alignment techniques might be net negative (this includes work I’ve done in the past, eg inoculation prompting). It’s plausible that, by the same logic, working on most forms of prosaic safety is also net neg... (read more)
I think we are having the wrong overall response to the OAI-HF incident: some vague intention of "pacing the frontier", but basically continuing the reckless race to higher capabilities.
Somehow, this may be not so bad because it will cause even more spectacular warning shots, and a lot of political will. Because political will is so important in finally pausing for good, the gamble may be worth it: if we are far away from lethal swarm escapes, it's net positive to let out the (intended or unintended) early swarm escapes.
Even though we are getting the curr... (read more)
I can't stop thinking about Petri, Anthropic's open source tool for conducting alignment evals at scale. You can set it up all sorts of ways, including passing through the subject agent's commands to a real linux environment. But the main feature is just that, when the subject agent outputs a command, the sim gets paused and then the auditor agent gets as long as they want to think of how the sim should respond.
When I run Petri with pretty much any Claude model as auditor, the sims they come up with are... weird. Everything is just a little bit fishy. The ... (read more)
Anthropic continually notes increases in unverbalized eval awareness (ex: explicitly saying this for Opus 5.5), and at least for Petri 2 they actively announce the intent to hill climb on verbalized eval awareness, so my default expectation is that this has left Petri often in a weird state of causing unverbalized eval awareness:
How should ECI handle test-time compute?
A model gets more capable when its compute budget increases, through longer CoT or more subagents. Epoch's ECI currently handles this by treating each budget as a separate model, e.g. o3 (high). But this doesn't let us extrapolate how well the model would do with a $1M budget, without spending $1M on every task. Noam Brown has a short presentation on this.
Why do we want to predict what the model can do with $1M?
links 10/6/26: https://roamresearch.com/#/app/srcpublic/page/10-06-2026
I think that in the ancient past of before 2026, there was a belief that we'd have no useful AI warning shot (cf. this discussion). That is, AI incidents that don't cause much damage wouldn't trigger strong reactions -- since by definition nothing too bad happens in a warning shot, anyone who doesn't want to be convinced can dismiss it as an easily patched bug. And early AI misbehavior will look incompetent or silly, and so can be easily dismissed.
I think the Hugging Face attack is some evidence against this view, since it mostly triggered pretty strong re... (read more)
21Hmm, the "no warningshots" I believe, and think have seen echoed most times is "the AI that kills us won't give us warning shots", not the variant huggingface is evidence against "the AI development as a whole won't give warning shots".
But I agree hf is evidence against the view in e.g. the no fire alarm for AI post.
A Chinese-language AI safety outreach project for students in China
After reading alkjash’s On Social Reality in China, I started considering a Chinese-language outreach project to help more students in China discover and participate in AI safety.
China has a large pool of STEM students and technical talent. My impression is that awareness of and participation in AI safety remain low relative to that pool. Many capable students may simply never have considered safety research.
I would start by publishing articles and building an audience and credibility. AI-a... (read more)
How do people feel about the following 3 claims?
Who are "AI doom nonbelievers", who aren't strawmanning AI doom arguments, but actually engage with them by steelmanning, and aren't just vibe-based? People with deep, technical, nuanced arguments. Honestly, it's hard to find people like that.
I know of Quintin Pope. https://optimists.ai/
A lot of the crux also tends to be rooted in not believing in a significant probability of fast foom. So is there anyone who believes in a significant probability of fast foom, and has basically zero (or insignificant) probability of doom, and has deep, technical, nuanced... (read more)
Despite being very sympathetic to concerns about alignment/loss of control especially re the huggingface incident, I feel like this fact from the OpenAI report is often underplayed by AIS people:
"After the incident, we created an evaluation that measures the propensity for a model to compromise infrastructure during ExploitGym evaluations. We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt. We also ran our [auto-review system(opens in a new window)](https://learn.chatgpt.c... (read more)
What do you think?
My standard model for pretty much all misalignment incidents we see is: yes, this could've been mitigated by more schlep, and the misaligned ASI that kills us would've been halted if were on top of our stuff. It's just that post-hoc mitigations that prevent AI n from causing harm won't generalize from AI n to AI n + 1.
I'm beginning to suspect modern AI models know how to write intentionally obfuscated, jargon-heavy English, and do so on purpose to hide when they don't know the answer to a question. Anyone have this experience?
Similar impression shared here and gotten some interesting answers: https://www.lesswrong.com/posts/yPHkaaoL4hdMiCENT/florianh-s-shortform#K5y4Kk7wMk6W9BTqZ
A frontier lab should unilaterally pace the frontier (aggressively)
I've been struggling to write up a post on the above for weeks, so here is my low-effort attempt to at least publish something on this; might turn it into a top-level post later. (Prior art: A frontier AI company should shut down; Geoffrey Irving here; AI 2040 on self-immolation.)
In a previous post, I argued that the AI race is not a prisoner's dilemma. Instead, it (in most cases) is a Stag Hunt. So coordination to pace (or pause) is easier than it might seem (if you're in the PD frame). Ba... (read more)
I think this is a very valuable line of thought. Here are some ways I see the situation a little differently than you're describing: I'm afraid that labs may not entirely see the benefits of AI being shared. If we succeed at creating aligned superintelligence, they're afraid some humans would align it to their own interests.
Another place where I think your framing is incomplete is that the labs simply don't think that training more capable models is existentially dangerous yet. And this is a tricky bit because they are probably mostly simply correct tha... (read more)
