Cross-posting from a Twitter thread responding to a recent viral comments by @Richard_Ngo about EA, Anthropic, and AI safety as a 'fake field.' Posting here because I expect this to be quite unpopular on LW.

(original thread: https://x.com/CRSegerie/status/2056737155880493357)

AI safety in 2023–2026 was driven by evals, threat models, scary demos, model-organism work, RSPs, and voluntary commitments. Richard calls this "much more of a fake field" and says it "won't generalize".

Here's why I disagree - 1/10

1/ I agree with Anthropic being now the biggest le... (read more)

I've been reading an anthropology about the Australian aboriginals, The Native Tribes of Central Australia (Baldwin Spencer/Francis James Gillen, 1899), and found some parts interesting enough to share them.

Content warning: description of gruesome (though consensual) mutilation.

Things that stood out about the aboriginals (highlighting not in the original text):

• Aboriginals experienced nocebo effects strong enough to result in death, even from mild injuries, if the weapon causing the injury was believed to be enchanted ^[1]
• Each aboriginal man ha

I have a bone to pick with high school chemistry textbooks.

When I took chemistry in high school, there was a section of the textbook that described some features of quantum mechanics and tried to explain electron orbitals. The orbitals chapter in particular made basically no sense; I was able to extract enough out of it to be able to do the homework problems, but most of the class was totally lost. When I took a semester of modern physics in college (and also read Eliezer's QM sequence), I learned that the reason the explanation made no sense was that it w... (read more)

a theory of assistant personas and superhuman capabilities

so you have a language model. you train it to embody some specific personality--Claude, ChatGPT, whatever. one of the miracles of AI is that this mostly works and gives you something that is mostly trying to help you and not trying to murder you. i claim that this is mostly because of the SL training objective and if you do just the intense RL thing you get the originally predicted spicy alignment failures.

suppose you tell the LM that Claude is actually a superhuman aligned AI. can you get superhuma... (read more)

It seems to me that the hardest part of getting ai right will be coordination and governance, not technical alignment. Under this view, I think the EA/LW forums are net negative. They are really bad at creating coordination environments and are actively crowding out similar but more coordination/governance environments from existing.

Xela's Learning Diary

In case some people want to see a (hopefully more filtered?) version of the 'learning diary' I mentioned before, I'll comment stuff as replies to this post.

These will usually be about stuff I learned before the date published - but the date on the entry should be roughly right. It'll also be more verbose, since I can't assume you already know what I do.

It's incredibly easy to be fooled by the capabilities of the current top-performing tech (LLM agents). It's easy because they have a vast amount of training data to interpolate from.

This works fine to acquire capabilities within our existing data distribution of the world (one that is also easy to verify), but what happens when they go out of distribution?

LLMs perform poorly! Yet, people seem to think they can actually generalize to new problems. Why is that?

It's, again, the vastness of their training data. It makes it hard to distinguish between interpola... (read more)

Negation Neglect kinda makes sense to me. The argument goes that (at least in SFT though people think it generalizes to pretraining and RL) if you train/fine-tune on a text that starts with something like "the following are all lies that are not true: <claims> what we said before are all lies that are not true", the next-token completion nature of LLMs means they ingest the local claims as credible. Updating too much on the opening warning is too rough[1] for a greedy optimization process.

Inoculation Prompting also kinda makes sense to me. The argume... (read more)

The unlikely hypothetical scenario in which hantavirus is on track to become a pandemic is becoming less likely day by day. We continue to see Hondius-linked infections limited exclusively to people who were aboard.

Yet Polymarket's had the price of "yes" on the "hantavirus pandemic 2026" market sitting on 7.1-7.2 cents for the last four days. Day by day, this is looking more like a price that reflects the opportunity cost of holding shares of "no" through the rest of 2026 to make a 7.6% profit (coupled with the risk of market tampering), rather than the market's judgment of the likelihood of a hantavirus pandemic.

Observation: Opus 4.7 (via chat) is near-useless for helping me iterate on google docs. For one, its reading comprehension seems very poor.

This is surprising because others say that the models give them tons of uplift. (And the main METR graph is just coding but still. And Opus 4.6 beat all humans on the GovAI work test. And the vibes!)

Hypotheses (not exclusive or exhaustive):

• Claude Code and Cowork are much better, even for tasks like "iterate on a google doc"
• Other models are much better for work like mine
• I'm prompting the model terribly (or it's better at

Occasionally there will be a piece of media that seems like it's glorifying violence (One Battle After Another, Joker), and people will say "this is unprofessional, it might lead to violence!" And then it won't be followed up by any obvious violence, and other people will say "ha, you were being ridiculous." But the more important effect might be to lower public trust, in the same way yelling "I would LOVE to kill my political enemies" hurts public trust even if you don't blow up a courthouse.

it's actually crazy how much ubering pareto dominates driving in a city like SF. you don't have to worry about parking, you can work while in transit, you can get a bigger car when needed, you don't need to round trip, etc. it's generally even cheaper once you take depreciation, parking, insurance, etc costs into consideration.

"to the success of our hopeless cause" is such a good toast and we should use it more often. i first learned of it from the book of the same name, and apparently it was a common refrain at gatherings of Soviet dissidents. i like it because it captures the feeling of trying really hard to succeed despite being in the basement of the logistic success curve, and somehow, despite all odds, actually succeeding in the end.

Are there any other mechanistic interpretability mentorship programs I should apply for in addition to MATS and the Anthropic Fellows Program? I think I know enough about the field and I'm semi-competant at ML but need more legible output and a network.

in the same way that Minecraft teaches you to exercise agency and Factorio teaches you to optimize, are there any games that teach you to stare into the abyss? the ideal game would (a) reward you on a tight feedback loop for constantly admitting that you were wrong, (b) give you the option to not admit that you were wrong but make that decision acutely hurt. pastcasting is good for (a) but not good for (b) because you are sort of forced to confront being wrong all the time, which maybe teaches you that it doesn't feel as bad as you might expect, but it doe... (read more)

What's the current state of play on "good fairly quiet indoor air purifiers"? Like, is there a better model than "whatever Lighthaven has" or is that the best call, and if so, what is it called on Amazon?

I think part of the intended interpretation of Dath Ilan is: "it would be easier to run an adequate society if everyone knew some basic economics".

I know almost no economics. Even my understanding of supply & demand is shaky. Is there some bit of economics I can learn and start applying on Earth to see why this claim might be reasonable?

For my own future reference, here are some "benchmarks" (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical. (The "serious" version would probably start with the Evals section of technicalities' 2025 shallow review of technical AIS, or SemiAnalysis' in-house view.)

• the AI village and blog, not really a "benchmark" per se but my richest source of intuitions about current frontier models' capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other "benchmark-equ