TurnTrout's shortform feed

TurnTrout

This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

When talking about "self-fulfilling misalignment", "hyperstition" is a fun name but not a good name which actually describes the concept to a new listener. (In this sense, the name has the same problem as "shard theory" --- cool but not descriptive unless you already know the idea.) As a matter of discourse health, I think people should use "self-fulfilling {misalignment, alignment, ...}" instead.

1koanchuk17d

Hyperstition ≈ self-fulfilling prophecy, with the caveat that the word is somewhat connected to other ideas associated with Nick Land (technological accelerationism, Capital as AI, AI that retrocausally brings itself into existence...).

6Kyle O’Brien18d

I'm of a like mind. I did not know what "hyperstition" meant until recently. While there is a chance I was uniquely uninformed, the fact that I had to consult LessWrong to familiarize myself with this term motivated my collaborators and I to intentionally avoid using it in our Alignment Pretraining paper. It sounds cool, but we thought it would make it more difficult to communicate our results.

9Jozdien18d

"Hyperstition" has the benefit of grouping "self-fulfilling {misalignment, alignment, ...}" together though. It's plausible I'm totally wrong here, but I think "self-fulfilling misalignment" being the commonly used term lends itself to people thinking of the "answer" being data filtering of alignment discourse, while I want people to think about things like upsampling positive data even more (which "self-fulfilling alignment" does get at, but I hear the term much less and I think people will end up just indexing on the name they hear most). I agree hyperstition isn't the best name for the reasons you describe to be clear, I just think "self-fulfilling {misalignment, ...}" also has a problem.

4TurnTrout18d

I think the problem of "may suggest a potentially suboptimal intervention" is less severe than "isn't descriptive." Plus, I think we're going to see "self-fulfilling alignment" be upsampled after the recent positive results. :)

2Jozdien18d

Yeah, that's fair. I guess my views on this are stronger because I think data filtering might be potentially negative rather than potentially sub-optimal.

Last week, I took the 10% giving pledge to donate at least 10% of my income to effective charities, for the rest of my life. I encourage you to think carefully and honestly about what you can do to improve this world. Maybe you should take the pledge yourself.

Thanks, I just took the pledge as well.

Based. Thank you for your altruism, Sheikh. :)

Apply for MATS mentorship at Team Shard before October 2nd. Alex Cloud (@cloud) and I run this MATS stream together. We help alignment researchers grow from seeds into majestic trees. We have fun, consistently make real alignment progress, and have a dedicated shitposting channel.

Our mentees have gone on to impactful jobs, including (but not limited to)

@lisathiergart (MATS 3.0) moved on to being a research lead at MIRI and now a senior director at the SL5 task force,
@cloud (MATS 6.0) went from mentee to co-mentor in one round and also secured a job at Anthropic, and
@Jacob G-W (MATS 6.0) also accepted an offer from Anthropic!

We likewise have a strong track record in research outputs, including

Pioneering steering vectors for use in LLMs (Steering GPT-2-XL by adding an activation vector),
Masking Gradients to Localize Computation in Neural Networks, and
Distillation Robustifies Unlearning.

Our team culture is often super tight-knit and fun. For example, in this last MATS round, we lifted together every Wednesday and Thursday.

Apply here before October 2nd. (Don't procrastinate, and remember the planning fallacy!)

7TurnTrout5mo

Testimonials

In a thread which claimed that Nate Soares radicalized a co-founder of e-acc, Nate deleted my comment – presumably to hide negative information and anecdotes about how he treats people. He also blocked me from commenting on his posts.

The information which Nate suppressed

The post concerned (among other topics) how to effectively communicate about AI safety, and positive anecdotes about Nate's recent approach. (Additionally, he mentions "I’m regularly told that I’m just an idealistic rationalist who’s enamored by the virtue of truth" -- a love which apparently does not extend to allowing people to read negative truths about his own behavior.)

Here are the parents of the comment which Nate deleted:

@jdp (top-level comment)
For what it's worth I know one of the founders of e/acc and they told me they were radicalized by a date they had with you where they felt you bullied them about this subject.
@Mo Putera (reply to jdp)
Full tweet for anyone curious:
i'm reminded today of a dinner conversation i had once w one of the top MIRI folks...
we talked AI safety and i felt he was playing status games in our conversation moreso than actually engaging w the substance of my questions- negging me

... (read more)

This post seems to me like very strong evidence that Nate was absolutely correct to block Alex.

For context, I have a deep and abiding fondness for both Alex and Nate, and have spent the last several years off to the side sort of aghast and dismayed at the deterioration in their relationship. I've felt helpless to bridge the gap, and have mostly ended up saying very little to either party about it.

But the above feels to me like a particularly grotesque combination of [petty] and [disingenuous], and it's unfortunately in-line with my sense that Alex has been something-like hounding Nate for a while. Actively nursing a grudge, taking every cheap opportunity to grind an axe, deliberately targeting "trash Nate's reputation via maximally uncharitable summaries and characterizations" rather than something like "cause people to accurately understand the history of our disagreement so they can form their own judgments," locating all of the grievance entirely within Nate and taking no responsibility for his own contributions to the dynamic/the results of his consensual interactions and choices, etc. etc. etc. I've genuinely been trying to cling to neutrality in this feud between ... (read more)

8TurnTrout1mo

"Hounding" and "every cheap opportunity"? From November 2023 (after the original thread wrapped up) to June 2025 (the date of OP), I made zero public comments about Nate's history of damaging behavior, nor have I contacted Nate. Duncan's characterization is not supported by the record.

9TurnTrout1mo

Duncan, your accusation of my being motivated by "romantic drama" is simply incorrect. Nate and I dated the same person for much of 2023 in an ethically non-monogamous fashion. Throughout that year, I had a few nice interactions with Nate and was actively working to become closer, though we never ended up close or anything. In late October, I stopped talking with that person for reasons that had nothing to do with Nate. A few weeks before that, I was concerned, but not really outraged. [1] However, in response to a discussion I started about social incentives, Kurt Brown (who I was close with) revealed Nate's abusive conduct while working at MIRI. These events --- cutting contact and the LessWrong thread --- were not related. I was upset by the behavior Kurt (and others) had recounted. That's a big part of why I got so mad after a year of nice interactions, and I even narrated as such in the Lightcone Slack at the time. I was standing up for myself by sharing my negative experience with Nate, but the deciding factor was standing up for others. You attempted to have it both ways by making a serious insinuation to undermine my credibility, without any real evidence. When I make a claim, I show receipts. I stand by every claim I made in the original thread exposing abusive behavior because they are factual and supported by my own experience or the experiences of people I interviewed. EDIT: The fact that this comment hit -9 agreement is wild. EDIT Jan 16, 2026: There's also the fact that I broke things off with the girl, which isn't very congruent with the implied narrative. His serious insinuation is wrong, and it's not close—it's not even close enough to true that rationalization could realistically cloud my answer. Duncan should strike his baseless claim and clear the air. 1. Basically, there were a bunch of factors which began to concern me over the course of 2023, having to do with my friends' and acquaintances' experiences with Nate. Also, I learned that N

Saying "ah, but when you talk to me in person I find it unpleasant, and so did these five other people" is, as Nate correctly characterized, barely topical. "Underdeployed Strategy X has powerful upsides; here's evidence of those upsides in a concrete case" is not meaningfully undercut by "your particular version of a thing that might not even be intended as a central example of Strategy X has sometimes had negative side effects."

I think I've already explained why this misses the point:

It is indeed more related than a randomly selected post would be. The relevant part is not because of the subject matter (as you solely talk about during this comment thread), but because of the nature of the evidence it uses to make its case.
The vast majority^[1] of the original post, along with subsequent comments, uses Nate Soares's personal experience meeting with cong

LESSWRONG
LW

LESSWRONG
LW

TurnTrout's shortform feed

28

Ω 10

The information which Nate suppressed