Noah Birnbaum — LessWrong

LESSWRONG
LW

Replying toPreparing for a Warning Shot

Noah Birnbaum7d

Preparing for a Warning Shot

Yea. In light of this, someone should start a AIS replication org 👀👀

Preparing for a Warning Shot

Noah Birnbaum

Crossposted to the EA Forum and my Substack.

Confidence level: moderate uncertainty and not that concrete (yet). Exploratory, but I think this is plausibly important and underexplored.

TL;DR

Early AI safety arguments often assumed we wouldn’t get meaningful warning shots (a non-existential public display of misalignment) before catastrophic misalignment, meaning things would go from “seems fine” to “we lose” pretty quickly. Given what we now know about AI development (model weight changes, jagged capabilities, slow or fizzled takeoff), that assumption looks weaker than it used to.

Some people gesture at “warning shots,” but almost no one is working on what we should do in anticipation. That seems like a mistake. Preparing for warning shots—especially ambiguous ones—could... (read 1112 more words →)

Noah Birnbaum20d*Quick Take

Confidence level: strongly held, mostly opinionated, based on observation of (imo) bad LW norms.

We should stop using the phrase “epistemic status” and start using “confidence level.” In principle, “epistemic status” is meant to convey richer meta-information than confidence alone (i.e. the kind of evidence or how seriously a claim should be taken). In practice, it almost never does—on LW it’s usually just a clunkier way of saying “x confidence.”

If we actually want to convey more with less, we should just say “confidence level” and briefly qualify it with the relevant epistemic details (I.e. “low confidence, based on analogy,” or “high confidence, but mostly theoretical”). That’s clearer, less in-group-y, and lower friction. I think this is a good way to save up some weirdness points.

(Alternatively, one can used “qualified confidence” - a bit more jargony but traded for a bit more accuracy, though I perosonally like confidence level most).

-28

•••

Noah Birnbaum22dQuick Take

I was pretty unimpressed with Dario Amodei in the recent conversation with Demis Hassabis at the World Economic Forum about what comes after AGI.

I don’t know how much of this is a publicity thing, but it felt he wasn't really taking the original reasons for going into AI seriously (i.e. reducing x-risk). The overall message seemed to be “full speed ahead,” mostly justified by some kinda hand-wavy arguments about geopolitics, with the more doomy risks acknowledged only in a pretty hand-wavy way. Bummer.

Replying toOpinionated Takes on Meetups Organizing

Noah Birnbaum2mo

Opinionated Takes on Meetups Organizing

As someone who runs a (university) rationality group, I am pretty unsure about this point. While we started off being more accommodating to those who haven’t (starting with someone basically summarizing the reading), I feel like--for a reason unbeknownst to me--the standard changed and now everyone does the readings by default. Not accommodating people, then, seems like something that pushes people in the right direction.

Replying toEmergent Introspective Awareness in Large Language Models

Noah Birnbaum3mo

Emergent Introspective Awareness in Large Language Models

In case anyone wants it, Rob Long wrote an excellent summary and analysis of this paper here.

Replying toThe Memetics of AI Successionism

Noah Birnbaum3mo

The Memetics of AI Successionism

I appreciate the memetic-evolution framing, but I’m somewhat skeptical of the strong emphasis on tension-reduction as the primary (or even a major) explanatory driver of successionist beliefs. Given that you take successionism to be “false and dangerous,” it seems natural that your preferred explanation foregrounds memetics; but that sits a bit uneasily with the stated goal of analyzing why people hold these views irrespective of their truth value, which you state you're doing at the beginning.

Even if we bracket the object level, a purely memetic or cognitive-dissonance-based explanation risks drifting into an overly broad epistemic relativism/skepticism. Under many accounts of truth—process reliabilism being one—what makes a belief true is precisely that it’s... (read more)

Noah Birnbaum4moQuick Take

While I think LW’s epistemic culture is better than most, one thing that seems pretty bad is that occasionally mediocre/shitty posts get lots of upvotes simply because they’re written by [insert popular rationalist thinker].

Of course, if LW were truly meritocratic (which it should be), this shouldn’t matter — but in my experience, it descriptively does.

Without naming anyone (since that would be unproductive), I wanted to know if others notice this too? And aside from simply trying not to upvote something because it’s written by a popular author, anyone have good ideas for preventing this?

Noah Birnbaum5moQuick Take

“Albania has introduced its first artificial intelligence “minister”, who addressed parliament on Thursday in a debut speech.” lol, what???

Not sure how much this really matters vs is just a PR thing, but it’s maybe something people on here should know about.

Replying toMech Interp Wiki Page and Why You Should Edit Wikipedia

Noah Birnbaum6mo*

Mech Interp Wiki Page and Why You Should Edit Wikipedia

Thanks for the comment!

I hear the critique, but I’m not sure I’m as confident as you are that it’s a good one.

The first reason is that I’m unsure whether the trade-off between credibility for having a wiki page doesn’t outweigh the loss of control.

The second reason is that I don’t really think there is much losing control (minus in extreme cases like you mention) - you can’t be super ideological on wiki sites, minus saying things like “and here’s what critics say”. On that point, I think it’s just pretty important for the standard article on a topic to have critiques of it (as long as they are honest/ good rebuttals, which I’m somewhat confident that the wiki moderators can ensure). Another point on this is that LWers can just be on top of stuff to ensure that the information isn’t clearly outdated or confused.

Curious to hear pushback, though.

-4

Replying toMech Interp Wiki Page and Why You Should Edit Wikipedia

Noah Birnbaum6mo

Mech Interp Wiki Page and Why You Should Edit Wikipedia

Good call - that was from an earlier version of this post.

Mech Interp Wiki Page and Why You Should Edit Wikipedia

Noah Birnbaum

Noah Birnbaum, Jo Jiao

6mo

TL;DR:
A couple months ago, we (Jo and Noah) wrote the first Wikipedia article on Mechanistic Interpretability. It was oddly missing despite Mech Interp’s visibility in alignment circles. We think Wikipedia is a top-of-funnel resource for journalists, policy staffers, and curious students, so filling that gap is cheap field-building. Seeing that the gap existed in one of the most important subfields in AIS, we suspect that there are probably many others. Below, we (1) list other alignment / EA topics that need pages or can be upgraded and (2) share some notes about editing Wikipedia. If you know the literature, an afternoon of edits may be surprisingly high-impact.

PS: We also think that there... (read 265 more words →)

•••

How to Update If Pre-Training is Dead

Noah Birnbaum

7mo

Note: This piece will not spend much time arguing that pre-training is dead—others have done that elsewhere. Instead, the point here is to explore how people ought to update if they believe pre-training is dead. I’m also setting aside questions of degrees-of-deadness and how confident we should be.

Newton’s third law of motion says that for every action, there is an equal and opposite reaction. Something similar applies to Bayesianism: if some piece of evidence E updates you by x in one direction, then ~E should (symmetrically) update you x back in the other direction. This symmetry matters—especially when thinking about the apparent plateauing of progress from AI pre-training.

A lot of AI excitement over the past... (read 282 more words →)

•••

I saw this good talk on the Manifest youtube channel about using historical circumstances to calibrate predictions - this seems better for training than regular forecasting because you have faster feedback loop between the prediction and the resolution.

I wanted to know if anyone had recommendations on where to find some software or site where I can do more examples of this (I already know about the estimation game). I would do this myself, but it seems like it would be pretty difficult to do the research on the situation without learning the outcome. I would also appreciate people giving takes about why this might be a bad way to get better at forecasting.

A Talmudic Rationalist Cautionary Tale

Noah Birnbaum

10mo

The following is a translated famous passage from the Babylonian Talmud (Bava Metzia 59a–b) that I believe has a good implicit rationalist intepretation. Here is the source to read it yourself.

Everything in [] was added by the publisher of the translation and () were added by me for clarity/ funny.

It was taught there [in a Mishnah (Vessels 5:10)]: If he cut it [an earthenware oven] into rings and placed sand between each ring: Rabbi Eliezer (even though he is the epistemic hero, it is not that one) declares it tahor (pure), and the sages declare that it's tamei (impure). And this is the oven of Achnai.

Why [is it called the over of]... (read 409 more words →)

I think people should know that this exists (Sam Harris arguing for misaligned AI being an x-risk concern on Big Think YouTube channel):

Chicago – ACX Meetups Everywhere Spring 2025

Noah Birnbaum

11mo

This year's Spring ACX Meetup everywhere in Chicago.

Location: On the UChicago campus - usually in the Bartlett Dining Commons lounge but this could be subject to change. – https://plus.codes/86HJQCR2+QJ

Group Link: Uchicago: https://forms.gle/LZPSxwd1gtHku2QbA General Chicago: https://chicagorationality.com

If you are on the mailing list, please fill out RSVP form that will be sent out so that I know how much food to get.

Contact: dnbirnbaum@uchicago.edu

US AISI will be 'gutted,' Axios reports: https://t.co/blQY9fGL1v. This should have been expected, I think, but it still seems worth sharing,

This is a really good debate on AI doom -- I thought the optimistic side was a good model that I (and maybe others) should spend more time thinking about (mostly about the mechanistic explanation vs extrapolation of trends and induction vs empiricist framings), even though I think I disagreed with a lot of it on an object level:

https://marginalrevolution.com/marginalrevolution/2024/11/austrian-economics-and-ai-scaling.html

A good short post by Tyler Cowen on anti-AI Doomerism.

I recommend taking a minute to steelman the position before you decide to upvote or downvote this. Even if you disagree with the position object level, there is still value to knowing the models where you may be most mistaken.

New UChicago Rationality Group

Noah Birnbaum

Hey y'all!

I just started a rationality group on the UChicago campus and wanted to post it here to advertise it to UChicago-affiliated LessWrong readers. We've had a couple meetings so far which have been great, and I'm excited for more! A few more things:

(1) You can join the email list by submitting this form to be updated about meeting times, hear about events, and more!

(2) While we do not yet have university RSO (Registered Student Organization) status, I plan on applying for fall of next year. This would allow us to promote the club at the annual club fair, put up posters around campus (for events, fellowships, or other opportunities), and get... (read more)

Tyler Cowen often has really good takes (even some good stuff against AI as an x-risk!), but this was not one of them: https://marginalrevolution.com/marginalrevolution/2024/10/a-funny-feature-of-the-ai-doomster-argument.html

Title: A funny feature of the AI doomster argument

If you ask them whether they are short the market, many will say there is no way to short the apocalypse. But of course you can benefit from pending signs of deterioration in advance. At the very least, you can short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends.

Still, in a recent informal debate at the wonderful Roots of Progress conference in Berkeley, many... (read more)

Against AI As An Existential Risk

Noah Birnbaum

I wrote a post to my Substack attempting to compile all of the best arguments against AI as an existential threat.

Some arguments that I discuss include: international game theory dynamics, reference class problems, knightian uncertainty, superforecaster and domain expert disagreement, the issue with long-winded arguments, and more!

Please tell me why I'm wrong, and if you like the article, subscribe and share it with friends!

New Blog Post Against AI Doom

Noah Birnbaum

I'm curious as to what y'all think of the points made in this post against AI risk from 2 AI researchers at Princeton. If you have reason to think any points made are particularly good or bad, write it in the comments below!

Funny Anecdote of Eliezer From His Sister

Noah Birnbaum

This comes from a podcast called 18Forty, of which the main demographic of Orthodox Jews. Eliezer's sister (Hannah) came on and talked about her Sheva Brachos, which is essentially the marriage ceremony in Orthodox Judaism. People here have likely not seen it, and I thought it was quite funny, so here it is:

https://18forty.org/podcast/channah-cohen-the-crisis-of-experience/

David Bashevkin:

So I want to shift now and I want to talk about something that full disclosure, we recorded this once before and you had major hesitation for obvious reasons. It’s very sensitive what we’re going to talk about right now, but really for something much broader, not just because it’s a sensitive personal subject, but I think your hesitation... (read 460 more words →)

219