Quick Takes — LessWrong

the no magic ood principle

a lot of alignment proposals have a step where you have some kind of magic ood generalization. depending on the shape of the proposal this could be obvious or subtle. i think a desirable property for alignment proposals is to avoid having a magic ood step, or to make a strong case why the amount of magic is smaller than competing proposals, or to empirically test the magic ood and understand it deeply.

Showing 3 of 7 replies (Click to show all)

DaemonicSigil1h20

Thanks for the reply. I don't have much to say about most of it because I already agree that capabilities are convergent while goals are not. The example with logarithm problems is very nice.

Just from the need to model the world well, LLMs probably already have a decent sense of what human values are. (Maybe they are represented on a manifold, with different value systems a person might have being represented by different points on the manifold.) A learner that needed billions of tokens to figure out this concept is probably not very sample efficient. I.e.... (read more)

6Anthony DiGiovanni8h

(Fwiw my impression is that a lot of capabilities forecasts also have a step where you have some kind of magic OOD generalization. Not saying that's an excuse, just registering my confusion about how commonly invoked this step is.)

2Dean Valentine8h

I agree that generalization should be measured. If you dismiss strategies with an OOD step ex-ante, though, you often end up with a world-model where neural nets don't work at all. Presumably dumb shit like "training on enough good examples" would work in the limit of samples and generality, because it worked for next-token prediction.

Q Home's Shortform

Q Home8h*2-3

In mesa-optimizers, "mesa-" is not actually a real prefix meaning the opposite of "meta-", illustrating the deceptiveness of those things.

You suspected something was off, but the prefix promised way too many capabilities and conveniences for you to question it.

edit: oops, see this

2CstineSublime3h

That is a surprising reading to me. I always assumed it was named for the very Roadrunner & Coyote looking hills with steep escarpments. It seems like it's real meaning is closer to the Greek adverb for things which are "in" (i.e. Mesangium, Mesopotamia, Mesoamerica). Thus an optimizer within the range of the base model.

Q Home1h10

Oops! Sorry, I ended up rediscovering America.

Mesa- is a Greek prefix which means the opposite of meta-. To “go meta” is to go one level up; to “go mesa” is to go one level down (nobody has ever actually used this expression, sorry). So a mesa-optimizer is an optimizer one level down from you. (c.) Scott Alexander

But I first encountered this reading in a Rob Miles' video. However, Rob's slides actually show that it's not valid Greek and the prefix literal meaning is different. (Though there's also some non-AI paper which conceptualizes meta/mesa as opposit... (read more)

leogao's Shortform

leogao8h124

why does the arc white box challenge use random networks? it seems likely that real trained networks have structure that makes it possible to estimate that random networks don’t

Showing 3 of 8 replies (Click to show all)

Dmitry Vaintrob2h40

The random part is fundamental to the result working, and makes things easier and not harder (i.e. structured nets are outliers from the point of view of this result). Assuming structure makes things hard. Saying something nontrivial for too general settings tends to run into cryptography results around hashes - though I think ARC is interested in extensions with controlled structure. In particular I think they know that if the structure comes from Bayesian-learning training on a small set of datapoints then a version of their result extends to that setting

2Dean Valentine3h

Also possible that they have structure that random networks are unlikely to have by default.

2williawa3h

I don't understand this either. My intuition mostly comes from https://www.alignmentforum.org/posts/ZvQfcLbcNHYqmvWyo/the-paper-that-killed-deep-learning-theory https://www.lesswrong.com/posts/zcGmdQHX66NhC69v6/the-other-paper-that-killed-deep-learning-theory so its not very deep, but I do get this vibe that, general explanations of neural networks is not possible, and that it requires analysis both of the individual mechanistic structure of the network (arc challenge check), and the specific properties "natural" data tends to have (not check).

CstineSublime's Shortform

CstineSublime3h10

Duolingo Alternatives?

How have you become conversant in a second language?

Duolingo is frustrating because it forces rote learning. This may not be the case for more popular languages, but I’m learning Greek. There is no grammatical instruction whatsoever. When I get something wrong, for example, selecting the third person verb rather than the second person, there is no feedback. I will only learn it by looking at the Wikipedia article on the language’s grammar.

The voice recognition is laughable too, I will accidentally mispronounce the “a” sound as a “e”, ... (read more)

Spective2h10

For me the most helpful one is just thinking in the target language as much as possible. YMMV depending on how much of a verbal thinker you are to begin with.

less_raichu's Shortform

less_raichu1d424

Sam Altman and Dario Amodei are the public faces of AI safety and that's poisoning the well

Normies just smell a rat whenever those two bring up AI safety. I think when the AI safety community gets a word in edge-wise (like you're arguing with your friends or maybe a tweet goes viral), we usually say something like

"it could kill everyone. but China could also kill everyone, so now more than ever we have to supercharge OpenAI and Anthropic."

and we just sound like we're "one of them."

I feel like I should be hearing "shut it down" about as often as I hear "bro... (read more)

Showing 3 of 7 replies (Click to show all)

1sanyer10h

You're painting a very binary, "us vs. them" picture of the option space here. There are many policy positions one might take between the extremes "datacenter moratorium" and "cheering the leading labs"; rejecting the idea of a moratorium does not mean you want the labs to be supercharged. For some examples of such policies, see e.g. the Radical Optionality suggestions.

2the gears to ascension3h

Consensus processes have to reach agreement among a lot of people, some of whom are in different belief states. It's not surprising to me that we'd see GPT2 level reasoning like "if you don't vocally support a datacenter ban, you don't believe in xrisk", which I read as being because to the weak model a crowd behaves as, datacenter ban is a lesser version of banning it all, and therefore if you support banning it all, you support a datacenter ban. The shallow model the averaged crowd behaves as, in the part of the individual's brains running the Keynesian beauty contest, sees a low dimensional subspace where there's a success policy vertex at the extrema "ban it all" and the consensus process is to push towards that with lesser policies incrementally. So then someone says, for example, "banning new datacenters is irrelevant because we need to shut down all computers capable of running a takeoff at all", and it sounds like not wanting to hike through the intermediate policies to get the one you want. I make no claim any of this is the slightest bit reasonable. I am reading off my intuitive model of crowds.

less_raichu2h10

I think the simple belief "it might kill everyone => shut it down" is a pretty good belief, and people rightly resist attempts to move them off it, and they are rightly skeptical of people who believe the first but not the second.

The crowd effect has more to do with the outgroup and less to do with matching what one's ingroup thinks. If a stranger say "x-risk => build data centers", they are not showing ingroup harmony, so they get evaluated for outgroup, and the answer is something like "I can easily see Sam and Dario lying about believing in x-risk as part of building hype, this stranger is a tech bro." I wish those two were not the most visible examples of people who believe in x-risk.

Daniel Kokotajlo's Shortform

Daniel Kokotajlo2d5910

https://www.astralcodexten.com/p/your-book-review-the-escape-artist

Reading about the unwillingness of Jews and Allies to believe that the holocaust was real, while it was happening, and even in some cases while being herded into the camps, was interesting and depressing.

Showing 3 of 7 replies (Click to show all)

roha3h30

From the opening:

There’s no redemption here, no moral uplift, no lessons save for perhaps the grimmest and most nihilistic “lesson” I’ve ever encountered in any story, Holocaust-related or otherwise: that when confronted with the unthinkable, most people’s natural tendency is denial.

From the conclusion:

From our modern vantage point, it’s easy to find this behavior inexplicable. But I think there are a few mitigating factors. One is that what these people were being warned of was unimaginable. That word—unimaginable—is itself another Holocaust cliché. But I

... (read more)

8Harjas22h

The review mentions it as an afterthought. In its opening section, the author writes: [...] And when discussing the actual lives saved: [...] 200,000 lives is not a small victory to me.

7Mo Putera21h

I agree it's not a small victory, e.g. it's in the ballpark of GiveWell's lifetime impact.

Wei Dai's Shortform

Wei Dai4mo*8411

A historical note re: MIRI vs EA

When I was arguing against MIRI trying to build Friendly AI (now "aligned ASI") circa 2012, I think 3 people at MIRI were thinking the most about strategy and occasionally responded to me: @lukeprog @CarlShulman @Eliezer Yudkowsky. Only one of them actually changed their mind about this (namely Eliezer), while the other two left MIRI and don't support AI pause to this day, AFAIK. (Luke now "leads our grantmaking on AI governance and policy" at OpenPhil/Coefficient. Carl became an advisor to OpenPhil and is now Research Direc... (read more)

Showing 3 of 41 replies (Click to show all)

5Lukas Finnveden6h

This sounds like it's making a bunch of assumptions that seem wrong to me. My guess is that Aschenbrenner wanted to found that hedge fund and that this wasn't rooted in any sort of EA consensus that it was a good use of his time. My guess is there wasn't, at the time, any sort of general opinion that founding that hedge fund was a particularly more important or central position than doing AI governance grantmaking at OpenPhil. (As Mueuhlhauser was picked to do.) Some EAs did choose to invest in the hedge fund (though my guess would've been it's a minority of the fund's money, and wasn't necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it's up like 80% this year? and great returns before 2026), so doesn't seem like we need to invoke any special kind of irrationality to explain that decision.

habryka5h*70

Some EAs did choose to invest in the hedge fund (though my guess would've been it's a minority of the fund's money, and wasn't necessary to get it off the ground). It still looks like the fund has made its investors lots of money (even post-crash it's up like 80% this year? and great returns before 2026), so doesn't seem like we need to invoke any special kind of irrationality to explain that decision

To be clear, public accounting suggests the hedge fund lost ~all money that wasn't invested in Anthropic. The only reason the fund is up is because they got i... (read more)

5Eliezer Yudkowsky7h

I rather expected from that time that Aschenbrenner would be trouble for EA, if they just couldn't stop themselves from promoting Harvard blondes; but I did not say anything against him then on the basis of that conversation, and only marked him down as actively harmful once I saw him starting to oppose all good ideas during his brief tenure at FTX philanthropy. He did of course further go on to actively stoke arms races with China. The relevance to the Wei Dai part is that Wei Dai seemed to me to be asking, "Aren't you deciding post facto who is 'EA' and who is 'rationalist'?" and my reply to Wei Dai is implicitly, "My memory claims that it is really really not hard to tell, in advance, early on."

Cat McGee's Shortform

Cat McGee6h10

I want to see work on verification of claims that AI labs make about their models. We put far too much trust into their research that is uncheckable by the public. Can existing cryptography help? ZK proofs of an LLM's internals?

Richard_Kennaway's Shortform

Richard_Kennaway20h*90

To develop armour to resist all missiles, we must develop missiles to break through all armour.

To develop vaccines against all diseases, we must develop diseases that defeat all vaccines.

To develop boxes to keep AI in, we must develop AI to break out of all boxes.

anithite7h10

Many problems can be solved without needing the opposing field developed. Software/hardware sandboxes especially. A theorem proven software/hardware sandbox developed with hardware that is sufficiently robust to errors that they don't pose a problem is going to just inescapable. Once you open a causal channel out all bets are off, but the hardware and software to box something up is not impossible to build without the opposing offense.

Other areas are harder and mostly come down to impossibilities. It's impossible to build physical protections that block al... (read more)

Celene Nightingale's Shortform

Celene Nightingale18h5930

Given the recent news events (like Pax Silica pushing for exclusivity and barring countries from joining WAICO), I've grown increasingly nervous about previous AI comms strategies. Everybody knows that technical ai safety is a very fraught field and a lot of empirical AI safety research carries great uncertainty over whether you are helping or harming, but I suspect the same is true of AI Governance. Looking at previous successful statements, AI 2027 was highly impactful but it also strongly framed AI as a national security issue with a race against China ... (read more)

4_rpd13h

it is pretty wild that we've managed to politically polarize AI safety. but I guess if you can polarize climate change, you can polarize anything.

14Seth Herd11h

We have not yet polarized AI safety. Politicians are polarized but their base is not. US liberal and conservative voters hate AI and data centers in roughly equal amounts. We must not allow the issue to become more polarized. It slowed climate change legislation to a crawl and we don't have time for that. For this reason I think we should probably quit advocating for AI safety to the left and ramp up efforts to reach politicians on the right

_rpd7h102

US liberal and conservative voters hate AI and data centers in roughly equal amounts

Dems seem to have a 10 to 20 percentage point lead in anti-datacenter sentiment, depending on how it is measured ...

How Americans view data centers’ impact in key areas, Pew Research Center, March 12, 2026
https://www.pewresearch.org/short-reads/2026/03/12/how-americans-view-data-centers-impact-in-key-areas-from-the-environment-to-jobs/

Americans Oppose AI Data Centers in Their Area, Gallup, May 13, 2026
https://news.gallup.com/poll/709772/americans-oppose-data-centers-area.as... (read more)

Richard Ngo's Shortform

Richard_Ngo1y*51-31

Crossposted from Twitter:

This year I’ve been thinking a lot about how the western world got so dysfunctional. Here’s my rough, best-guess story:

1. WW2 gave rise to a strong taboo against ethnonationalism. While perhaps at first this taboo was valuable, over time it also contaminated discussions of race differences, nationalism, and even IQ itself, to the point where even truths that seemed totally obvious to WW2-era people also became taboo. There’s no mechanism for subsequent generations to create common knowledge that certain facts are true but usefully ... (read more)

Showing 3 of 53 replies (Click to show all)

Eli Tyre8h51

That is, while it was bad for the people who didn't get rule of law, they were a separate enough category that this mostly didn't "leak into" undermining the legal mechanisms that helped their societies become productive and functional in the first place

Coming back to this a year later...this seems pretty implausible to me, especially when the people in a separate category are physically proximal, for largely for social-psychological reasons.

It seems hard to maintain a trust in the justice of the court system, if a whole class of people is colluding agains... (read more)

0lilkim20256mo

This feels like you started from a position that you wanted to believe and worked backwards, looking for reasons to believe it, rather than the other way around. The overwhelming majority of successful civilizations throughout history have been nation-states, at least to some degree, which should be addressed in some capacity by any argument claiming that it's a suboptimal way to organize a society. It is true that large empires are often not monoethnic, but this seems like a structural challenge of building a large empire[1] rather than a question of how each system scales. Indeed, large empires tend to be authoritarian rather than democratic, and few people use this as an argument for why authoritarianism is intrinsically better than democracy. 1. ^ Which may not be intrinsically desirable - the U.S., for example, built its hegemonic status on a web of economic and military relationships rather than on conquest, and I don't consider it self-evident that this is a worse strategy. China is frequently said to be working towards the same ends as a (relatively) monoethnic state.

3TAG6mo

The vast majority of unsuccessful ones have been ethnostates, as well, because the vast majority are ethnostates ... on average, ethnostates are average. [...] I spoke of greatness, not goodness.

DanielW's Shortform

DanielW2d190

It is brought up quite often, but I have not (yet!) heard a satisfactory answer to the question: "why are so many leading people in advocating AI risk also investing in developing LLMs and AI-dependent ecosystem?"

It seems to me, the position (in simple form) holds:

Divesting from a technology discourages it's development/advancement n aggregate (investing in a technology encourages it)
Therefore, if the development/advancement of a technology is, in aggregate, undesirable, one should be divest from it and minimize any investment (not withstanding benefits t

1DanielW10h

Yes, and I am saying you can (under the described beliefs) use carbon emissions or promoting AI as heuristics for the harms outweighing the benefits. If the only information I know about an action is that it increases carbon emissions or it promotes AI then if I believe those things are--in aggregate--bad then my expectation for any given action should be that it is bad. If I know the net expected utility of set is negative (i.e. where ). Without additional information about expected utility of any given element , I can say . We never have perfect knowledge of utility expectations. [...] The actual utility calculation is unclear and depends on one's personal values which I wasn't working on. But going by what you gave me for gwern's expected utility, I should be able to say: * * *

2Nisan10h

I disagree with that. So, that's my utilitarian counterargument. What do you think of the (generally reasonable) deontological language I highlighted in the quotes above? And do you want to choose a concrete example of an ordinary, everyday act of compassion you wholeheartedly endorse?

DanielW9h10

I disagree with that

I disagree with it as well, but it is the estimated utility in the post from gwern you directed me to.

And do you want to choose a concrete example of an ordinary, everyday act of compassion you wholeheartedly endorse?

Hugging a loved one, holding the door for a stranger, giving a dollar to charity, etc.

Ryan Kidd's Shortform

Ryan Kidd1d7-1

Interesting perspective I heard recently: board members should all be "too senior" for C-suite roles. This could be because they are CEOs themselves, or are otherwise too qualified to work at the company. If they aren't overqualified, why are they board members? Just hire them!

I think this is a little oversimplified, particularly for nonprofits, as board members are also selected for mission alignment and trustworthiness. You also don't want your board to be all "prestige picks", with no relevant experience.

However, I think there's something important indi... (read more)

Ryan Kidd10h20

Note that I think this frame is significantly less relevant to the board composition of frontier AI companies than to most other companies. Ideally, the boards of frontier AI companies possess key expertise in noticing and averting impending AI catastrophes.

Kabir Kumar's Shortform

Kabir Kumar11h10

i've been coming around to lately, the idea that just strong moral fiber, with deep deep care about truth and caring for people, being willing to sacrifice things like 'winning', 'ego, 'attached aesthetics', etc for the sake of truth and caring about/helping people, is what matters more than anything else, when combined with some decent iq (like, 120+ is prob pretty nice, 115+ can work).

And every attempt I've seen so far at things like "yeah, they're not completely honest and willing to update all the time, but they're useful because...", "yeah, they aren't very considerate, but they're useful because...", etc, just goes wrong eventually.

Kabir Kumar11h22

And also, examining where I'm lacking in this, - when a less willing to update, where I'm less considerate - to others and myself - has had very large feeling changes to myself that I endorse.

cousin_it's Shortform

cousin_it12h*Ω53-8

A few days ago I tried AI again, after several months of not using it at all. We're dealing with some housing-related matters, and I asked Claude, ChatGPT and Gemini for advice what to do in a certain situation. (I won't give the details here, because they're personal and not that important for my point.) All three models independently gave the same advice, which sounded really convincing to me. But we have some friends with experience in such matters, who said that following this advice would be catastrophic and we should do the opposite. Eventually we fo... (read more)

StanislavKrym11h20

How could one test this empirically and/or find the causes?

Suppose that you have noticed a joke about a connection between two niche interests. How similar is asking the LLMs to evaluate the joke and doublechecking the explanation for errors to the phenomenon which you describe?
How similar is this to the phenomenon where the LLMs did things like miscounting letters in words or asking the user to walk to the car wash?
There also is the issue that the LLMs could absorb biases from the broader culture. I have encountered conservatives making cases for SOTA cu

... (read more)

_rpd's Shortform

_rpd13h30

NVIDIA releases quantized Kimi K3 optimized for running on B300 hardware ...
https://huggingface.co/nvidia/Kimi-K3-NVFP4

Richard Ngo's Shortform

Richard_Ngo14d*2233

A few weeks ago, I was offered a role as a Staff Research Scientist at Resolution (formerly Sequent) to do agent foundations research. However, they later rescinded the offer due to concerns that having someone with my (publicly expressed) political opinions as an early hire might make it harder for them to hire other researchers. I want to give some brief thoughts on this, starting with the concrete details of what happened.

I talked to Jesse Hoogland, Dan Murfet, and Geoffrey Irving before they gave me the offer. These were fairly informal chats, since th... (read more)

Showing 3 of 108 replies (Click to show all)

Elias Schmied13h10

how much people in the past cared about "honor" (e.g. to the point of duelling to the death over it)

Honor cultures are not generally seen as enlightened liberal Western stuff, so I'm a bit confused by this connection. Would you call gang members shooting each other in the streets over small slights today an example of this good thing you're pointing at? (EDIT: Oh, I guess you mean specifically that they're more honest)

1Elias Schmied13h

Cool, makes sense! [...] Yeah, I had something like this intuition as well but wanted to keep it simple in my original comment. One might more precisely say that there's some intrinsic adversariality in any ideological conflict, but the lever is more about not being reactive about it, processing it before putting it back into the world, making it conscious/overt so our system-2 can deal with it, etc. And that this generally leads to less intense polarization (since it will be more about high-level beliefs and less (though still nonzero) about deep gut-level emotions). [...] Yeah that's true, agreed now that you say it - I didn't think of that option some reason. [...] That's interesting, though I'm not sure I understand it well. What gets lost if I round this off to "more explore cognitive patterns, and less exploit ones"?

5Richard_Ngo1d

Both your questions and mine felt very pointed. I think questions can be asked from a stance of genuine curiosity, versus a stance of "justify yourself!", and the difference is very noticeable and also changes the dynamics of the resulting conversation significantly. If I'd had that curious stance, for example, I would have scrolled the DoD feed and then agreed that it was distasteful, saving myself from what has turned out to be a fairly unproductive and time-consuming thread.

Mo Putera's Shortform

Mo Putera14h112

On Dec 13th 2017, a still-anonymous redditor announced they would donate 5,057 BTC, the majority of their wealth, to charitable causes via the Pineapple Fund.

I remember staring at bitcoin a few years ago. When bitcoin broke single digits for the first time, I thought that was a triumphant moment for bitcoin. I watched and admired the price jump to $15.. $20.. $30.. wow!
Today, I see $17,539 per BTC. I still don't believe reality sometimes. Bitcoin has changed my life, and I have far more money than I can ever spend. My aims, goals, and motivations in life h

... (read more)

EniScien's Shortform

EniScien14h10

I already for a long time thought that examples like "super in superintelligence isn't about how Einstein is superior to a peasant, it is how human is superior to mouse" isn't quite good example. As well as ant example. Because we differ from mice and ants not only in intelligence, but just in size. Better example would be something of similar to human size. But chimps are too close to humans. Jaguars as I know are similar to humans in size. But most people probably never saw the behaviour of jaguars. So maybe better example may be, say, a cow. Cows are ph... (read more)

Mo Putera's Shortform

Mo Putera16h60

Out of curiosity I asked Sol (max) what Screwtape's Mythos-powered Dr Doom might be able to do:

Imagine the date is August 15th, 2016. You, and only you, have access to Mythos. Everyone else in the world has only the machine learning tools of 2016. That's a crazy advantage, right? That is the kind of supertech that comic books are made of. You'd look like Dr Doom.

Sol's overall take:

The most impressive capability-only achievement would be building, several years early, a tiny institute that continuously turns public code and public data into validated secur

... (read more)