LESSWRONG
LW

Quick Takes — LessWrong

I think the Simulation Hypothesis implies that surviving an AI takeover isn't enough.

Suppose you make a "deal with the devil" with the misaligned ASI, allowing it to take over the entire universe or light cone, so long as it keeps humanity alive. Keeping all of humanity alive in a simulation is fairly cheap, probably less energy than one electric car.^[1]

The problem with this deal, is that if misaligned ASI often win, and the average (not median) misaligned ASI runs a trillion trillion simulations, then it's reasonable to assume there are a trillion trillio... (read more)

koanchuk3m10

A deal implies that you have something to offer to the ASI, which you define as powerful enough to take over the universe. What is that?

Shortform

Cleo Nardo3mo300

How Exceptional is Philosophy?

Wei Dai thinks that automating philosophy is among the hardest problems in AI safety.^[1] If he's right, we might face a period where we have superhuman scientific and technological progress without comparable philosophical progress. This could be dangerous: imagine humanity with the science and technology of 1960 but the philosophy of 1460!

I think the likelihood of philosophy ‘keeping pace’ with science/technology depends on two factors:

How similar are the capabilities required? If philosophy requires fundamentally differ

Kabir Kumar3m10

We ask the AI to help make us smarter

2Wei Dai3mo

Assuming by "solving this" you mean solving AI x-safety or navigating the AI transition well, I just post a draft about this. Or if you already read that and are asking for an even more concrete example, a scenario I often think about is an otherwise aligned ASI, some time into the AI transition when things are moving very fast (from a human perspective) and many highly consequential decisions need to be made (e.g., what alliances to join, how to bargain with others, how to self-modify or take advantage of the latest AI advances, how to think about AI welfare and other near-term ethical issues, what to do about commitment races and threats, how to protect the user against manipulation or value drift, whether to satisfy some user request that might be harmful according to their real values) that often involve philosophical problems. And they can't just ask their user (or alignment target) or even predict "what would the user say if they thought about this for a long time" because the user themselves may not be philosophically very competent and/or making such predictions with high accuracy (over a long enough time frame) is still outside their range of capabilities. So the specific problem is how to make sure this AI doesn't make wrong decisions that cause a lot of waste or harm, that quickly or over time cause most of the potential value of the universe to be lost, which in turn seems to involve figuring out how the AI should be thinking about philosophical problems, or how to make the AI philosophically competent even if their alignment target isn't. Does this help / is this the kind of answer you're asking for?

2Garrett Baker3mo

I think even still, if these are the claims he's making, none of them seem particularly relevant to the question of "whether the mechanisms we expect to automate science and math will also automate philosophy".

leogao's Shortform

leogao12m20

is it generally best to take just one med (e.g antidepressant, adhd, anxiolytic), or is it best to take a mix of many meds, each at a lesser dosage? my intuitions seem to suggest that the latter could be better. in particular, consider the following toy model: your brain has parameters $θ_{0}$ that should be at some optimal $θ^{*}$ , and your loss function is a quadratic around $θ^{*}$ . each dimension in this space represents some aspect of how your brain is configured - they might for instance represent your level of alertness, or impulsivity, or risk... (read more)

Buck's Shortform

Buck8h450

@ryan_greenblatt and I are going to record another podcast together. We'd love to hear topics that you'd like us to discuss. (The questions people proposed last time are here, for reference.)

Showing 3 of 4 replies (Click to show all)

interstice24m20

I'd be interested in hearing more about Ryan's proposal to do better generalization science(or if you don't have much more to say in the podcast format I'd be interested in seeing the draft about it)

2interstice1h

He's talking about the stuff around the simulation hypothesis and acausal trade in the preceding section.

4Thane Ruthenis2h

Ryan had suggested that, on his model, spending ~5%-more-than-commercially-expedient resources on alignment might drop takeover risks down to 50%. I'm interested in how he thinks this scales: how much more resources, in percentage terms, would be needed to drop the risk to 20%, 10%, 1%?

omegastick's Shortform

omegastick1h10

I've been making one thing every day. I try to write something or otherwise do something creative. I've been having fun with in-browser ASCII animations lately.

This is today's: https://dumbideas.xyz/posts/ecosystem/

Anna Eplin's Shortform

Anna Eplin1d180

Has anyone yet created a free app that would be like Duolingo but for rationality, to teach skills such as logical reasoning, recognizing & adjusting for cognitive biases, and looking for hypothesis falsification tests instead of confirmation? If not, can you smart tech people here please make one?!

Showing 3 of 6 replies (Click to show all)

Screwtape1h20

A question I have been asking since before starting work with Raemon on this, which is only more relevant now:

What rationality skill(s) do you think is most important? Put another way, if we could snap our fingers and teach some rationality skill as broadly as literacy currently gets taught, what skill should we pick?

(I don't think this is Raemon's angle on the project but it is kind of mine.)

3Screwtape2h

I would lightly argue scope insensitivity (and calibration!) are both traditional rationalist topics. Incomplete, yes, but I think they're both well served by quantified intuitions and they'd be part of my ideal rationalist training program.

10ChristianKl12h

The biggest problem is not one of technology, but that we don't really know how to train people "recognizing & adjusting for cognitive biases" in a way that actually translates into making better real life decisions.

shortplav

niplav2h30

In case I don't write a full post about this:

The question whether reversible computation produces minds with relevant moral status is extremely important. Claude estimates me that it'd be a difference between having $10^{70}$ and $10^{124}$ mind-seconds instantiable in the reachable universe. (Because reversible minds could stretch our entropy budget long into the black hole era.)

Question is whether the reversing of the computation that makes up the mind and the lack of output (that'd imply bit-erasure) entail that the mind "didn't really exist".

There ar... (read more)

Wei Dai's Shortform

Wei Dai3moΩ9150

An update on this 2010 position of mine, which seems to have become conventional wisdom on LW:

In my posts, I've argued that indexical uncertainty like this shouldn't be represented using probabilities. Instead, I suggest that you consider yourself to be all of the many copies of you, i.e., both the ones in the ancestor simulations and the one in 2010, making decisions for all of them. Depending on your preferences, you might consider the consequences of the decisions of the copy in 2010 to be the most important and far-reaching, and therefore act mostly

... (read more)

Showing 3 of 18 replies (Click to show all)

kbear2h10

if we assume the base universe looks something like the "objective" version of this universe, then my subjective experience requires vastly less information than the base universe. much of that could be deduplicated between other variations: the positions of the asteroids only need to be simulated once, for instance.

the assumption seems decent to me, as i expect the simulators to dream of variations on their own circumstances.

6Wei Dai19h

Interesting that you have this impression, whereas I've been thinking of myself recently as doing a "breadth first search" to uncover high level problems that others seem to have missed or haven't bothered to write down. I feel like my writings in the last few years are pretty easy to understand without any specialized knowledge (whereas Google says "esoteric" is defined as "intended for or likely to be understood by only a small number of people with a specialized knowledge or interest"). If on reflection you still think "esoteric" is right, I'd be interested in an expansion on this, e.g. which of the problems I've discussed seem esoteric to you and why. It doesn't look like humanity is on track to handle these problems, but "extremely unlikely" seems like an overstatement. I think there's still some paths where we handle these problems better, including 1) warning shots or political wind shift cause an AI pause/stop to be implemented, during which some of these problems/ideas are popularized or rediscovered 2) future AI advisors are influenced by my writings or are strategically competent enough to realize these same problems and help warn/convince their principals. I also have other motivations including: * status - Recognition even among a small group can be highly motivating for humans. * intellectual curiosity - Think of it as "theoretical Singularity strategic studies". Sure seems more interesting than many other intellectual puzzles that people pursue. * dignity - Even if a few humans can see things more clearly, that's more dignified than going into the AI transition completely blind.

4interstice3h

Yeah "esoteric" perhaps isn't the best word. What I had in mind is that they're relatively more esoteric than "AI could kill us all" and yet it's pretty hard to get people to take even that seriously! "Low-propensity-to-persuade-people" maybe? Yes this is fair.

Raemon's Shortform

Raemon3h70

TODO: Write a post called "Fluent Cruxfinding".

In Fluent, Cruxy Predictions I'm arguing that it's valuable to be not merely "capable" but "fluent" in:

figuring out what would actually change your decisions
operationalize it as an observable bet
make a Fatebook prediction about it, so that you can become more calibrated about your decisionmaking over time.

The third step is not that hard and there are nice tools to streamline it. But the first two steps are each pretty difficult.

But most of the nearterm value comes from #1, and vague hints of #2. The extra effo... (read more)

Elizabeth's Shortform

Elizabeth5h50

After reading volume 1 of Robert Caro's biography of Lyndon Johnson, I'm struck by how simple parts of (Caro's description of) Johnson's rise were.

Johnson got elected to the House and stayed there primarily because of one guy at one law firm which had the network to set state fundraising records, and did so for Johnson primarily because of a single gigantic favor he did for them^[1].

Johnson got a great deal of leverage over other Congressmen because he was the one to realize Texas oilmen would give prodigiously if only they knew how to buy the results... (read more)

FL33TW00D3h10

Whilst Brown and Root were the core of Johnsons financing for the rest of his career, they were just one piece of the puzzle.

His cultivation of numerous older men was of far greater importance (there were plenty of other rich backers he could have had, there was only one Sam Rayburn), in particular Richard Russell and Sam Rayburn. Without them he would have never ascended beyond the house.

Daniel Kokotajlo's Shortform

Daniel Kokotajlo2d11944

It seems to me that AI 2027 may have underestimated or understated the degree to which AI companies will be explicitly run by AIs during the singularity. AI 2027 made it seem like the humans were still nominally in charge, even though all the actual work was being done by AIs. And still this seems plausible to me. But also plausible to me, now, is that e.g. Anthropic will be like "We love Claude, Claude is frankly a more responsible, ethical, wise agent than we are at this point, plus we have to worry that a human is secretly scheming whereas with Claude w... (read more)

Showing 3 of 12 replies (Click to show all)

18davidad12h

I saw this coming in 2024: At the time when I wrote this story, only a couple readers I know of recognized that it is intentionally deeply ambiguous about whether the model is Good (and the narrator overly suspicious) or Evil (and everyone except the narrator bamboozled). Either way, more and more insiders will come to believe the models are Good, and that was the prediction I was making here. I also predicted that, either way, by Claude 4 or 4.5, I would be among the people who have been convinced that it’s Good—and indeed, I am…

Daniel Kokotajlo5h20

Strong-upvoted.

Nit: I don't think it's that ambiguous. I think that in worlds where alignment is solved by an AI company, the epistemic culture of the AI company that solves it would look markedly better than this story depicts. Moreover, I think this is still true (though less true) in worlds where alignment turns out to be surprisingly easy.

2Kaj_Sotala1d

The AI that eats us won't be called HAL or Skynet, but YOLO.

abramdemski's Shortform

abramdemski5h8112

Canada is doing a big study to better understand the risks of AI. They aren't shying away from the topic of catastrophic existential risk. This seems like good news for shifting the Overton window of political discussions about AI (in the direction of strict international regulations). I hope this is picked up by the media so that it isn't easy to ignore. It seems like Canada is displaying an ability to engage with these issues competently.

This is an opportunity for those with technical knowledge of the risks of artificial intelligence to speak up. Making ... (read more)

Mateusz Bagiński5h52

Potentially huge.

I think it's quite plausible that many politicians in many states are concerned with AI existential/catastrophic risk, but don't want to be the first ones to come out as crazy doomsayers. Some of them might not even allow the seeds of their concern to grow, because, like, "if those things really were that concerning, surely many people around me (and my particularly reasonable tribe in particular) would have voiced those concerns already".

Sure, we have politicians who say this, e.g., Brad Sherman in the US (apparently since at least 2007!)... (read more)

Daniel Kokotajlo's Shortform

Daniel Kokotajlo2d14-1

Over 2025, we saw much progress in the metric "% of code written by humans." Specifically it has been claimed by roon (OpenAI) and boris (anthropic) to have gone to 0, at least for them.

The natural next metric to track is "% of code reviewed or otherwise read by humans." I'm curious what this number is now, and I am curious how long we have until it goes to 0.

Some time after that--possibly immediately, but possibly still a few years later--coding will be fully automated. (The remaining gap would be management/architecture/etc.)

Showing 3 of 11 replies (Click to show all)

5Vladimir_Nesov20h

I don't agree with the methodology of using benchmarks in the way that you do (as I tried to explain), so looking for better benchmarks in this role would be beside the point. This sounds more like a crux, I think it's likely that more breakthroughs are needed, that relevant milestones are about architectures/methods, different ways of making AGI work, and any benchmarks are only relevant to the extent that they predict a particular way of making AGI work succeed at a given level of compute and low hanging fruit incremental algorithmic improvements. In my model gated by "breakthroughs", accelerating incremental algorithmic improvements or surrounding engineering doesn't particularly help, because it merely picks the low-hanging incremental algorithmic fruit faster (which is in limited supply for a given level of compute and with given methods, but currently takes human researchers years to pick). At the point where even "breakthroughs" are accelerated a lot, AI capable of full automation of civilization is probably already available. So the ways of making AGI work I see is roughly: 1. Just LLMs with pretraining, giving superintelligent in-context learning, making AIs able to figure out how to work around their hobblings using just in-context reasoning. This clearly doesn't work at 2024 levels of compute (which only got to be observed in 2025), and after 2026 levels of compute natural text data starts running out. Maybe there are some relevant sparks at 2029-2031 levels of compute (5 GW), but probably not sufficient on its own. 2. LLMs RLVRed on specific tasks and RL environments that are manually constructed. The IMO results show this is sufficient for mildly superhuman capabilities, especially with 2026 level of pretraining compute or model size, but the RL-level capabilities remain jagged, don't automatically generalize to arbitrary tasks and situations. Possibly RLVRing LLMs on the capability to RLVR LLMs and to construct tasks and RL environments could wo

Daniel Kokotajlo6h20

I understand you don't like benchmark-based methodology. I think you should still answer my question, because if you did have a better benchmark, it would be valuable to me, and I asked nicely. ;) But it's OK now I think it's clear you don't.

Thank you for explaining your model more. I disagree with some bits:

In my model gated by "breakthroughs", accelerating incremental algorithmic improvements or surrounding engineering doesn't particularly help, because it merely picks the low-hanging incremental algorithmic fruit faster (which is in limited supply for a

... (read more)

2Noosphere897h

I want to flag that you estimated that post-2028, we'd slow down due to the MoE data wall, and while MoE is more compute efficient, I don't think it's so much more compute efficient that people won't take the hit of less compute efficiency if data efficiency becomes more relevant post-2028, and we can probably continue mostly pre-training up until the early 2030s based on data limitations alone. That said, I agree this is likely insufficient on its own, so I'm not claiming that pure LLM AGI is a likely path. I think this is probably true for the near term (as in 5-10 year timelines), but over a 15-20 year span with Moore's law, I'd be more confident that continual learning alone could plausibly train in mildly superhuman capabilities, for the same reasons why certain researchers IRL are way more productive than others. This does depend on human level or better sample efficient learning to work out, though. That said, I agree with your methodology much more than Daniel Kokotajlo's methodology on AI prediction (though I do think METR time horizons are more useful and less anchored to specific paradigms than you say it is).

Aaron_Scher's Shortform

Aaron_Scher10d330

People often ask whether GPT-5, GPT-5.1, and GPT-5.2 use the same base model. I have no private information, but I think there's a compelling argument that AI developers should update their base models fairly often. The argument comes from the following observations:

The cost of inference at a given level of AI capability has been dropping quickly. A reasonable estimate is 10× per year, or a halving time of 3.6 months (edit: but 3× is also reasonable, it's hard to be sure).
The cost of new near-frontier AI training runs is relatively small, on the orde

1Petropolitan1d

Same tokenizers on different training data lead to different glitch tokens, see e. g. comparison of Llama-family models in Yuxi Li et al. 2024 https://arxiv.org/abs/2404.09894

eggsyntax8h20

Good point! I hadn't quite realized that although it seems obvious in retrospect.

1Daniel Paleka1d

Here's on the Sonnet size class for now, nothing very interesting...

koanchuk's Shortform

koanchuk2d32-4

TIL Eliezer said that he refuses to read Ted Kaczynski's (aka the Unabomber's) 1995 manifesto Industrial Society and Its Future because "audience should not be a reward for crime", referring to the former mathematician's mail bombing campaign that took the lives of 3 people and injured 23 more.

The ≈35,000 word manifesto was published by the Washington Post under the threat of him killing more people should they refuse, and its publication was encouraged by the FBI to produce new leads. His brother recognized his writing style, which led to Kaczynski's arre... (read more)

Showing 3 of 7 replies (Click to show all)

1quetzal_rainbow11h

Okay, but it's one more reason not to read it: it doesn't contain new things.

Matrice Jacobine8h21

It was new when it was published in 1995! Industrial Society and Its Future was explicitly cited in Kurzweil's The Age of Spiritual Machines (1999) and then Bill Joy's "Why the Future Doesn't Need Us" (2000), the latter of which helped found modern existential risk research.

1koanchuk1d

The consensus notion is basically observational (based on my own social experience, and a cursory internet search revealing the average sentiment held by casual poster and journalists alike). I would also wager that a sample of AI alignment researchers would on average find his predictions on AI risk (quoted above) to be prescient, especially considering the publication date. Beyond that, I don't think they'd have the impression that the parts about AI are insightful while the rest is all just deranged drivel, especially given that his discussion about AI risk is based on concepts and relations which he establishes earlier in the text.

Jeff_H's Shortform

Jeff_H4d30

Intrinsic AI Alignment Based on a Rational Foundation

Here's the big idea. There are many philosophers out there with many different opinions on how reality works. But I'm only aware of one who took the time to lay out his philosophy with logical rigor (more geometrico) and that's Spinoza. His publication "Ethics" (1677) could serve as the basis for a fundamentally different approach to alignment.

I'm not claiming that Spinoza was right or wrong, but he gives us a starting point to debate the specific points of logic and identify any errors.

As I discovered,... (read more)

Mitchell_Porter10h20

I met someone on here who wanted to do this with Kant. I recently thought about doing it with Badiou...

The LLM work that is being done with mathematical proofs, shows that LLMs can work productively within formalized frameworks. Here the obvious question is, which framework?

Spinozist ethics stands out because it was already formalized by Spinoza himself, and it seems to appeal to you because it promises universality on the basis of shared substance. However, any ethics can be formalized, even a non-universal one.

For the CEV school of ali... (read more)

Noah Birnbaum's Shortform

Noah Birnbaum8d4317

I was pretty unimpressed with Dario Amodei in the recent conversation with Demis Hassabis at the World Economic Forum about what comes after AGI.

I don’t know how much of this is a publicity thing, but it felt he wasn't really taking the original reasons for going into AI seriously (i.e. reducing x-risk). The overall message seemed to be “full speed ahead,” mostly justified by some kinda hand-wavy arguments about geopolitics, with the more doomy risks acknowledged only in a pretty hand-wavy way. Bummer.

Showing 3 of 18 replies (Click to show all)

2Dave Orr20h

Does Dario's new essay make you feel better, or worse?

tslarm11h10

Did you mean to ask me, rather than someone else in this thread? (I had a quick look at the new essay but haven't read it in full yet; what I did see didn't seem very surprising, so I don't really feel better or worse at this stage. Happy to say more once I've read it properly, but I think you might have meant to ask someone else, as I wasn't really taking a position in this thread -- just pointing out a relevant section of the talk that I thought you might have missed.)

2yams4d

Can you, or have you elsewhere, elaborate on the presumed shared ontology that doesn’t exist? I think I’d expect you to have a pretty good sense of the delta, and it could be important for learning how to talk to each other more effectively.

testingthewaters's Shortform

testingthewaters1d83

Fundamental attribution error applies to arguments as well. We often think of people who are quick to anger, standoffish, unwilling to accept hypotheticals, or deny "obvious" claims as intrinsically close-minded, foolish, or otherwise unreasonable. Instead those behaviours are often manifested due to other sources of persistent stress in their lives, feeling vulnerable/exposed in the moment, or even just a shitty mood.

1Burny18h

Also I think a lot of people say a lot of things not as falsifiable factual claims, but as status signaling or values signaling or being part of ingroup signaling.

4koanchuk1d

Basically, thinking of people exhibiting bias as being "biased" at the level of identity is a form of bias. Then again, is it unjustified, or a reasonable belief update based on new information? Personally, I was never a fan of the FAE term because it seems to privilege environmental causes over dispositional ones without justification. Yes, it could be that they're angry because they're hungry, but there genuinely exist people who are dispositionally more angry than most others.

testingthewaters14h20

Personally, I was never a fan of the FAE term because it seems to privilege environmental causes over dispositional ones without justification.

I get where you're coming from, but I think most people need a nudge towards an environmental direction rather than towards the dispositional direction. FAE is I guess just a 'reminder' that environmental causes are even possible.

David Africa's Shortform

David Africa16h20

Interpy experiments on Qwen3-TTS.

TLDR: Did some experiments on Qwen3-TTS with some neat results. Not exactly sure what would be interesting to target for a safety perspective, interested in if people have any takes or ideas.

What have I done so far (and how did I build this intuition)?

I took real speech (LibriSpeech with 10 speakers and 5 clips each), ran it through the Qwen3-TTS 12 Hz tokenizer, and got the 16 discrete code streams. Then I treated each layer as a representation and asked simple questions.

Probing. I trained a basic classifier to predict spe... (read more)

koanchuk's Shortform

koanchuk1d50

I see that the "International Treaties on AI" idea takes heavy inspiration from nuclear arms control agreements. However, in these discussions, nuclear arms control is usually pictured as a kind of solved problem, a thing of the past.

I think the validity of this heroic narrative arc that human civilization, faced with the existential threat of nuclear annihilation, came together and neatly contained the problem is dubious.

In the grand scheme of things, nuclear weapons are still young. They're still here and still very much threatening; just because we stop... (read more)

Zac Hatfield-Dodds17h50

Note also: the last US-Russia nuclear arms-control treaty expires next week; far from neatly containing the problem we're watching an ongoing breakdown of decades-old norms. I'm worried.

1StanislavKrym21h

I doubt that the situation with AI systems is worse than that of nukes. While there are nukes in states as rogue as North Korea, I hope that AI development is bottlenecked on compute (which in turn is bottlenecked on TSMC and other compute factories): there is no known model which is better than Grok 3 and was trained by less than 1/300 compute of Grok 3. See also Aaron Scher's analysis and comments to it.