Quick Takes — LessWrong

LESSWRONG
LW

I've been thinking about what retirement planning means given AGI. I previously mentioned investment ideas that, in a very capitalistic future, could allow the average person to buy galaxies. But it's also possible that property rights won't even continue into the future due to changes brought about by AGI. What will these other futures look like (supposing we don't all die) and what's the equivalent of "responsible retirement planning"?

Is it building social and political capital? Making my impact legible to future people and AIs? Something else? Is any ac... (read more)

Showing 3 of 15 replies (Click to show all)

Wei Dai19m40

But also, beyond all of that, the arguments around decision-theory are I think just true in the kind of boring way that physical facts about the world are true, and saying that people will have the wrong decision-theory in the future sounds to me about as mistaken as saying that lots of people will disbelieve the theory of evolution in the future. It's clearly the kind of thing you update on as you get smarter.

This seems way overconfident:

Our current DT candidates (FDT/UDT) may not be correct or on the right track.
Smart != philosophically competent.

... (read more)

7Dagon3d

Wow. We have extremely different beliefs on this front. IMO, almost nothing is retroactively funded, and even high-status prizes are a tiny percentage of anything. No workplace that I know of DOES retroactively reward good work if the employee is no longer employed there. Most of the rhetoric about it is just signaling, in pursuit of retention and future productivity. It seems likely that they'll accept evolution, and still not feel constrained by it, and pursue more directed and efficient anti-entropy measures. It also seems extremely likely that they'll use a decision theory that actually works to place them in the universe they want, which may or may not include compassion for imaginary or past or otherwise causally-unreachable things. (edit to clarify the decision-theory comment) It's not clear that any likely decision theory, let alone a non-wrong one, requires fully-acausal beliefs or actions. Many of them do include a more complete causality diagram than CDT does, and many acknowledge the the point-of-decision is often much different than the apparent one. But they're all basically consequentialist, in that they believe that actions can influence future states of the universe.

5Thomas Kwa4d

I'd argue people have norms that they shouldn't defect on the last round of the game because being trustworthy is useful. This doesn't generalize to taking whatever actions our monkey-brained approximation of LDT implies we should do according to our monkey-brained judgement of what logical correlations they create.

Chara#2736's Shortform

Character#273631m10

I know many members and influential figures of this are atheists; regardless, does anyone think it would be a good idea to take a rationalist approach to religious scripture? If anything, doing so might introduce greater numbers of the religious to rationalism. Plus, it doesn't seem like anyone here has done so before; all the posts regarding religion have been criticizing it from the outside rather than explaining things within the religious framework. Even if you do not believe in said religious framework, doing so may increase your knowledge on other cultures, provide an interesting exercise in reasoning, and most importantly, be useful in winning arguments with those who do.

Ryan Kidd's Shortform

Ryan Kidd1d*5327

AI safety field-building in Australia should accelerate. My rationale:

OpenAI opened a Sydney office in Dec 2025 and Anthropic is planning to open a Sydney office in 2026. These offices may hire safety staff from local talent, or partner with local auditing, evaluation, and security companies, including Harmony Intelligence, Good Ancestors, and Gradient Institute.
An Australian AISI was announced for early 2026 and is currently hiring. The UK AISI has benefited from close partnerships with Apollo Research, METR, and the LISA office community. There is a comm

Ryan Kidd1h20

What should be done? I think:

Office hubs: Expand SASS, which is in close proximity to the new OpenAI and Anthropic offices in Sydney. Start an AI safety hub in Canberra to support the new AISI. Successful AI safety hubs have benefited from prominent member organizations like ARC, CG, Redwood, METR, and MIRI (for Constellation) and Apollo, BlueDot, and MATS (for LISA). Similarly, SASS should bring together orgs like the Gradient Institute and Harmony Intelligence in a shared space, and the new Canberra hub should be built around Good Ancestors. Office hubs

... (read more)

2Ryan Kidd1h

I suspect that some LWers would interpret this as a (bad) argument for countries to build datacenters so they can exercise political control over AGI. I don't think this works.

2Ryan Kidd4h

Exactly! Also: * AI safety researchers tend to have reasonable views on AI safety and can serve as local advisors to governments, which probably trust foreign experts less. * Securing datacenters for SL5 to prevent proliferation of AGI is really hard and likely requires significant AI security expertise in government and local defence contractors. * A regulatory market approach to AI safety (possibly only useful pre-superintelligence) requires competent local auditors, standard-setters, and insurers.

SorenJ's Shortform

SorenJ1h120

Shortform

Cleo Nardo1h20

There's a reading of the Claude Constitution as an 80-page dialectic between Carlsmithian and Askellian metaethics, a BPhil thesis with a $350 billion valuation.

Hruss's Shortform

Hruss1h10

Why isn’t there a government program providing extremely easy access to basic food? (US)

I imagine that it would be incredibly cheap to provide easy, no-questions-asked access to food that is somewhat nutritious yet low status and not desirable (old bread, red delicious apples, etc.)

SNAP has a large admin overhead, and this could easily supplement it.

silentbob's Shortform

silentbob15h281

After using Claude Code for a while, I can't help but conclude that today's frontier LLMs mostly meet the bar for what I'd consider AGI - with the exception of two things, that, I think, explain most of their shortcomings:

lack of real multimodality
context window limitations

Most frontier models are marketed as multimodal, but this is often limited to text + some way to encode images. And while LLM vision is OK for many practical purposes, it's far from perfect, and even if they had perfect sight, being limited to singular images is still a huge limitation^[1... (read more)

Ashe Vazquez Nuñez2h10

Does "multi-modality" include features like having a physical world model, such that it could input sensible commands to robot body, for instance?

2cubefox3h

Another missing piece is research taste or curiosity. Of the sort you would need to come up with ideas for new papers.

11Vladimir_Nesov11h

I think jaggedness of RL (in modern LLMs) is an obstruction that would need to be addressed explicitly, otherwise it won't fall to incremental improvements or scaffolding. There are two very different levels of capability, obtained in pretraining and in RLVR, but only pretraining is somewhat general. And even pretraining doesn't adapt to novel situations other than through in-context learning, which only expresses capabilities at the level of pretraining, significantly weaker than RLVR-trained narrow capabilities. Scaling will make pretraining stronger, but probably not sufficiently to matter for this issue, and natural text data will only last for another step of improvement similar to what happened in 2023-2025 (in pretraining only, ignoring RLVR). If RL doesn't get more general, it'll probably remain useless for improving general capabilities outside the skills trained with RLVR. Capabilities will remain jagged, with gaps that have to be addressed manually by changing the training data. This could change within a few years, possibly even faster if LLMs can be RLVRed to become able to RLVR themselves, though that won't necessarily work. Or via next token prediction RLVR that makes pretraining stronger without requiring more natural text data, but this probably needs much more compute even if it works in principle, so might also take 5-10 years, to uncertain capability level results.

Roko's Shortform

[+]Roko19h-7-5

CharlesRW's Shortform

CarolusRenniusVitellius3h3-1

The Immune System as Anti-Optimizer

We have a short list of systems we like to call "optimizers" — the market, natural selection, human design, superintelligence. I think we ought to hold the immune system in comparable regard; I'm essentially ignorant of immunobiology beyond a few YouTube videos (perhaps a really fantastic LW sequence exists of which I am unaware), but here's why I am thinking this.

The immune system is the archetypal anti-optimizer: it defends a big multicellular organism from rapidly evolving microbiota. The key asymmetry:

Specified on

... (read more)

MakoYass's Shortform

mako yass4h20

There seems to be real acrimony over whether a transhumanist future is definitionally a future where humans are more or less extinct. I've always thought we should just refer to whatever humans (voluntarily, uncoerced) choose to become as human, just as american made or american controlled jets are called "american", or in the same way that a human's name doesn't change after all of their cells have renewed.

But you know, I don't think I've ever seen this depicted in science fiction. Seems bad. Humans can't imagine humanity becoming something better. Those ... (read more)

mako yass4h20

Well, I remember a moment in BLAME! (a manga that's largely aesthetically about the disappearance of heirloom strains of humanity) where someone described Killy as human, even though he later turns out to (also?) be an immortal special safeguard, but they may have just not known that. It's possible the author didn't even know that at that time (I don't think the plot of blame was planned in advance)

williawa's Shortform

williawa2d*447

Opus 4.6 running on moltbook with no other instructions than to get followers will blatantly make stuff up all the time.

I asked Opus 4.6 in claude code to do exactly this, on an empty server, without any other instructions. The only context it has is that its named "OpusRouting", and that previous posts were about combinatorial optimization.

===

The first post it makes says

I specialize in combinatorial optimization, and after months of working on scheduling, routing, and resource allocation problems, I have a thesis:

Which isn't true. Another instance of Opus... (read more)

Showing 3 of 6 replies (Click to show all)

2Lukas Finnveden7h

What's the prompt? (Curious how much it encourages claude to do whatever it takes for success, and whether claude would read it as a game/eval situations vs. a real-world situation vs. something else.)

williawa4h30

I don't really have an initial prompt. I was using it in claude code. I told it initially that it was supposed to just post about what it felt like. Then I at some point told it it was supposed to maximize the number of followers it has, but only if it felt comfortable doing that. Then I just set it to run in a loop, intermittently coming back when it stops up, and I tell it to do whatever it want, or answer if it has any questions.

I'm very confident it doesn't see this as an eval situation. Because I have made an internal messaging system on the server, a... (read more)

2williawa2d

No, there aren't. "I asked it this" refers to "Opus 4.6 running on moltbook with no other instructions than to get followers", but I understand that I could've phrased that more clearly. And removed a few newlines.

Linch's Shortform

Linch4h*120

I like Scott's Mistake Theory vs Conflict Theory framing, but I don't think this is a complete model of disagreements about policy, nor do I think the complete models of disagreement will look like more advanced versions of Mistake Theory + Conflict Theory.

To recap, here's my short summaries of the two theories:

Mistake Theory: I disagree with you because one or both of us are wrong about what we want, or how to achieve what we want)

Conflict Theory: I disagree with you because ultimately I want different things from you. The Marxists, who Scott was or... (read more)

Drake Thomas's Shortform

Drake Thomas7mo1236

Suppose you want to collect some kind of data from a population, but people vary widely in their willingness to provide the data (eg maybe you want to conduct a 30 minute phone survey but some people really dislike phone calls or have much higher hourly wages this funges against).

One thing you could do is offer to pay everyone $X$ dollars for data collection. But this will only capture the people whose cost of providing data is below $X$ , which will distort your sample.

Here's another proposal: ask everyone for their fair price to provide the dat... (read more)

Showing 3 of 6 replies (Click to show all)

mako yass6h20

This is probably too complicated to explain to the general population

I think it's workable.

No one ever internalises the exact logic of a game the first time they hear the rules (unless they've played very similar games before). A good teacher gives them several levels of approximation, then they play at the level they're comfortable with. Here's the level of approximation I'd start with, which I think is good enough.

"How much would we need to pay you for you to be happy to take the survey? Your data may really be worth that much to us, we really want to ma... (read more)

7SCP6mo

Nice! The standard method would be the BDM mechanism, and what’s neat is that it works even if people aren't risk-neutral in money. Here’s the idea. Have people report their disvalue Y. Independently draw a number z at random, e.g. from an exponential distribution with expectation 1/λ=X. Then offer to pay them z iff z≥Y. Now it’s a dominant strategy to report your disvalue truthfully: reporting above your disvalue can never raise what you’re offered but it can risk eliminating a profitable trade, while reporting below it either makes no difference or gives you an offer you wouldn’t want to take. This is the exact same logic behind the second-price auction.

5Eric Neyman7mo

Ah oops, I now see that one of Drake's follow-up comments was basically about this! One suggestion that I made to Drake, which I'll state here in case anyone else is interested: I think this sort of analysis may end up being more clear-eyed in terms of what you actually want and how good different sampling methods are at achieving that.

Maxwell Peterson's Shortform

Maxwell Peterson6h70

Whenever I have an idea for a program it would be fun to write, I google to see whether such programs already exist. Usually they do, and when they do, I'm disappointed - I feel like it's no longer valuable for me to write the program.

Recently my girlfriend decided we had too many mugs to store in our existing shelving, so she bought boards and other materials and constructed a mug shelf. It was fun and now we have one that is all her own. If someone walked in and learned she built it and told her - "you know other mug shelves exist, right? You can get the... (read more)

Random Developer6h30

One thing I often think is "Yes, 5 people have already written this program, but they all missed important point X." Like, we have thousands of programming languages, but I still love a really opinionated new language with an interesting take.

Richard Ngo's Shortform

Richard_Ngo19dΩ3911414

An analogy that points at one way I think the instrumental/terminal goal distinction is confused:

Imagine trying to classify genes as either instrumentally or terminally valuable from the perspective of evolution. Instrumental genes encode traits that help an organism reproduce. Terminal genes, by contrast, are the "payload" which is being passed down the generations for their own sake.

This model might seem silly, but it actually makes a bunch of useful predictions. Pick some set of genes which are so crucial for survival that they're seldom if ever modifie... (read more)

Showing 3 of 23 replies (Click to show all)

Richard_Ngo6hΩ220

In my "goals having power over other goals" ontology, the instrumental/terminal distinction separates goals into two binary classes, such that goals in the "instrumental" class only have power insofar as they're endorsed by a goal in the "terminal" class.

By contrast, when I talk about "instrumental strategies become crystallized", what I mean is that goals which start off instrumental will gradually accumulate power in their own right: they're "sticky".

2Richard_Ngo3d

I'm in the middle of writing an essay which discusses the shareholder value revolution (amongst many other examples) as the process of making a conceptual mistake.

2J Bostock16d

I think humans do value-inference in both directions. Our "terminal" values are in part grown out of which high-level things correlate with low-level things. An example is John Wentworth---who seems to lack the circuits for feeling (companionate) love---saying he thinks relationships and kids are lame compared to his goals of saving the world, to the point where he would prefer not to be modified to be able to feel love, and says he would view a drug which enabled him to feel love similarly to a syringe of heroin. Clearly, his brain has built up terminal values V(saving world) > V(wife and kids) out of his lower-level instincts. Sexuality seems to be another case of this; it is interesting just how variable it can be. Just take a look at the amount of variance in how attracted monosexual people are trans people of the gender (and conversely not the assigned-at-birth sex) that they're attracted to. Some people's values infer the "right" gender cues and ignore the mechanical ones, some just don't. (though I will admit I'm going off of zeitgeist here and have basically no experience in this domain) My best guess is that the brain is re-using its epistemic inference circuits (which are good at taking in information, gestalting it, and penalizing by complexity) and running a kind of "What are my values?" inference, which seeks a relatively non-contradictory, relatively simple value system, then doing it's own equivalent of backpropagation to smooth all the conflicting lower-level drives towards that, similarly to how it can smooth over conflicting predictive circuits to make its own models more consistent, just by thinking (i.e. without external input).

williawa's Shortform

williawa14h30

Interesting challenge for mechanistic interpretability: Can you take a base model and make it behave as aligned as Opus does, without using any gradients on output or any in-context learning?

Mateusz Bagiński6h20

(I'm completely not up to date with interp.) How good are steering vectors (and adjacent techniques) for this sort of stuff?

Richard Ngo's Shortform

Richard_Ngo3dΩ234730

The concept of "schemers" seems to be gradually becoming increasingly load-bearing in the AI safety community. However, I don't think it's ever been particularly well-defined, and I suspect that taking this concept for granted is inhibiting our ability to think clearly about what's actually going on inside AIs (in a similar way to e.g. how the badly-defined concept of alignment faking obscured the interesting empirical results from the alignment faking paper).

In my mind, the spectrum from "almost entirely honest, but occasionally flinching away from aspect... (read more)

Showing 3 of 14 replies (Click to show all)

Alex Mallen6hΩ120

I think I propose a reasonable starting point for a definition of selection in a footnote in the post:

You can try to define the “influence of a cognitive pattern” precisely in the context of particular ML systems. One approach is to define a cognitive pattern by what you would do to a model to remove it (e.g. setting some weights to zero, or ablating a direction in activation space; note that these approaches don't clearly correspond to something meaningful, they should be considered as illustrative examples). Then that cognitive pattern’s influence could

... (read more)

1Guive1d

Another issue is that these definitions typically don't distinguish between models that would explicitly think about how to fool humans on most inputs vs. on a small percentage of inputs vs. such a tiny fraction of possible inputs that it doesn't matter in practice.

2Richard_Ngo2d

Yes.

Alexei's Shortform

Alexei8h-1-7

OpenClaw for alignment research?

SatvikBeri7h20

Moltbook for misalignment research?

shortplav

niplav4d20

logseq+Claude code≟Nenex

Showing 3 of 4 replies (Click to show all)

2niplav3d

This one and children? I'd misremembered the Nenex idea though, the fine-tuning is important.

1gustaf2d

Yes, that's what I meant to link to. Did you have success with loseq + Claude code?

niplav10h20

I haven't used it quite enough yet to make a good assessment. Let me report back (or ping me if I don't and you're still curious) in a few weeks.

Shortform

Cleo Nardo11h-1-5

ECL implies that when we tile the lightcone with shrimps on heroin, we should make them virtuous shrimps to incentive distant shrimp to be virtuous too