Quick Takes — LessWrong

LESSWRONG
LW

A lot of the rationalist discourse around birthrates don't seem to square away with AGI predictions.

Like the most negative predictions of AGI destroying humanity in the next century or two leaves birthrates completely negligent as an issue. The positive predictions with AGI leave a high possibility of robotic child rearing and artificial wombs (when considering the amount of progress even us puny humans have already made) in the next century or two which also makes natural birthrates irrelevant because we could just make and raise more humans without the n... (read more)

Showing 3 of 9 replies (Click to show all)

lilkim20251h10

Though noteworthy that he has taken no steps towards technical solutions for the birth rate issue

I mean, as individual men in the West go, having fourteen children is pretty above-average, and he seems to have gotten the process down to a science. He's not a Saudi oil baron with three digits of offspring, but he's certainly taken a Silicon Valley approach to it.

Pro-Natalists, in general, seem to take a 'lead-by-example' tack, which isn't horrible considering that it demonstrates an understanding of the consequences of materially encouraging people who woul... (read more)

4Zack_M_Davis6h

There's calculated rational "what if we're wrong" hedging, but then there's ... holding out hope? (I'm not claiming it's rational; I'm trying to articulate the psychology.) To conclude "AGI is coming, no point in having children" amounts to betting on death, giving up on believing in a human future. It kind of makes sense that an evolved creature would be inclined to cling to the belief in a future and live as if it were true despite evidence to the contrary; as irrationalities go, it's quite adaptive.

4Mateusz Bagiński7h

A simple steelman is something like "if we're very wrong about A[G/S]I, then birth rates are a big issue, so we better invest some resources into it, in case we're wrong". This steelman is a valid position to have, but is not good as a steelman in this context, because attributing this view to people like Musk is probably a great stretch (and probably also to other people the OP is referring to, but I'm not tracking that kind of stuff, so unsure). If the issue to be fixed is just population (growth), then yeah. But if we're fine with a lower population[1] without the demographic collapse degrading everything the way it would degrade by default as a consequence, then a gradual development and adoption of "sub-AGI" AI could automate a bunch of stuff in a way that roughly catches up with the declining population. (Assuming you meant "AGI-ward progress pause", which maybe you didn't mean.) 1. ^ I'm not in favor, but also wouldn't consider it a great tragedy if we can intervene out the consequences.

eggsyntax's Shortform

eggsyntax2mo*240

Quasi-beliefs

This shortpost is just a reference post for the following point:

It's very easy for conversations about LLM beliefs or goals or values to get derailed by questions about whether an LLM can genuinely said to believe something, or to have a goal, or to hold a value. These are valid questions! But there are other important questions about LLMs that touch on these subjects, which don't turn on whether an LLM belief is a "real" belief. It's not productive for those discussions to be so frequently derailed.

I've taken various approaches to this proble... (read more)

Showing 3 of 15 replies (Click to show all)

lilkim20251h10

Assuming people accept the model that LLM behavior is primarily determined by modeling the behavior of some subset (such that fine-tuning works primarily by shaping the subset of humans that the model emulates) of the human writers it was trained on, it might be simplest to ask whether the model "behaves like a person who believes X".

This framing carries practical benefits (again, so long as you agree with the assumption above), in that the fine-tuning paradigm can be examined in the context of identifying what causes the model to upweight, say, the "a bla... (read more)

2Adele Lopez4h

I've been using a tilde (e.g. ~belief) for denoting this, which maybe has less baggage than "quasi-" and is a lot easier to type. It funny, one of the main use-cases of this terminology is when I'm talking to LLMs themselves about these things.

2eggsyntax2h

One thing that would make me hesitate to use ~ is that it already commonly means 'approximately equal to' (as a more-easily-typed substitute for ≈). That certainly feels like a related meaning, but what I appreciate about Chalmers' coinage is that it's very precise about what you are and are not claiming.

Richard Ngo's Shortform

Richard_Ngo2d55-5

A response to someone asking about my criticisms of EA (crossposted from twitter):

EA started off with global health and ended up pivoting hard to AI safety, AI governance, etc. You can think of this as “we started with one cause area and we found another using the same core assumptions” but another way to think about it is “the worldview which generated ‘work on global health’ was wrong about some crucial things”, and the ideology hasn’t been adequately refactored to take those things out.

Some of them include:

Focusing on helping the worst-off instead of em

... (read more)

Showing 3 of 18 replies (Click to show all)

plex2h42

Some of them include:

imo a larger one is something like not rooting the foundations in "build your own models of the world so that you contain within you a stack trace of why you're doing what you're doing" + "be willing to be challenges and update major beliefs based on deep-in-the-weeds technical arguments, and do so from a highly truth-seeking stance which knows what it feels like to actually understand something not just have an opinion".

Lack of this is fineish in global health, but in AI Safety generates a crop of people with only surface deferral flavor understanding of the issues, which is insufficient to orient in a much less straightforward technical domain.

2Caleb Biddulph4h

Interesting, I'd never explicitly considered that Peter Singer (you should expand your moral circle and do as much good as you can) and GiveWell (given that you want to do good, how to do it?) started as totally different memeplexes and only merged later on. It makes sense in retrospect.

2bfinn11h

To clarify, I think your criticism of utilitarianism/consequentialism is of a naive form of it that only looks at first-order effects. Not ‘proper’ utilitarianism. But yes no doubt many are naive like this, and it’s v hard to evaluate second- and higher-order effects (such as exploitation and coordination). Also, this kind of naivety is particularly common on the left.

Elizabeth's Shortform

Elizabeth1d430

I'm working on a piece for Asterisk magazine on how biohackers (broadly defined) think about risk. Are you interested in being interviewed? Do you know someone who is? I'm taking a very broad definition of biohacker (self-medicating to save money counts), but some people I'm especially interested in talking to are those who:

biohack for its own sake- no one needs magnets in their fingernails, but you want them.
receive treatment within the standard medical system, but playing an active role in personalizing it (e.g. trans people adjusting their HRT, diabetic

... (read more)

2AlphaAndOmega16h

I would sign up, but I have a feeling that a doctor self-prescribing isn't quite what you're looking for.

Elizabeth4h20

Depends. Are you strictly following standard of care, or personalizing for yourself?

Linch's Shortform

Linch21h*168

"Most people make the mistake of generalizing from a single data point. Or at least, I do." - SA

When can you learn a lot from one data point? People, especially stats- or science- brained people, are often confused about this, and frequently give answers that (imo) are the opposite of useful. Eg they say that usually you can’t know much but if you know a lot about the meta-structure of your distribution (eg you’re interested in the mean of a distribution with low variance), sometimes a single data point can be a significant update.

This type of limited conc... (read more)

5Drake Morrison5h

See also: The First Sample Gives the Most Information

Linch4h20

wow thanks! It's the same point but he puts it better.

2cubefox12h

That's a case of reducing a high uncertainty (high entropy). The more classical Bayesian case where you learn a lot is when you were previously very certain about what the first data point will look like (i.e. you "know" a lot in your terminology, though knowledge implies truth, so that's arguably the wrong term), but then the first data point turns out to be very different from what you expected. So in summary, you will learn very little from a single example if you are both a) very sure about what it will look like and b) it then actually very much looks like you expected.

evhub's Shortform

evhub1y*Ω521323

I'm interested in soliciting takes on pretty much anything people think Anthropic should be doing differently. One of Alignment Stress-Testing's core responsibilities is identifying any places where Anthropic might be making a mistake from a safety perspective—or even any places where Anthropic might have an opportunity to do something really good that we aren't taking—so I'm interested in hearing pretty much any idea there that I haven't heard before.^[1] I'll read all the responses here, but I probably won't reply to any of them to avoid revealing anythin... (read more)

Showing 3 of 66 replies (Click to show all)

Ted Sanders5h10

Glad to see prefill was disabled for Opus 4.6!

4Drake Thomas8mo

Coming to this comment late, but off the cuff takes: 1. Seems good. 2. Unclear to me if the decisionmaking friction of implementing this now is worth the costs; agree there are some worlds where you want to do it when things really heat up. 3. Not a lawyer but my guess is that there's no way to do this which doesn't let the US legal system access the recordings before 10y are up in the event of litigation. 4. Kinda torn on this; I think this is what organizations in a sane world should do, but on actual Earth I don't know if it's worth the weirdness points. I care a lot about Anthropic spending down its credibility as a normal serious organization on actions that have a low ROI, and I think the audience that would actually understand and respect this in the right way is pretty small. 5. This sort of stuff generally seems good, haven't thought about the doc's particular proposal in detail. I think stuff like welfare reports in the Claude 4 model card are moving in this direction already. 6. Agree you want this option, not sure what concrete actions you think would be good besides "think about the fact that you might do this"? 7. Yep, would happy about more of this on the margin. 8. I think as of June 2025 A\ has basically done this? 9. Agree this would be good.

4Chris_Leong1y

I believe that Anthropic should be investigating artificial wisdom: I've summarised a paper arguing for the importance of artificial wisdom with Yoshua Bengio being one of the authors. I also have a short-form arguing for training wise AI advisors and an outline Some Preliminary Notes of the Promise of a Wisdom Explosion.

Linch's Shortform

Linch16h51

One thing that I find somewhat confusing is that the "time horizon"-equivalent for AIs reading blog posts seems so short. Like this is very vibes-y, but if I were to think of a question operationalized as "I read a blog post for X period of time, at what X would I think Claude has a >50% chance of identifying more central errors than I could?" intuitively it feels like X is very short. Well under an hour, and likely under 10 minutes.

This is in some sense surprising, since reading feels like a task they're extremely natively suited for, and on other tasks like programming their time horizons tend to be in multiple hours.

I don't have a good resolution to this.

Seth Herd5h20

The task you're talking about isn't reading, it's logical analysis or something like that.

I think it is noteworthy that current llms are bad at this.

Of course, what counts as an error within the local context of the essay and what counts as an error? Given all the shared context, the writer and reader rely on to interpret. It, is highly subjective and debatable. So you need some sort of committee of expert humans to compare to.

3yams15h

My best guess for this is something like coding is both more routinized and more explicit. That is, the thinking and the tokens are synonymous. One can't think the correct code and not be able to actually implement the correct code if you're literally thinking about the exact text of the code. Whereas one can think the correct text in natural language and not implement the correct idea. Another way of saying this is: there are more reasoning steps in excess of the tokens implicitly required by the most bare bones version of the reasoning process. Probably you've already thought of this and thought of a reason that it's wrong.

sarahconstantin's Shortform

sarahconstantin5h20

links 2/11/26: https://roamresearch.com/#/app/srcpublic/page/02-11-2026

https://www.clinicaltrialsabundance.blog/p/the-moderna-rtf-and-the-cost-of-regulatory Ruxandra Tesloianu on why FDA changing its mind to reject trial protocols it previously approved is harmful for biomedical progress
https://www.drmaciver.com/2015/04/some-empirically-derived-testing-principles/ D.R. Maciver advice on software testing
https://notebook.ldeming.com/garden/index.html Laura Deming on how, if longevity ever really works, it would be creating a new sort of creature

... (read more)

Zac Yauney's Shortform

Zac Yauney6d172

Elon Musk is what Scott Adams was trying to be.

In Scott Alexander's "Dilbert Afterlife" he pointed out that Adams had the idea that if one was so smart, the highest ROI use of that strength was to try to control lesser minds by taking advantage of their vulnerabilities. His hypnosis and "linguistic killshots" were attempts at this.

Listening to Elon on the Dwarkesh pod and I feel like they are finally exposing him. Dwarkesh and John Collison are engaging in more or less rationalist discourse and asking all the obvious follow-up questions, and Elon just has ... (read more)

Viliam12h20

we try to operate on simulacra level 1 (genuinely saying what we mean) and it turns out that that is by far the weakest of the four.

Level 1 allows you to work on something alone. So if you can't find other people who would cooperate with you (e.g. because you are a little autistic), level 1 is the only one you can work at. Your options are to work at level 1 alone, or serve other people, or fail.

Which I suspect might have also been Scott Adams' problem. He could see that the other levels work better, but he couldn't work at them efficiently. So he just kep... (read more)

sam's Shortform

sam2d112

Are models like Opus 4.6 doing a similar thing to o1/o3 when reasoning?

There was a lot of talk about reasoning models like o1/o3 devolving into uninterpretable gibberish in their chains-of-thought, and that these were fundamentally a different kind of thing than previous LLMs. This was (to my understanding) one of the reasons only a summary of the thinking was available.

But when I use models like Opus 4.5/4.6 with extended thinking, the chains-of-thought (appear to be?) fully reported, and completely legible.

I've just realised that I'm not sure what's goin... (read more)

Showing 3 of 11 replies (Click to show all)

sam12h10

Hmm, but when you use these models in the chat interface, you can literally open up the reasoning tab and watch it be generated in real time? It feels like there isn’t enough time here for that reasoning to have been generated by a summarizer

2Kei Nishimura-Gasparian1d

One more data point: During the time that the reasoning traces for Gemini 2.5 Pro were publicly available, I noted they also looked especially structured/legible. Though I never ran any quantitative analysis on it and it's no longer possible to do.

1Rauno Arike1d

Probably not, but the summarizer could additionally be triggered by unsafe CoTs.

MaxIRL's Shortform

MaxIRL5d120

The benefits of lazy parenting.

My wife and I often look at ourselves as lazy parents. From as early as the kids could manage, we would expect them to get up and get their own water or snack or dress themselves or even shower themselves. On some occasions, the kids, at 4 and 6, would do the entire bedtime routine, saying goodnight downstairs, and then go up, shower, dress, teeth, and bed on their own.

Sometimes we feel guilty, but the kids love the independence we give them. We give them endless love, we show it, but we avoid doing things for the... (read more)

Viliam12h20

I agree. The question is when to start being lazy about what.

warner's Shortform

warner6d30

I have a concept that I am considering making a post about: How culture war and political signaling is all about boundary maintenance rather than truth seeking. And that if you have a label, which you decide to believe in as your identity, then the person who controls the definition of that label by proxy controls you. If you are a son of god, then who are under the control of whoever decides what god is and what he wants. Similarly, if you believe that being a liberal is the only way to be a person who “believes in human rights” than you are under the con... (read more)

Viliam13h20

The idea seems correct. If you identify as X, and an authority that you respect says "all true X believe Y", you are more likely to accept Y uncritically. Especially if other Xs around you accept it; that creates social pressure.

Politics is considered a minefield here, so you would have to write the article carefully, to avoid tribalism. (Basically, do not mix "this is an analysis of how political tribes work" with "this is my favorite political tribe" in the same article.) It would probably be better to use multiple examples from various sides, rather tha... (read more)

Joseph Van Name's Shortform

Joseph Van Name5d90

For machine learning, it is desirable for the trained model to have absolutely no random information left over from the initialization; in this short post, I will mathematically prove an interesting (to me) but simple consequence of this desirable behavior.

This post is a result of some research that I am doing for machine learning algorithms related to my investigation of cryptographic functions for the cryptocurrency that I launched (to discuss crypto, leave me a personal message so we can discuss this off this site).

This post shall be about linear ... (read more)

Showing 3 of 6 replies (Click to show all)

1Joseph Van Name3d

For experiments, I just used a convex combination of point mass measures for μ where the point masses are generated uniformly at random (though I might get something more complicated if I tried evaluating the integrals). I then attempted to find multiple local maxima by the usual gradient ascent. If I always end up with the same local maximum, I presume that there is only one local maximum even though I have no mathematical proof that this is the case. I am redoing the experiments and the only way I can get pseudodeterminism to fail is by using real inner product spaces instead of complex inner product spaces and by setting n=1 (and in this case, pseudodeterminism fails because set of all points where the fitness function returns a real number instead of negative infinity has multiple components). When pseudodeterminism fails, it does not even fail that badly. The distribution of all models that we get has low collision entropy -log(X=Y), so P(X=Y) when X,Y are trained models with different initializations is still high. Pseudodeterminism does not seem to be rare, but the problem in machine learning is to pseudodeterministically train machine learning models that can solve interesting and challenging problems; I have been working on this in my spare time (without anyone's help), but since people don't seem to be interested in this, progress has been slow.

1kbear1d

two questions: 1. this seems to include an assumption regarding the points between training samples. if we take the point masses as the known values, then with this step we're adding some interpolation between those. (that is, if we were "really training" these things, then the integral would look like a sum, since it would only have support at those (x,y) that represent points in our training data.) 2. have you tried adversarialy constructing \mu such that the integral has multiple maxima? if so, what did you run into? i worry that this could be a case where "random" examples mostly have this property, but some important subset does not (similar to how random functions are nowhere differentiable, but we can still do calculus.) i'm interested in this! but i've only been introduced to these ideas recently, by your post! please read these questions as my own attempts to understand how you're thinking about it.

Joseph Van Name14h*10

Yes. When we take convex combinations of finitely many point mass measures, the integral is just a sum. I use the sum of finitely many elements for ease of calculations, but to prove theorems, I should use measures for full generality.

The idea of finding an object $A$ along with distinct local optima $G_{A} (x_{1}), \dots, G_{A} (x_{n})$ with $n$ maximized looks like an interesting problem to work on. I have not worked on this kind of objective before, but I can certainly try this, as I have a few ideas of how to do this. This might work better for dis... (read more)

shortplav

niplav13d*30-3

In case I don't write a full post about this:

The question whether reversible computation produces minds with relevant moral status is extremely important. Claude estimates me that it'd be a difference between having $10^{70}$ and $10^{124}$ mind-seconds instantiable in the reachable universe. (Because reversible minds could stretch our entropy budget long into the black hole era.)

Question is whether the reversing of the computation that makes up the mind and the lack of output (that'd imply bit-erasure) entail that the mind "didn't really exist".

There ar... (read more)

Showing 3 of 9 replies (Click to show all)

4Gurkenglas5d

if it's just about whether there was no effect, you could totally reversibly compute 10124−70 minds for one second, summarize their outputs into one bit (e.g. whether most of them voted in favor of a proposal), observe that bit, and then reverse the mind-computation to the initial state.

niplav14h20

Yeah… seems right. I could cop out with "ah, the number of bit erasures by a program matters, not just whether it did", but I don't have any good reason for believing this.

4Mateusz Bagiński5d

I think you meant to say "is not conscious".

papetoast's Shortforms

papetoast20h*10

Update: Brushing after eating acidic food is likely fine.

Context: 7 months ago, me in Adam Zerner's shortform:

I remember something about not brushing immediately after eating though. Here is a random article I googled. This says don't brush after eating acidic food, not sure about the general case.
https://www.cuimc.columbia.edu/news/brushing-immediately-after-meals-you-may-want-wait
“The reason for that is that when acids are in the mouth, they weaken the enamel of the tooth, which is the outer layer of the tooth,” Rolle says. Brushing immediately afte

... (read more)

3silentbob16h

"was not associated" tells us more about the sample size than the effect, as far as I can tell, though, doesn't it? The 0.82-2.42 CI does not seem very reassuring. Especially given this is just observational - it could well be that people who brush immediately after intake of something that's bad for your teeth are generally conscientious about their dental health, so if they still end up with worse outcomes in this study (albeit not reaching statistical significance), then brushing quickly after acid intake could potentially be even worse than this CI (weakly) suggests. That said, the measured odds ratios for fruit/acids between meals were so much larger that it might indeed make more sense to focus on these than on the exact timing of brushing.

papetoast15h20

Thank you for double checking.

Overall I am still very uncertain, but lean towards it being fine. Even dentists are giving mixed signals.

Unfortunately Chinese sources but these are dentists saying you can brush immediately

Definitely there are dentists saying you should wait too

Jacobson's Shortform

Jacobson18h10

Just thinking out loud: If AI (or something else) causes the S&P500 to go up by 50%...

By January 21st, 2028 then you can achieve a 1324% ROI (https://finance.yahoo.com/quote/SPY280121C01000000/)
By June 16, 2028 then you can achieve a 659% ROI (https://finance.yahoo.com/quote/SPY280616C01000000/)
By December 15, 2028 then you can achieve a 315% ROI (https://finance.yahoo.com/quote/SPY281215C01000000/)

Curious to hear others thoughts, especially those who have short AGI timelines. *Of course this is not financial advice

John Coleman's Shortform

John Coleman19h10

The music industry has already passed the threshold for digital AGI, but seems to be experiencing disruption dramatically less than AI song generation capabilities might suggest. Anyone can create a song of expert level quality in under 10s with an automatically generated prompt (or write one themselves if they want). The Chainsmokers took about 8 hours in total to make the single Roses, which has over 1.3bn streams as of 02/10/2026 and I believe is on the quicker end of writing time for pop songs, although I am not sure reliable data exists for this anywh... (read more)

adamzerner's Shortform

Adam Zerner7mo354

I remember a hygienist at the dentist once telling me that toothpaste isn't a huge deal and that it's the mechanical friction of the toothbrush that provides most of the value. Since being told that, after a meal, I often wet my toothbrush with water and brush for 10 seconds or so.

I just researched it some more and from what I understand, after eating, food debris that remains on your teeth forms a sort of biofilm. Once the biofilm is formed you need those traditional 2 minute long tooth brushing sessions to break it down and remove it. But it takes 30+ mi... (read more)

Showing 3 of 6 replies (Click to show all)

papetoast20h10

Update: Brushing after eating acidic food is likely fine.

Should We Wait to Brush Our Teeth? A Scoping Review Regarding Dental Caries and Erosive Tooth Wear

Key messages: Although the available evidence lacked robust clinical studies, tooth brushing using fluoridated products immediately after an erosive challenge does not increase the risk of ETW [Erosive Tooth Wear] and can be recommended, which is in line with recommendations for dental caries prevention. Furthermore, we suggest updating the international guidelines to promote individualized recommendatio... (read more)

2Adam Zerner7mo

My understanding is that toothpicks are for scraping the area in between teeth, not the surface of the tooth itself.

1papetoast6mo

I think the area in between teeth can only be thoroughly cleaned by flossing unless you brush really really carefully

Linda Linsefors's Shortform

Linda Linsefors2d*80

Even now and then I meet someone who tries to argue that if I don't agree with them this is because I'm not open mided enough. Is there a term for this?

Epistemically I'm not convinced buy this type of arugment, but socialy it feels like I'm beeing shamed, and I hate it.

I also find it hard to call out this type of behaviur when it happens, even when I can tell exactly what is going on. I think it I had a name for this behaviour it would be easier? Not sure though?

Edit to add:

I've now got some more time to figure out what I want and don't want out of this th... (read more)

Showing 3 of 11 replies (Click to show all)

cubefox22h20

A more transparent term would be psychologizing:

psychologize: to speculate in psychological terms or on psychological motivations