LESSWRONG
LW

Quick Takes

I very roughly polled METR staff (using Fatebook) what the 50% time horizon will be by EOY 2026, conditional on METR reporting something analogous to today's time horizon metric.

I got the following results: 29% average probability that it will surpass 32 hours. 68% average probability that it will surpass 16 hours.

The first question got 10 respondents and the second question got 12. Around half of the respondents were technical researchers. I expect the sample to be close to representative, but maybe a bit more short-timelines than the rest of METR staff.

The average probability that the question doesn't resolve AMBIGUOUS is somewhere around 60%.

4Cole Wyeth6h

This would be way above-trend??

StanislavKrym38m10

Nikola's comment about the 20hr median, let alone the 29% probability of a 32hr horizon or higher, does require more than two doublings (and, in the case of 20hr, far closer to three doublings) of GPT-5.1-Codex-Max's result of 2h42m. The most recent trend of a doubling per 7 months is the trend observed between o3 and GPT-5.1-Codex-Max. But there was the less recent trend of Claude 3.5 Sonnet-o3 where a doubling would happen in 4 months.

I suspect that METR will soon publish information about Gemini 3 Pro, Claude Opus 4.5 and GPT-5.2, and it will let ... (read more)

9Nikola Jurkovic16h

This very roughly implies that the median of "50% time horizon as predicted by METR staff" by EOY 2026 is a bit higher than 20 hours.

RationalElf's Shortform

RationalElf4d5717

Am I understanding correctly that recent revelations from Ilya's deposition (e.g. looking at the parts here) suggest Ilya Sutskever and Mira Murati seem like very selfish and/or cowardly people? They seem approximately as scheming or manipulative as Sam Altman, if maybe more cowardly and less competent.

My understanding from is that they were basically wholly responsible for causing the board to try to fire Sam Altman. But when it went south, they actively sabotaged the firing (e.g. Mira disavowing it and trying to retain her role, Ilya saying he regr... (read more)

Showing 3 of 14 replies (Click to show all)

2Shankar Sivarajan6h

From what I understand, they executed a risky maneuver, lost, tried to salvage what they could from the wreckage (by pinning the blame on someone else), but got pushed out anyway. So I can see where you're getting "scheming or manipulative" (them trying this scheme), "less competent" than Altman (losing to him), and "very selfish" (blaming other people). But where are you getting "cowardly" from? From their attempt at minimizing their losses after it became clear their initial plan had become exceeding unlikely to succeed? If so, I'd say it speaks to how poorly you think of valor if you believe it precludes so sensible an action.

RationalElf5h32

"cowardly" because my strong guess is that their actions were driven by fear of social censure rather than calculated attempts to minimize losses. If they were trying to minimize losses to their non-selfish goals of ousting Sam A, who I think they believed to be a bad and dangerous actor, that would have been better served by coming clean about why they did what they did.

2robo2d

Make sure the people on the board of OpenAI were not catastrophically naive about corporate politics and public relations? Or, make sure they understand their naïveté well enough to get professional advice beforehand? I just reread the press release and can still barely believe how badly they effed it up.

Annabelle's Shortform

Annabelle3d3316

One theme I've been thinking about recently is how bids for connection and understanding are often read as criticism. For example:

Person A shares a new idea, feeling excited and hoping to connect with Person B over something they've worked hard on and hold dear.

Person B asks a question about a perceived inconsistency in the idea, feeling excited and hoping for an answer which helps them better understand the idea (and Person B).

Person A feels hurt and unfairly rejected by Person B. Specifically, Person A feels like Person B isn't willing to give their sinc... (read more)

Showing 3 of 5 replies (Click to show all)

Roman Malov11h10

The post "Butterfly Ideas" seems relevant.

1Annabelle1d

I feel like I could have written this (and the rest of your comment)! It's confusing and deflating when deep engagement and joy aren't recognized as such. I've tried the same with mixed effectiveness. In in-person contexts, nonverbal information makes it much easier to determine when and how to do this. I've found it's more difficult online, particularly when you don't know your interlocutor--sometimes efforts to affirm the connection and points of non-contention are read as pitying or mocking. I imagine this is partially attributable to the high prevalence of general derision on social media.

3DirectedEvolution2d

Yeah, I think that part of the training my PhD program offers is learning how to handle and deliver criticism. Besides the common advice of choosing your language carefully, being praiseful, and limiting questions/critiques to zero or one most important area in most settings, my favorite tactic when giving my own presentations is to frontload limitations. Just to be clear (this is frontloading!), what follows is speculation based on my own personal experiences. When I start presenting my research, I introduce the audience first to the biology (3D genome architecture), then the technology I use to study it (the Hi-C family of assays). That technology is complex and has important limitations (sparsity, averaging over time/homologs/cells). If I don't frontload limitations, what's an informed audience going to be thinking about once I introduce the technology? They're going to be thinking about those limitations, figuring them out in their own head. "Hey, wouldn't an assay that works like that end up averaging over time/homologs/cells?", they'll think. And while they're thinking those thoughts, I'll be trying to push forward into my own projects. So they'll be irritated by the distraction of trying to think about those limitations while also keeping up with what I'm saying about my projects. They'll be associating those limitations and that irritation with my projects. And the natural question for them to ask will be about the existence of those limitations, rather than about my findings. Not good. By frontloading limitations, it alleviates the cognitive burden on the audience of thinking up those limitations for themselves and compartmentalizes the limitations so that the cognitive load of thinking about them isn't interfering with attempts to understand what I'm saying about my project. They can then ask me questions about the specifics of my project, which is what's most interesting, what I'm most excited to talk about, and will lead to positive feeling on both si

Viliam's Shortform

Viliam5d7024

Do we have some page containing resources for rationalist parents, or generally for parents of smart children? Such as recommended books, toys, learning apps, etc.

I found tag https://www.lesswrong.com/w/parenting but I was hoping for some kind of best textbooks / recommendations / reference works but for parents/children.

Showing 3 of 10 replies (Click to show all)

2Gunnar_Zarncke3d

From DeJong et al. (2019): https://scispace.com/papers/alcohol-use-in-pregnancy-1tikfl3l2g (page 3)

2Viliam2d

I think the pro-pregnancy-alcohol argument was that the small amount of alcohol won't hurt the fetus, not that it won't reach it.

Gunnar_Zarncke2d20

I'm not arguing either way. I just note this specific aspect that seems relevant. The question is: Is the babies body more susceptible to alcohol than an adults body. For example, does the liver work better or worse than for a baby? Are there developmental processes that can be disturbed by the presence of alcohol? By default I'd assume that the effect is proportional (except maybe the baby "lives faster" in some sense, so the effect may be proportional to metabilism or growth speed or something). But all of that is speculation.

Oliver Sourbut's Shortform

Oliver Sourbut4d154

Learned about 'Harberger tax' recently.

The motivation is like

private assets (private use and excludable) incentivise investment/productive improvement/development well
but private ownership of non-liquid, non-fungible, idiosyncratic things (factories, houses, natural resources maybe, AI systems maybe?) acts like a monopoly, distorting allocation (more productive/valued uses aren't often realised)
- e.g. NIMBYism
- e.g. 'holdout pricing'
- e.g. inefficient bargaining due to information asymmetry
common ownership and lease auctioning incentivises efficient al

J Bostock11h30

Strongly agree. Sell-price tax with forced sales sounds like something a cryptocurrency would implement. It might work there, since if a malicious bidder tried to buy your TOKEN at above-market price, you could automatically buy a new one within the same block, at actual-market price. This could also work for fungible but rapidly-transferrable assets like cloud GPU time.

If taxing physical goods (like infrastructure or even land) which is where a lot of value in the world lies, this does just open up companies for extortion. E.g. what if I demand to buy one... (read more)

1metachirality15h

Good thing you're only incentivized to sell at market price then, not how much value you'd assign personally!

2homosapien9712h

I'm not sure what you mean. You can list something at market price rather than how you actually value it, but that opens you up to significant risk from market volatility and malicious bids (from competitors looking to cripple your business, rich people you've offended, etc). If sales cannot actually be forced this seems more workable, but still exploitable by malicious actors.

Caleb Biddulph's Shortform

Caleb Biddulph1d96

Consider saying "irrational" instead of "insane"

Rationalists often say "insane" to talk about normie behaviors they don't like, and "sane" to talk about behaviors they like better. This seems unnecessarily confusing and mean to me.

This clearly is very different from how most people use these words. Like, "guy who believes in God" is very different from "resident of a psych ward." It can even cause legitimate confusion when you want to switch back to the traditional definition of "insane". This doesn't seem very rational to me!

Also, the otherizing/dismissiv... (read more)

Showing 3 of 7 replies (Click to show all)

Caleb Biddulph14h20

Yeah, I think some rationalists, e.g. Eliezer, use it a lot more than the general population, and differently from the popular figurative sense. As in "raising the sanity waterline."

2ChristianKl20h

I think the key difference between a normal guy who believes in God and someone in a psych ward is often that the person who believes in God does so because it's what other people in authority told him but the person in the psych ward thinks for themselves and came up with their delusion on their own. This often means their beliefs are self-referential in a way that prevents them from updating due to external feedback. If you believe that the word rational is supposed to mean something along the lines of "taking actions that optimize systematically for winning", believing something just because an authority told you can sometimes be irrational and sometimes be rational. If you want to talk well about behavior you don't like, it makes sense to use words that refer to the reason for why the behavior is bad. Credulous or gullible might be words that better describe a lot of normie behavior. The person who believes in god just because his parents and teachers told him is credulous.

8Eli Tyre1d

I think there's some bad knock-on effects for normalizing the use of "insane" to talk about very common features of the world: I think it makes social-rationalists to willing to disparage people and institutions, as part of a status-signaling game, often without much careful thought. But I think there's also something valuable about eg. calling belief in God "insane". There's a kind of willingness to call a spade a spade, and not back away from how the literal stated beliefs, if they were not pervasive, would in fact be regarded as signs of insanity.

Parker Conley's Shortform

Parker Conley15h10

An AI content X/Twitter account with nearly 100k followers blocked me, and I got a couple of disapproving replies for pointing out that the account was AI-generated. I quote-tweeted the account mostly to share a useful Chrome Extension that I've been using the detect AI content, but I was surprised that there was a negative reaction in the form of a few replies pointing out the account was AI-generated. I am neither pro- nor anti-AI accounts, but being aware of the nature of the content seems to be useful.

Would be curious to hear others' thoughts on the ph... (read more)

shortplav

niplav21h-1-20

I found Yarrow Bouchard's quick take on the EA Forum regarding LessWrong's performance in the COVID-19 pandemic quite good.

I don't trust her to do such an analysis in an unbiased way ^[[1]] , but the quick take was pretty full of empirical investigation that made me change my mind wrt to how well LessWrong in particular did.

There's much more historiography to be done here, who believed what, when, what the long-term effects of COVID-19 are, which interventions did what, but this seems like the state of the art on "how well did LessWrong actually p... (read more)

Garrett Baker15h112

None of the evidence or framing here is particularly worse than what I would expect from a biased against LW search & retelling of the history, and that is what you're getting if you read Bouchard's writing, so this seems like effectively zero evidence.

It is also notable that they have nothing good to say about LessWrong's response. I think at minimum you have to credit the trading gains many LessWrongers made at the time, and MicroCovid.

That is not to say they are wrong about how early LessWrong was, just that it provides extremely little evidence. Be... (read more)

2Unnamed18h

Previous discussion

Cole Wyeth's Shortform

Cole Wyeth1d252

OpenAI claims 5.2 solved an open COLT problem with no assistance: https://openai.com/index/gpt-5-2-for-science-and-math/

This might be the first thing that meets my bar of autonomously having an original insight??

2interstice1d

Is this really that different from previous papers where a model provided a key step of the argument?

8Cole Wyeth1d

I mean, I think so. In those papers it's often not clear how "elicited" that key step was. The advantage of this example is that it very clearly claims the researchers made no contribution whatsoever, and the result still seems to settle a problem someone cares about! Only caveat is that it comes from OpenAI, who has a very strong incentive to drive the hype-cycle about their own models (but on the other hand, also has access to some of the best models which are not publicly available yet, which lends credibility).

Amalthea16h43

If you follow maths, one can be reasonably confident that the models can now sometimes solve math problems that are "not too hard in retrospect". I don't know how substantial this particular problem was supposed to be, but it feels like it tracks?

Zach Stein-Perlman's Shortform

Zach Stein-Perlman22d343

You suspect someone in your community is a bad actor. Kinds of reasons not to move against them:

You're uncertain
1. Especially if your uncertainty will likely be largely resolved soon
You lack legible evidence (or other ways of convincing others), and they're not already seen as sketchy
1. Especially if you'll likely get better legible evidence soon
They're doing some good stuff; you need them for some good stuff
They're popular, politically powerful, or have power to hurt you
1. And so you'd fail
  1. And maybe you'd get kicked out or lose power (especially if your move is un

... (read more)

Viliam16h64

More reasons:

2.b. The problem is not lack of legible evidence per se, but the fact that the other members of the group are too stupid to understand anything; from their perspective even quite obvious evidence is illegible.

7. If you attack them and fail, it will strengthen their position; and either the chance of failure or the bonus they would get is high enough to make the expected value of your attack negative.

For example, they may have prepared a narrative like "there is a conspiracy against our group that will soon try to divide us by bringing up unfounded accusations against people like me", so if your fail to convince the others, you will provide evidence for the narrative.

11RationalElf22d

Another is concern that the cure is worse than the disease. I.e. the drama and relationship damage caused by trying to expel them in the community might hurt the community more than removing them. Like there are scissor statements, there are also scissor people. You might be in a community where you don't think people will agree with you that they're a bad actor, even if you can establish the truth about what events occurred in the world, because there's a value disagreement between you and your community. Also concern about them and their well-being. Being publicly ostracized is very traumatizing and scary for most people. Particularly if they seem mentally fragile, you might fear the consequences for them or potentially for others who aren't just you if they're forced to endure a public ousting. You might fear or be averse to causing them pain. You might have sympathy for them, particularly if you think the sense in which they're a bad actor was in turn caused by something bad happening to them. You might fear that exposing their bad behavior will bring harm to others who are associated with them. For example, if they're part of some oppressed minority group and you fear that people will overgeneralize from their bad behavior to being mistrustful of or more prejudiced against others.

7Buck22d

I'd add that a common reason to choose not to act against someone is that many of those factors are combined. I think situations where it's (e.g.) purely "they have power to hurt you" or "you lack legible evidence" are much rarer than situations where it's an awkward combination of those with other things, and so it's hard to even know whether you should take on the project of acting against someone carefully and well.

xpostah's Shortform

[+]samuelshadrach22d-14-2

Misha Ramendik's Shortform

Misha Ramendik19h10

It seems from some testing that CorDA and the Muon optimizer form a synergy. I need to test this MUCH more before actually stating something, though.

Bart Bussmann's Shortform

Bart Bussmann21h30

On the origins of "you can just do things"

About once every 15 minutes, someone tweets "you can just do things". It seems like a rather powerful and empowering meme and I was curious where it came from, so I did some research into its origins. Although I'm not very satisfied with what I was able to reconstruct, here are some of the things that I found:

In 1995, Steve Jobs gives the following quote in an interview:

Life can be much broader, once you discover one simple fact, and that is that everything around you that you call life was made up by people that w

... (read more)

yrimon's Shortform

yrimon3d80

Gpt5.2 seems to have been trained specifically to be better at work tasks, especially long ones. It was also released early, according to articles about a "code red" in openAI. As such, (I predict) it should be a jump on the metr graph. It will be difficult to differentiate progress because it was trained to do well at long work tasks from the results of the early release and from any actual algorithms progress. (An example of algorithms progress would be a training method for using memory well - something not specific to eg programming tasks.)

2Baybar3d

What do you mean by "a jump on the metr graph"? Do you just mean better than GPT-5.1? Do you mean something more than that?

yrimon1d10

Here is the graph I'm talking about. Given that 5.1-codex max is already above the trend line, a jump would be a point outside the shaded area, that is bucking the de facto trend.

faul_sname's Shortform

faul_sname1d*80

[Fiction]

A novice needed a map of the northern mountain passes. He approached the temple cartographer.

"Draw me the northern passes," he said, "showing the paths, the fords, and the shelters."

The cartographer studied many sources and produced a map. The novice examined it carefully: the mountains were drawn, the paths clearly traced, fords and shelters marked in their proper notation. The distances seemed reasonable. The penmanship was excellent.

"This is good work," said the novice, and he led a merchant caravan into the mountains.

On the third night, they r... (read more)

Hastings1d40

Flagging this one as worth re-reading if you don't catch it. Took me three rounds (first was admittedly skimming)

Pavrati Jain's Shortform

Pavrati Jain2d20

Dopamine might be what regulates top-down, "will-imposing" action.
Stimulants are great for increasing attention, motivation and mood. However, they also cause downregulation of dopamine receptors, thus potentially causing dependence and the opposite of the benefits when not taking them.

Some lesser-known ways to upregulate the dopaminergic system without (or with less of) this effect:

Aspirin (upregulates TH - dopamine synthesis): https://pubmed.ncbi.nlm.nih.gov/30187283/
Creatine (potentiates dopaminergics somehow): https://pubmed.ncbi.nlm.

... (read more)

3Seth Herd2d

Yes, dopamine certainly plays an important role in "will imposing" actions (and will-distracting actions if you've got too much dopamine activity). I don't think there's a free lunch on downregulation as a result of upregulation. The brain has those negative feedback loops at many levels. (I'm told the rest of biology does too- it's a big part of how a messy genome makes something that works under a lot of conditions). What goes up, must come down. Artifrically changing how your brain works means it will adapt, and work more the opposite way when you're not doing that thing. But there can still be a big upside of choosing when your brain is in different states! And there's not a hard and fast zero-sum rule. If you upregulate hard, I'd guess that on average the downregulation adaptive response doesn't totally compensate for it. There's going to be a lot of important complexity, but that's a pretty strong general rule.

Pavrati Jain1d10

I agree that in general, downregulation is to be expected, but it doesn't always happen (depending on the specific receptor, affinity for their presynaptic counterpart, or biased agonism).

E.g.

the psychedelic DMT doesn't cause downregulation of the receptors it agonises (5-HT2A), because it doesn't target β-arrestin2, which is what supposedly leads to downregulation of this receptor.
my optimism about non-agonist (and non-dopamine-releasing) compounds affecting dopamines comes partially from the supposedly non-withdrawal-causing drug bromantane (which

... (read more)

Richard Ngo's Shortform

Richard_Ngo15dΩ256218

Someone on the EA forum asked why I've updated away from public outreach as a valuable strategy. My response:

I used to not actually believe in heavy-tailed impact. On some gut level I thought that early rationalists (and to a lesser extent EAs) had "gotten lucky" in being way more right than academic consensus about AI progress. I also implicitly believed that e.g. Thiel and Musk and so on kept getting lucky, because I didn't want to picture a world in which they were actually just skillful enough to keep succeeding (due to various psychological blockers).... (read more)

Showing 3 of 18 replies (Click to show all)

2Richard_Ngo2d

I'm not surprised by this, my sense is that it's usually young people and outsiders who pioneer new fields. Older people are just so much more shaped by existing paradigms, and also have so much more to lose, that it outweighs the benefits of their expertise and resources. Also 1993 to 2000 doesn't seem like that large a gap to me. Though I guess the thing I'm pointing at could also be summarized as "why hasn't someone created a new paradigm of AI safety in the last decade?" And one answer is that Paul and Chris and a few others created a half-paradigm of "ML safety", but it hasn't yet managed to show impressive enough results to fully take over. However, it did win on a memetic level amongst EAs in particular. The task at hand might then be understood as synthesizing the original "AI safety" with "ML safety". Or, to put it a bit more poetically, it's synthesizing the rationalist approach to aligning AGI with the empiricist approach to aligning AGI.

3Wei Dai1d

All of the fields that come to my mind (cryptography, theory of computation, algorithmic information theory, decision theory, game theory) were founded by much more established researchers. (But on reflection these all differ from AI safety by being fairly narrow and technical/mathematical, at least at their founding.) Which fields are you thinking of, that were founded by younger people and outsiders? Perplexity AI Pro (with GPT-5.1-Thinking)'s answer to "Who were the founders of academic cryptography research as a field and what where their jobs at the time?" There isn’t a single universally agreed-on “founder” of academic cryptography. Instead, a small group of researchers in the 1940s–1970s are usually credited with turning cryptography into an open, university-based research field. No single founder Histories of the subject generally describe a progression: Claude Shannon’s mathematical theory of secrecy in the 1940s, followed by the public‑key revolution of the 1970s and early 1980s that created today’s academic cryptography community. Shannon’s work was foundational, but it did not yet create an academic field in the modern sense; that came later with Whitfield Diffie, Martin Hellman, Ralph Merkle, and the inventors of RSA, whose work is often described as pioneering “modern” cryptography and has been recognized by ACM Turing Awards for cryptography pioneers.wikipedia+1 Early mathematical groundwork Claude Shannon is widely regarded as the founder of mathematical cryptography; in the 1940s he worked at Bell Labs as a researcher, where he developed the information‑theoretic framework for secrecy systems that later influenced public‑key cryptography. At roughly the same time and into the 1960s, cryptography research also existed in industry—most notably at IBM, where Horst Feistel headed an internal cryptography research group that designed ciphers such as Lucifer, which evolved into the Data Encryption Standard (DES), but this work was largely not yet

Cole Wyeth1dΩ120

Chaitin was quite young when he (co-)invented AIT.

faul_sname's Shortform

faul_sname2d45-6

RL capability gains might mostly come from better self-elicitation.

Ran across a paper NUDGING: Inference-time Alignment of LLMs via Guided Decoding. The authors took a base model and a post-trained model. They had the base model try to answer benchmark questions, found the positions where the base model was least certain, and replaced specifically those tokens with tokens from the post-trained model. The base model, so steered, performed surprisingly well on benchmarks. Surprisingly (to me at least), the tokens changed tended to be transitional phrases rat... (read more)

Caleb Biddulph2d81

The way they use the word "aligned" in that paper is very weird to me :P (they basically use it as a synonym for "instruction-following" or "post-trained").

But I feel like this method could actually be adapted for AI alignment/safety. It's kind of similar to my "incremental steering" idea, but instead of a strong untrusted model guiding a weak trusted model, there's a weak post-trained model guiding a strong base model. This also looks more practical than incremental steering, because it alternates between the weak model and the strong model, rather than g... (read more)

1anaguma2d

Cool result! Do you know they used Llama 2 instead of Llama 3? The paper was released recently.

Thomas Kwa's Shortform

Thomas Kwa2d*120

Superstable proteins: A team from Nanjing University just created a protein that's 5x stronger against unfolding than normal proteins and can withstand temperatures of 150C. The upshot from some analysis on X seems to be:

They used 3 different AI tools (RFdiffusion, ProteinMPNN, ESMFold/AlphaFold2), plus molecular dynamics software
The protein uses standard amino acids (edit: and was made in a standard bacterial cell)
The strength comes from increasing the number of backbone hydrogen bonds from 4 to 33.

So why is this relevant? It's basically the first step to... (read more)

16eniteris2d

It's not really that they made it have more hydrogen bonds, they made it longer and therefore more hydrogen bonds. It's like having a piece of velcro, and then using a bigger piece of velcro. Yes, the bigger velcro will be stronger. The AI was mostly used to design the scaffolding of the velcro. They didn't test whether it's biologically functional (though that's not really important if you only care about biomechanics). Though the tintin protein is already absurdly huge, and I'm pretty sure that the time it takes to translate it is longer than the division time of a(n average) cell. And these scientists made it even bigger (or rather, one domain of it even bigger and didn't make the rest of the protein).

2Thomas Kwa2d

Thanks, this is good context. So they didn't even simulate if it would remain biologically functional? Seems to make it less impressive.

eniteris2d30

Yeah the paper seems more like a material science paper than a biology paper. There was no test/simulations/discussion about biological function; similar to DNA computing/data storage, it's mostly interested in the properties of the material than how it interfaces with pre-existing biology.

They did optimize for foldability, and did successfully produce the folded protein in (standard bacterial) cells. So it can be produced by biological systems (at least briefly), and more complex proteins had lower yields.

Their application they looked at was hydrogels, and it seems to have improved performance there? But functioning in biological systems introduces more constraints.