I find the METR task length measurements very helpful for reasoning about timelines. However, they also seem like approximately the last benchmark I want the labs to be optimizing: high agency over long timescales, and also specifically focused on tasks relevant to recursive self-improvement. I’m sure that the frontier labs can (and do) run similar internal evals, but raising the salience of this metric (as a comparison point between models from different companies) seems risky. What has METR said about this?

My preference ordering over approaches to surviving the next decade, assuming I had unlimited political capital, is (don't build ASI and):

1: Solve mind uploading

2: ... or at least intelligence amplification

(before proceeding, and then maybe do)

3: Imitation learning (with radically improved theoretical foundations)

4: ... or corrigibility / oracles

(and don't)

5: Try to build aligned (or controlled) autonomous ASI agents

Unfortunately 5 seems to be profitable, in the short term, so I guess that's what we're doing. In fact, the "plan" seems to be the exact reverse or so of my preference ordering on what it should be. 

Where is the hard evidence that LLMs are useful?

Has anyone seen convincing evidence of AI driving developer productivity or economic growth?

It seems I am only reading negative results about studies on applications.

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

https://www.lesswrong.com/posts/25JGNnT9Kg4aN5N5s/metr-research-update-algorithmic-vs-holistic-evaluation

And in terms of startup growth: 

https://www.lesswrong.com/posts/hxYiwSqmvxzCXuqty/generative-ai-is-not-causing-ycombinator-companies-to-grow

apparently wider economic measurements are not clear?

Also agency still seems very bad, about what I would have expected from decent scaffolding on top of GPT-3:

https://www.lesswrong.com/posts/89qhQH8eHsrZxveHp/claude-plays-whatever-it-wants
 

(Plus ongoing poor results on Pokémon, modern LLMs still can only win with elaborate task-specific scaffolding)

Though performance on the IMO seems impressive, the very few examples of mathematical discoveries by LLMs don’t seem (to me) to be increasing much in either frequency or quality, and so far are mostly of type “get a better lower bound by combinatorially trying stuff” which seems to advantage computers with... (read more)

I want an option to filter for writing with zero LLM influence.

I do not trust LLMs and I am not sure how I feel about LLM / human collaboration. As systems become more powerful, I am worried that they may become a serious epistemic hazard, up to and including actually hacking my brain. I would like to be able to protect myself from this aggressively. 

For that reason, I think that the current LW policy on LLM usage is insufficient. Every post that uses an LLM in any part of its production process whatsover should be flagged as such. Personally, I am currently willing to accept some LLM usage upstream of the writing I read and I would not routinely filter such posts out of my feed, but I would like the option to do so (which I would occasionally use as a sanity check) very aggressively and with no exceptions. Basically, an off-switch.

I would also like to be able to filter out any writing of which even a single word is LLM generated (except perhaps parenthetically). I think I would use this option routinely, though perhaps I would also like to exempt specific users (e.g. authors I have followed). But this softer option should allow consultation with LLMs, experiments with LLMs, etc. 

I consider it epistemic violence that I occasionally discover, after the fact, that an LLM was used extensively in the writing process of a post.

UPDATE: the blog post I referred to from Scott Aaronson has now been updated to reflect that A) the trick from GPT-5 “should have been obvious” but more importantly a human has already come up with a better trick (directly replacing the one from GPT-5) which resolves an open problem left in the paper. To me, this opens the possibility that GPT-5 may have net slowed down the overall research process (do we actually want papers to be written faster, but of possibly-lower quality?) though I would guess it was still a productivity booster. Still, I am less impressed with this result than I was at first. 

Awhile back, I claimed that LLMs had not produced original insights, resulting in this question: https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights

To be clear, I wasn’t talking about the kind of weak original insight like saying something technically novel and true but totally trivial or uninteresting (like, say, calculating the value of an expression that was maybe never calculated before but could be calculated with standard techniques). Obviously, this is kind of a blurred line, but I don’t think it’s an empty claim at all: an LLM could falsify ... (read more)

I wonder if the reason that polynomial time algorithms tend to be somewhat practical (not runtime n^100) is just that we aren’t smart enough to invent really necessarily complicated polynomial time algorithms.

Like, the obvious way to get n^100 is to nest 100 for loops. A problem which can only be solved in polynomial time by nesting 100 for loops (presumably doing logically distinct things that cannot be collapsed!) is a problem that I am not going to solve in polynomial time… 

A fun illustration of survivorship/selection bias is that nearly every time I find myself reading an older paper, I find it insightful, cogent, and clearly written.

Rationality (and other) heuristics I've actually found useful for getting stuff done, but unfortunately you probably won't:

1: Get it done quickly and soon. Every step of every process outside of yourself will take longer than expected, so the effective deadline is sooner than you might think. Also if you don't get it done soon you might forget (or forget some steps).

1(A): 1 is stupidly important. 

2: Do things that set off positive feedback loops. Aggressively try to avoid doing other things. I said aggressively.

2(A): Read a lot, but not too much.*

3: You are probably already making fairly reasonable choices over the action set you are considering.  It's easiest to fall short(er) of optimal behavior by failing to realize you have affordances. Discover affordances.

4: Eat.

(I think 3 is least strongly held)

*I'm describing how to get things done. Reading more has other benefits, for instance if you don't know the thing you want to get done yet, and its pleasant and self-actualizing. 

The primary optimization target for LLM companies/engineers seems to be making them seem smart to humans, particularly the nerds who seem prone to using them frequently. A lot of money and talent is being spent on this. It seems reasonable to expect that they are less smart than they seem to you, particularly if you are in the target category. This is a type of Goodharting. 

In fact, I am beginning to suspect that they aren't really good for anything except seeming smart, and most rationalists have totally fallen for it, for example Zvi insisting that anyone who is not using LLMs to multiply their productivity is not serious (this is a vibe not a direct quote but I think it's a fair representation of his writing over the last year). If I had to guess, LLMs have 0.99x'ed my productivity by occasionally convincing me to try to use them which is not quite paid for by very rarely fixing a bug in my code. The number is close to 1x because I don't use them much, not because they're almost useful. Lots of other people seem to have much worse ratios because LLMs act as a superstimulus for them (not primarily a productivity tool). 

Certainly this is an impressive technology, surpris... (read more)

OpenAI claims 5.2 solved an open COLT problem with no assistance: https://openai.com/index/gpt-5-2-for-science-and-math/

This might be the first thing that meets my bar of autonomously having an original insight??

Mathematics students are often annoyed that they have to worry about "bizarre or unnatural" counterexamples when proving things. For instance, differentiable functions without continuous derivative  are pretty weird. Particularly engineers tend to protest that these things will never occur in practice, because they don't show up physically. But these adversarial examples show up constantly in the practice of mathematics - when I am trying to prove (or calculate) something difficult, I will try to cram the situation into a shape that fits one of the theorems in my toolbox, and if those tools don't naturally apply I'll construct all kinds of bizarre situations along the way while changing perspective. In other words, bizarre adversarial examples are common in intermediate calculations - that's why you can't just safely forget about them when proving theorems. Your logic has to be totally sound as a matter of abstraction or interface design - otherwise someone will misuse it. 

“If you really believed X, you would do violence about it. Therefore, not X.”

I’ve seen this a few times with X := “AI extinction risk” (maybe fewer, seems like something I’d be prone to overestimate)

This argument is pretty infuriating because I do really believe X, but I’m obviously not doing violence about it. So it’s transparently false (to me) and the conversation is now about whether I’m being honest about my actual beliefs, which seems to undermine the purpose of having a conversation.

But it’s also kind of interesting, because it’s expressing a heuristic that seems valid - memes that incite “drastic” actions are dangerous, and activate a sort of epistemic immune system in the “uninfected.”

In this particular case, it’s an immune disorder. If you absorb the whole cluster of ideas, it actually doesn’t incite particularly drastic actions in most people, at least not to the point of unilateralist violence. But the epistemic immune system attacks it immediately because it resembles something more dangerous, and it is never absorbed. 
 

(And yes, I know that the argument is in fact invalid and there exist some X that justify violence, but I don’t think that’s really the crux)... (read more)

From soares and fallenstein “towards idealized decision theory”:

“If someone cannot formally state what it means to find the best decision in theory, then they are probably not ready to construct heuristics that attempt to find the best decision in practice.”

This statement seems rather questionable. I wonder if it is a load-bearing assumption.

I think that “ruggedness” and “elegance” are alternative strategies for dealing with adversity - basically tolerating versus preparing for problems. Both can be done more or less skillfully: low-skilled ruggedness is just being unprepared and constantly suffering, but the higher skilled version is to be strong, healthy, and conditioned enough to survive harsh circumstances without suffering. Low-skilled elegance is a waste of time (e.g. too much makeup but terrible skin) and high skilled elegance is… okay basically being ladylike and sophisticated. Yes I admit it this is mostly about gender.

Other examples: it’s rugged to have a very small number of high quality possessions you can easily throw in a backpack in under 20 minutes, including 3 outfits that cover all occasions. It’s elegant to travel with three suitcases containing everything you could possibly need to look and feel your best, including a both an ordinary and a sun umbrella.

I also think a lot of misunderstanding between genders results from these differing strategies, because to some extent they both work but are mutually exclusive. Elegant people may feel taken advantage of because everyone starts expecting them to do ... (read more)

Since this is mid-late 2025, we seem to be behind the aggressive AI 2027 schedule? The claims here are pretty weak, but if LLMs really don’t boost coding speed, this description still seems to be wrong.

[edit: okay actually it’s pretty much mid 2025 still, months don’t count from zero though probably they should because they’re mod 12]

It seems like we actually do not have a good name for the things that AI companies are building, weirdly enough.

LLMs? A hard-to-pronounce acronym for Large Language Models, which they were a couple of years ago, but now kind of aren't. 

Frontier Models? Then how do we talk about models which are not frontier? Also, "models" of what? Also, does this term include AI systems which reach the frontier by a very different paradigm?

Reasoning Models? Better (best on this list?), though it seems to refer specifically to the models for math / coding problems, that use e.g. chain-of-thoughts, not the agents.

Models? I think this is what we are settling on? Seems fine, but again, not a crisp semantic boundary. 

LLM Agents? Same problems as "LLMs," though it at least indicates that there exist later stages of training on top of language modeling.

AIs? Not specific.

AGI / ASI?  Disputed, probably not yet, and also not specific about the algorithms / technology.

Prosiac AI Systems? That is descriptive, but awful(ly wordy).

Shoggoths? Makes a specific claim about what's going on, also seems rather unlikely to catch on.

Transformers? We would probably like a term that also includes e.g. Mamb... (read more)

It looks like the market is with Kokotajlo on this one (apparently this post must be expanded to see the market). 

Particularly after my last post, I think my lesswrong writing has had bit too high of a confidence / effort ratio. Possibly I just know the norms of this site well enough lately that I don't feel as much pressure to write carefully. I think I'll limit my posting rate a bit while I figure this out.

Updates about LLM agency. 

The AI 2027 forecast for mid-2025 scores on SWE-bench was not correct:

For example, we think coding agents will move towards functioning like Devin. We forecast that mid-2025 agents will score 85% on SWEBench-Verified.

(From the footnotes here.)

As of December 2025, the SOTA is around 81% for Claude 4.5 Opus, so this threshold probably will not be passed until 2026. Still, it does not seem far off.

Also, GPT-5.1-Codex-Max has a longer task length than I expected (perhaps because it is specifically for coding? But it seems there a... (read more)

This doesn't seem to be reflected in the general opinion here, but it seems to me that LLM's are plateauing and possibly have already plateaued a year or so ago. Scores on various metrics continue to go up, but this tends to provide weak evidence because they're heavily gained and sometimes leak into the training data. Still, those numbers overall would tend to update me towards short timelines, even with their unreliability taken into account - however, this is outweighed by my personal experience with LLM's. I just don't find them useful for practically ... (read more)

It seems like the most impressive discoveries from AI still consistently come from narrow systems: https://deepmind.google/discover/blog/discovering-new-solutions-to-century-old-problems-in-fluid-dynamics/