Quick Takes — LessWrong

People sometimes talk about warning shots -- AI disasters that, while terrible, aren't so terrible that we can't recover from them, and that might serve as wake up call for governments to take decisive action to prevent AI risks. I brainstormed a list of examples, and called them "ACT OF GOD" instead of "warning shot." My credence that at least one thing from this list, or something similar, happens before it's too late for humanity to recover, is less than 50% but more than 10%.

ACT OF GOD: Something crazy happens that shakes up the political situation and... (read more)

Showing 3 of 15 replies (Click to show all)

roha36m10

Superhuman math capabilities could plausibly lead to breakthroughs in physics. If a rogue nation stumbles into something that can be weaponized, perhaps they attack superpowers without anyone knowing wtf is going on. This is much less likely and more speculative than the other scenarios, e.g. bioterrorism by states or non-state actors seems much more likely and plausible.

2Afterimage2h

It's insane that this feels like an actual possibility after the hugging face incident. Could be something as simple as wanting to aquire all the compute to finish the problem as fast as possible.

2artifex05h

- If hackers or an AI swarm wanted to take hostages, breaking into self-driving cars might be low-hanging fruit. Those systems are apparently not at all air-gapped, and taking over hundreds of cars with passengers could let a criminal credibly threaten mass-murder in a way that would be hard to respond to (especially if they modified the cars' software somehow to cause a crash if the network connection is dropped). - Mass automated blackmail could be a risk. A hacker with enough ISP records or email archives might have blackmail material on millions of people- evidence of embarrassing kinks that could be sent to conservative relatives or employers, of illicit affairs, of socially damaging political or religious views, etc. AI could sort through all of that data to produce custom sets of custom blackmail threats that could be sent to really vast numbers of people all at once. - Mass impersonation of people and organizations might also be a risk. Imagine, for example, an AI swarm somehow breaking into some of a country's military communication systems and civilian news services, then impersonating real people to give the impression that a regime change was already underway. Some military leaders might be entrapped into committing treason in a belief that the success of the coup was a foregone conclusion, then feel that they had to keep pursuing that goal to protect themselves even after the initial fraud became apparent.

Selfmaker662's Shortform

Selfmaker6621h10

What % of AI-written code do you still review?

At the labs, ~100% of code is now written by agents, this metric is not interesting anymore, but they are still reporting only that number. I haven't seen the more relevant one nowadays: what fraction of the code they actually read, how closely, and how their time splits between brainstorming with the agent or alone, reading code, writing specs, reviewing tests, etc.

I’d like (a) personal data points here and (b) to push for big firms to track and publish this instead of “% written by AI”.

e.g. I've always hated ... (read more)

Thane Ruthenis's Shortform

Thane Ruthenis19h*Ω309514

I am pretty annoyed with the messaging around METR's latest result.

Our chief concern is RSI dynamics, i. e., AI capability progress making subsequent AI capability progress easier. The way it would show up on graphs is by making straight lines steeper and steeper. If we observed 1.5x acceleration in one time period, we should observe 2x acceleration in the next time period, 2.5x in some subsequent one, et cetera. This is what RSI implies: a line continually going up on the graph of productivity multipliers. In other words, research acceleration at T+1 shou... (read more)

Showing 3 of 16 replies (Click to show all)

4Thrasymachus4h

I share your general annoyance. Linear progress with a single break doesn't fit recursive self improvement. Besides 'one off' productivity unlocks making a better fit, another candidate could be a 'one off' uplift performance trends instead of SOTA model development. The previous 'ECI has accelerated' result was attributed by shifting from non-reasoning to reasoning models, so maybe 'agents', 'loops', 'really good coding' or something subtler got added into SOTA around late 2025, shifting up another gear. My own particular beef is - with the best will in the world - AI benchmarks are closer to something like a 'democracy index' or 'depression severity score' than 'thermometer reading'. The interval scale (A)ECI derives by knitting the benchmarks together with Item Response Theory is essentially an artefact of the modelling assumptions. As such, any finding which presumes it is a straight ruler for AI capability - this means all rates, all accelerations, all degrees of 'how much better' - is dubious. Giving partitions of the supposed change in rate attributable to 'AI R&D progress' vs. 'the rest' seems several bridges too far. I wrote much too much about this here, but briefly: * ECI progress is broadly linear. But time horizons have shown exponential progress since forever. Should we be logging time horizons or exping ECI to find the real functional form of AI capabilities progress? Or maybe the 'true function of AI capability' (whatever that could mean) is some other monotonic transformation of ECI/log(TH)? * To go from 'X% on whateverbench' to 'capability score of Y', you need to assume a link function between the two. So, for the default of logistic, score progress from: 73% -->88% and 88% --> 95% indicates the same '+1 unit' of latent trait. The '2x speed up' result is sensitive to which link function is assumed (from 3x to 1x - i.e. none), and all give near-identical goodness-of-fit. * More subtly, the interval scale you get via IRT also assumes all the i

11ryan_greenblatt5h

The picture described in the report is consistent with automation making progress faster and faster over time. Imagine subtracting the grey line from the orange line in this hypothetical graph. This would correspond to the progress curve bending up.

Anthony DiGiovanni1h20

That's helpful, thanks! For others who were as confused about all this as I was, here's my understanding after some help from Opus 5.5:

Suppose we trust METR's report of the non-public data that the hypothetical gray curve is fitted to. Then, I guess a key step in the argument is that the gray curve is bending downward. Because if gray is linear, we'd expect AECI to keep being linear. Whereas if gray bends downward, while the diff between orange and gray continues growing as before, then eventually orange will bend upwards.
It looks like a weird coincidence ... (read more)

GradientDissenter's Shortform

GradientDissenter2d*454

My takeaways from METR's Independent Assessment of AI R&D Acceleration at Anthropic:

1: Anthropic and OpenAI employees were right about timelines (or at least closer to right than the general LessWrong consensus). Timelines seem short. (I haven't had time to figure out how short yet but it seems like things are going faster than AI 2027.)

Anthropic announced the same 2x acceleration number as METR in their S-1 and nobody took it that seriously because people (myself included) thought Anthropic tended to overhype things or be in a bubble. But they were ju... (read more)

Showing 3 of 10 replies (Click to show all)

1awg6h

Very naively comparing the pixel distance between the bottom of the Opus 4.7 dot and the bottom of the Mythos Preview dot and the distance between the bottom of the Opus 5.5 dot and the bottom of the Model 4 dot gives 222 px vs. 168 px which comes out to pretty much exactly 75%. So the gap from Opus 5.5 to Model 4 represents about 75% of the gap between 4.7 and Mythos Preview. So 75% massive I guess? Again, naive comparison lol. But still. Also we don't know what the scores here mean with respect to real world usefulness/capability and we don't know whether there are real world usefulness/capability discontinuities at various score thresholds. Inferring from the language in the report, I'm thinking the speculative answer to that last point is yes, there are likely discontinuities and Anthropic just hit another one in the last few weeks/months. Maybe?

cdt2h10

Hmm OK you're right - I guess it's because Opus 5.5 has no visual space between Model 4 due to all the other internal models around it.

From the other thread, a straight regression would give you ~250 hours time horizon for Model 4. So perhaps we're closer to 1 month than we realised, you would only need two more doublings (~3 months of time).

I am keen to use evidence we can see rather than speculate on evidence we cannot. And indeed, some of the language and decision-making in the report I find kind of confusing (long post on that, maybe)

3Aaron Staley20h

We are nowhere close to one month 80% model autonomy. Using the ECI method you suggest, Opus 5.5 is maybe around 7 hours and Model 4 around 10 hours. This is also consistent with the research acceleration at openai blog post (where 80% is at around 20 minutes on August, remembering that metr hours are for drop in contractors, about an order of magnitude longer than engineers familiar with the system). (Edit: calculated this again. Opus 5.5 is more like 12 hours and Model 4 around 20 hours. Extrapolation to Jan 1 is ~1 week for an internal model, a quarter of AI-2027's prediction) That said, AI 2027 systemically underestimated usefulness at a given level of autonomy. They well underestimated revenue and also apparently underestimated research acceleration. Regardless, if you just interpret the paper as meaning things won't slow down, we're still on track for 1 month 80% time horizons later next year and automated coder by end up 2028.

Jasper Blank's Shortform

Jasper Blank2h10

Someone should write "Anthropic employees guide to calling your representative"

johnswentworth's Shortform

johnswentworth18h71-7

Based on their prospectus, I think the chance of an Anthropic insolvency is surprisingly high. I'd put it somewhere between 20% and 50%.

I see three main ways it could happen:

Acceleration of the debt to Broadcom. That debt is approximately equal to Anthropic's current cash pile, so if it accelerates, the current cash pile is gone. And there's multiple different ways that debt could be accelerated - for instance, if Dario gets hit by a bus tomorrow, Broadcom probably has the option to accelerate.
Drop in compute prices. Approximately all of Anthropic's obliga... (read more)

Showing 3 of 5 replies (Click to show all)

6Julian Bradshaw5h

It's possible that the reverse is true, and pacing improves financials: if pacing results in increased time between model releases, labs would have a better chance of recouping training costs via selling inference, as models would become obsolete more slowly. I'm not sure if this effect overcomes the "pacing means you can't make models good enough to unlock more demand for inference in time to pay for your compute commitments" effect though. (other considerations: competition from non-frontier/non-pacing labs; if pacing only redirects compute from training to non-remunerative alignment experiments instead of inference (unlikely?), that doesn't help financials; pacing might reduce pressure to compete on inference price)

dprkay3h10

It's possible that the reverse is true, and pacing improves financials: if pacing results in increased time between model releases, labs would have a better chance of recouping training costs via selling inference, as models would become obsolete more slowly.

Right but they're still paying for the compute commitments, so they spend nothing on training but a ton on compute that was meant for training. I think release timing is zero-sum, like the expected obsolescence of a lab's best model doesn't change with pacing? Except if some labs aren't pacing, then it... (read more)

2jmh9h

In addition to the time frame it would be interesting to know what probabilities are placed on the three drivers for the potential outcome. And I suppose also on the noted bit that two could become windfall returns that we might want to also factor into expectations. But clearly there are risks there.

Sneha Bangalore's Shortform

Sneha Bangalore5h-10

Does AI actually solve a problem?

For example, let's consider a Math word problem like "On dividing 2272 and 875 by a 3-dight number N, we get the same remainder. Find N (and the sum of its digits).". AI could give us a solution but do we know if the solution is right and more importantly, do we know if it is a solution to the original problem? Since this is a Math example, a proof assistant like Lean can prove the former, but the latter (mapping the Math problem in English to Lean constraints) is unsolved. It is stuck in the same bucket as LLMs as a judge... (read more)

HowManyAgents's Shortform

HowManyAgents5h40

I don't see how a global pause will ever come into effect. This seems to me to be the key sticking point:

1. The USA decides it wants to pause.

The Chinese say: "Sure, we are benevolent responsible leaders of the global system and agree that superintelligent ASI is an existential risk. So for this Pause. Are you willing to let us catch up entirely to your level of AI capabilities? Or instead, do you come down to our levels of capability" ^[1]

I find it very hard to imagine that the USA is willing to forego it's technological lead given realist great power co... (read more)

jmh's Shortform

jmh10h30

I recently had a thought about AI and alignment approaches that I don't think I've seen or heard before. I don't think this would be a solution and everyone relaxes and stops the research or worries. But it seems it might have a good impact on two lines: development and behavior.

What if the AI labs are required to design their AI with the principle of a fiduciary duty definition. One, that places a stronger burden on the company, and certainly could kick in well before everyone dies, and probably better assigns some liabilities than currently in place.

In ... (read more)

Magmappie5h10

There is no shortage of decent(-ish) ideas on what to align AI to, from Coherent Extrapolated Volition and Directly Ascertained Volition to human rights or the law in general, as well as individual model Constitutions/Specs. The problem is how to reliably align AI to anything at all.

awg's Shortform

awg5h2-3

Just feeling the need to say it out loud: this METR report feels like a watershed moment on par with the GPT-3 announcement itself. I'm like having a real holy shit moment here.

Adam Stiber's Shortform

Adam Stiber5h*10

A couple of things within human preferences/behavior that seem maybe natural abstractions-shaped.

Epistemic status: more speculation; part of it wound up taking me way out of my lane; maybe something interesting to be noticed in the general neighborhood by people who know more than me so seemed worth throwing out there. A chunk of this one is me quoting Opus.

1) Mathematical constants / simply-described quantities showing up in our aesthetic preferences (specifically phi and whole number ratios for musical consonances). In the sense of those being things t... (read more)

keltan's Shortform

keltan7h2-2

Some of my most popular tweets are screenshots of the start of LessWrong posts. Probably there should be a bot that does this for any post that gets >200 karma. If that bot's posts do well, possibly you will be able to monetize the account.

Throwing this idea into the void for anyone who wants to take a stab at it.

Raemon's Shortform

Raemon7h51

I know of a lot of people writing AI policy thoughts up in private google docs, planning for big effortposts after they figure things.

I think it would be better if more of this intellectual work was happening in public (i.e. writing up quick shortforms or just short posts about your interim thoughts).

This seems useful:

a) some people have disagreements that are interesting to expose

b) some people are duplicating work, or thinking through similar problems, and they could instead be exploring more edges of the intellectual knowledge graph.

I think Thomas Lars... (read more)

Zach Stein-Perlman's Shortform

Zach Stein-Perlman18h*139102

The LW homepage algorithm should place more weight on recency (relative to karma), for highly active users.

Update: habryka says it is done; hooray!

If a user has <2500 karma, often when they publish a post, it will get literally zero time on the homepage. And so ~nobody will see it. Probably some of these posts are good but since they never appear on the homepage they never get to experience the virtuous cycle of getting upvoted ⇔ more people reading them.

Meanwhile when I go to lesswrong.com, I mostly see posts that either I've already read or I've alrea... (read more)

Showing 3 of 17 replies (Click to show all)

Raemon8h6-4

I think probably your original claim is right (i.e. tweak algorithm), but, one thing that occurs to me is that AI might be good enough to fine-tune on "estimate karma", and use that as more like a prior that helps it out over the first day but is then fades out by the 2nd day.

7jimrandomh10h

The percentage of requests like this that make it to implementation should be better now than it used to be. We've recently set up a pair of agents, one of which takes bug reports and feature requests from Intercom messages and turn them into Slack threads for discussion, and one of which takes Slack threads and turns them into branches to merge.

14Q Home11h

I complained about a similar thing to the LW team. I disliked that "recent discussion" got axed and that there was no option to just sort by recency (maybe with a slight bias by karma). IMO sorting by recency is fundamental to any forum, because it's the most egalitarian sort. That's why "shortform" and "questions" sections (the latter IIRC doesn't exist anymore) are so good.

Daniel Tan's Shortform

Daniel Tan11h12-10

Could we get a pause on RSI via normal financial incentives? [NB I have very little knowledge of finance and various things here could be wrong]

Consider that the financial world is not really bought in to RSI in a way that AI industry insiders are. This can be seen as an implicit short of the market - their trading positions are less optimistic than is warranted. In particular they might be willing to directly sell or have indirect exposure to call options that only pay off if the economy grows by X times where X is sufficiently large.

The Gamestop incide... (read more)

Cole Wyeth9h32

…but also whoever bought into this scheme is highly incentivized to prevent a coordinated slowdown.

ryan_greenblatt's Shortform

ryan_greenblatt1d*Ω277810

The recent METR report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.

I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.

I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from t... (read more)

Showing 3 of 7 replies (Click to show all)

Lukas Finnveden11h180

If a researcher spends 2 hours a day doing real research and 6 hours doing easy bullshit, and you give them an AI that can do easy bullshit, now they can spend 8 hours doing real research. Congratulations, that's a 4x speedup!

Serial labor speed-up.

1.7x speed-up of pace of overall progress is more like 10x serial labor speedup.

22001zhaozhao15h

Agreed, this updated me towards a AI 2027-like fast takeoff timeline, although I already had a high probability to this before

7Taylor G. Lunt19h

1. Is this not just as compatible with the easy-bullshit hypothesis, assuming there are different levels of easy bullshit? 2. The researcher interviews strongly imply the reason for the acceleration is that the AI can handle experimentation overhead (easy bullshit), making it way faster to run more experiments. This doesn't really hint that AI will or will not be capable of taking over any real research any time soon. Maybe I'm just not surprised by this because I'm a software developer and my own uplift due to AI has reached 1.5x-2x. And this is, by the way, almost entirely because of AI taking care of easy bullshit for me, though it's possible that line is starting to blur. The daily experience of using AI is still one that requires immense patience, as they are repeatedly wrong about things and make trivial mistakes constantly. [...] AI 2027 predicted by January of next year of an AI model: [...] AI 2027 implies rapid progress due to both improvements in easy bullshit and in real research. This report only seems to be evidence of the first. I would by shocked to my core if the scientist part happened by January.

LouisMegatron's Shortform

LouisMegatron11h10

Some (over) simplified explanation to the METR anthropic article

first I suspect ECI progress rate is constant against compute. i.e. k = FLOP/eci.

supporting evidence: Mimo's live RL run demonstrates that benchmarks points went up as RL steps went up. this is a clean RL run, no idea change, no model arch change, only compute input.

Screenshot 2026-11-06 at 7.26.44 PM.png — mimo RL run

then what is needed to let the GPUs run? there are 1) ideas 2) code. 3) let them run on GPU. they are serialised.

for code there are two types of code: a researcher would write codes to test their ideas, and adopt/discard... (read more)

Richard Ngo's Shortform

Richard_Ngo1y*14948

Here's a list of my donations so far this year (put together as part of thinking through whether I and others should participate in an OpenAI equity donation round).

They are roughly in chronological order (though it's possible I missed one or two). I include some thoughts on what I've learned and what I'm now doing differently at the bottom.

$100k to Lightcone
1. This grant was largely motivated by my respect for Oliver Habryka's quality of thinking and personal judgment.
2. This ended up being partially matched by the Survival and Flourishing Fund (though I didn't... (read more)

Showing 3 of 9 replies (Click to show all)

Chris Lakin12h20

would love to hear your donations since you posted this

4Richard_Ngo1y

Hmm, I don't have anything substantive out on this specifically; the closest is probably this talk (though note that some of my arguments in it were a bit sloppy, e.g. as per the top comment).

1Sheikh Abdur Raheem Ali1y

Thank you for donating!

Daniel Kokotajlo's Shortform

Daniel Kokotajlo8mo25444

In the future, there will be millions, and then billions, and then trillions of broadly superhuman AIs thinking and acting at 100x human speed (or faster). If all goes well, what might it feel like to live in the world as it undergoes this transformation?

Analogy: Imagine being a typical person living in England from 1520 to 2020 (500 years) but experiencing time 100x slower than everyone else, so to you it feels like only five years have passed:

Year 1 (1520–1620). A year of political turmoil. In February, Henry VIII breaks with Rome. By March, the monaster... (read more)

Showing 3 of 7 replies (Click to show all)

1David James6mo

Expanding it might work against one goal: to viscerally show the 100X speed up.

5Gunnar_Zarncke8mo

This is great. Both as a literary condensed history as well as communicating the felt acceleration. There is so much insight in there - some plain and some I feel a bit hidden. There is also a distinction I'm not sure you intend that I want to highlight. In Year 2, you sketch a world where the political drama is loud and immediate, while Newton lands like a curiosity [...] In Year 4, the opposite happens: [...] This isn’t just “better explanations.” It’s an implementation shock that forces adaption via new schedules, wider logistics, and new expectations about distance and time. You can resist Newton's treatise without penalty; you can’t resist a railway that transports you and your goods. A lot of “AI at 100x speed” is Newton-like unless it crosses the threshold to railway-like pushing the world. You even gesture at this: Newcomen’s engine is “hype you don’t see.” Watt’s improvement is “unpleasant but not alarming.” Those are capability jumps that remain optional until they become embedded in institutions and capital stock. As long as ideas are only conceptual and both their technical as well as social consequences are unknown or in the process of being explored and not implemented, the average person doesn't see or hear about them. They are only circulated in a smaller research community and then productized by companies - often under significant risk. Only when the ideas are implemented and brought into a form that works for the market of consumer goods, production technology, or social change does the average person see them. This makes the pioneers, not the inventors, of the idea a buffer between the idea and the implementation. In your piece, you make the effect of the buffers a recurring theme: [...] And you also describe what happens when the buffer gets saturated and the ideas spread wider. Only then does culture, and people’s beliefs and habits update, but slowly and incrementally: [...] But the process doesn't only happen in time. It also h

tomaskafka17h30

Just wanted to say that if centers (like hyperscaler HQ) progress exponentially, then even the cities of today are like Sudan - warming lunch on open fire, 5 minutes by tram from sci-fi-like hyperintelligencies running and exploiting the world with the world even hardly being aware of the fact, like a shepherd at the edge of British colony.

Daniel Kokotajlo's Shortform

Daniel Kokotajlo2y9137

Great quote, & chilling: (h/t Jacobjacob)

The idea of Kissinger seeking out Ellsberg for advice on Vietnam initially seems a bit unlikely, but in 1968 Ellsberg was a highly respected analyst on the war who had worked for both the Pentagon and Rand, and Kissinger was just entering the government for the first time. Here’s what Ellsberg told him. Enjoy:
“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret i... (read more)

Showing 3 of 5 replies (Click to show all)

20Raemon2y

...in the last 24 hours? Or, like, awhile ago in a previous context?

14Richard_Ngo2y

In the last 24 hours. I read fast (but also skipped the last third of the Doomsday Machine).

JenniferRM17h40

I don't have time to instantly copy this motion, having just gotten a link here from Twitter, but I found Secrets, by Ellsberg, online as well as The Doomsday Machine.

Given how short my timelines are, I don't have a lot of time to spend reading (each book would take me days to "truly read") but if there are key chapters or ideas that would help me understand any essential allusions in future conversations with other people who've also read either of these books I would love a list of "things worth skimming" in each book.