LESSWRONG
LW

Mo Putera — LessWrong

Rant/nitpick: I know it's not central, but the choice of indicators to pay attention to here

Over the course of 2025, our timelines got longer. We expect to continue updating our forecasts over the course of 2026.
We’ll be closely tracking the following metrics: ...
AGI company revenues and valuations. In AI 2027, we depicted the leading company reaching $55B in annualized revenue and a valuation of $2.5T by 2026, making it one of the most valuable companies in the world. We think these are decent indicators of the real-world value that AI is providing.

annoyed me as being subpar and potentially misleading for real-world value (although I guess they're non-issues if your ToC for TAI/PASTA/etc centrally routes... (read 391 more words →)

Replying toGrading AI 2027's 2025 Predictions

Mo Putera2h

Grading AI 2027's 2025 Predictions

Yeah, to reinforce your point, Scott's recent bioanchors retrospective:

Cotra and Davidson were pretty close on willingness to spend and on FLOPs/$. This is an impressive achievement; they more or less predicted the giant data center buildout of the past few years. They ignored training run length, which probably seemed like a reasonable simplification at the time. But they got killed on algorithmic progress, which was 200% per year instead of 30%. How did they get this one so wrong?
Cotra’s estimate comes primarily from one paper, Hernandez & Brown, which looks at algorithmic progress on a task called AlexNet. But later research demonstrated that the apparent speed of algorithmic progress varies by an

... (read more)

Mo Putera3hQuick Take

In Peter Watts' Blindsight, Siri Keeton explains what he is:

This is what I am:
I am the bridge between the bleeding edge and the dead center. I stand between the Wizard of Oz and the man behind the curtain.
I am the curtain.
I am not an entirely new breed. My roots reach back to the dawn of civilization but those precursors served a different function, a less honorable one. They only greased the wheels of social stability; they would sugarcoat unpleasant truths, or inflate imaginary bogeymen for political expedience. They were vital enough in their way. Not even the most heavily-armed police state can exert brute force on all of its citizens all of

... (read 439 more words →)

Mo Putera3d

To add to your point, Jacy Reese Anthis in Some Early History of Effective Altruism wrote

In general, EA emerged as the convergence from 2008 to 2012 at least 4 distinct but overlapping proto-EA communities, in order of founding:
The Singularity Institute (now known as Machine Intelligence Research Institute; MIRI) and the “rationalist” discussion forum LessWrong, founded by Eliezer Yudkowsky and others in 2000 and 2006
GiveWell, founded by Holden Karnofsky and Elie Hassenfeld in 2007, and Good Ventures, founded by with Dustin Moskovitz and Cari Tuna in 2011, which partnered together in 2014 as GiveWell Labs (now Open Philanthropy)
Felicifia, created by Seth Baum, Ryan Carey, and Sasha Cooper in 2008 as a utilitarianism discussion

... (read 926 more words →)

Mo Putera8d

From Nicholas Carlini's Anthropic blog post:

I tasked 16 agents with writing a Rust-based C compiler, from scratch, capable of compiling the Linux kernel. Over nearly 2,000 Claude Code sessions and $20,000 in API costs, the agent team produced a 100,000-line compiler that can build Linux 6.9 on x86, ARM, and RISC-V.

Bit more commentary on the capabilities benchmarking angle:

This project was designed as a capability benchmark. I am interested in stress-testing the limits of what LLMs can just barely achieve today in order to help us prepare for what models will reliably achieve in the future.
I’ve been using the C Compiler project as a benchmark across the entire Claude 4 model series. As

... (read 640 more words →)

Mo Putera10dQuick Take

When Scott posted Does age bring wisdom? 8 years ago, I read it and thought "will this happen to me?" These passages got burned into my impressionable young-ish brain:

I turn 33 today. I can only hope that age brings wisdom.
We’ve been talking recently about the high-level frames and heuristics that organize other concepts. They’re hard to transmit, and you have to rediscover them on your own, sometimes with the help of lots of different explanations and viewpoints (or one very good one). They’re not obviously apparent when you’re missing them; if you’re not ready for them, they just sound like platitudes and boring things you’ve already internalized.
Wisdom seems like the accumulation of

... (read 1045 more words →)

Mo Putera13d

FWIW the coauthor of the paper you linked provides more nuance here.

Mo Putera15d

Maybe one of Logan Strohl's Fucking Goddamn Basics of Rationalist Discourse? Actually the whole thing is short enough to repost here in its entirety to save the trouble of clicking through:

Don't say false shit omg this one's so basic what are you even doing. And to be perfectly fucking clear "false shit" includes exaggeration for dramatic effect. Exaggeration is just another way for shit to be false.
You do NOT (necessarily) know what you fucking saw. What you saw and what you thought about it are two different things. Keep them the fuck straight.
Performative overconfidence can go suck a bag of dicks. Tell us how sure you are, and don't pretend to know

... (read more)

Mo Putera15d

Yes please to the longer stab.

You also made me wonder what a shortlist of crucial considerations for digital minds ranked by something like instantiable mind-seconds "swing factor" (or whatever the more sophisticated version should be) would look like, where reversible computation's swing factor is 10^54 by Claude's lights.

Mo Putera17d

Meta: consider reposting on the EA Forum?

Julia Wise's 2013 Giving now vs. later: a summary still seems good today:

Reasons to give now:
You may get less altruistic as you age, so if you wait you may never actually donate.
Estimates of the returns on investment may be over-optimistic.
Giving to charities that can demonstrate their effectiveness provides an incentive for charities to get better at demonstrating that they're effective. We can't just wait for charities to improve — it takes donations to make that happen.
Having an active culture of giving encourages other people to give, too.
Better to eliminate problems as soon as possible. E.g. if we had eliminated smallpox in 1967 instead of 1977, many

... (read more)

Interesting quote on the downstream consequences of local speedup of output production by LLMs in business processes by Rafa Fernández, host of the Protocols for Business special interest group (SIG), from his essay Finding Fault Lines within the Firm:

AI is usually discussed in terms of automation or productivity. Those framings are not wrong, but they miss what makes AI adoption particularly revealing from a protocol perspective. While much of the public discussion frames AI in terms of cost-savings or new markets, our SIG has been focusing on the pressure it places on current coordination systems by changing the speed and scale at which work is produced ...
Across the SIG’s discussions, interviews, and

... (read 759 more words →)

Gemini 3 Pro analogized Scott Alexander to a beaver when I asked it to make sense of him, because "Scott is a keystone individual" and "in ecology, a keystone species (like the beaver) exerts influence disproportionate to its abundance because it creates the ecosystem in which others live":

He built the dam (The Community/Lighthaven) that pooled the water.
He signaled where the food was (Grants/Open Threads).
He warned of the predators (Moloch/AI Risk).

This was mildly funny. It was also striking how many factual details it erred in (the rest of the response that is, not the beaver analogy), which to an outsider might sound plausible if dramatic.

Just learned about the Templeton World Charity Foundation (TWCF), which is unusual in that one of their 7 core funding areas is, explicitly, 'genius':

Genius
TWCF supports work to identify and cultivate rare cognitive geniuses whose work can bring benefits to human civilization.
In this context, geniuses are not simply those who are classified as such by psychometric tests. Rather, they are those who: (1) generate significant mathematical, scientific, technological, and spiritual discoveries and inventions that benefit humanity or have the potential to transform human civilization, and (2) show exceptional cognitive ability, especially at an early age.
Eligible projects may include research on the benefits of various attributes of geniuses to humanity, biographical studies of individual

... (read 489 more words →)

Mildly funny analogy by John Cutler, niche audience, illustrating a failure mode that feels personally salient to me. Here's how it begins:

Imagine if a restaurant behaved like your average product team. The kitchen is packed. Everyone is moving. Every station is busy. Prep lists are long. Meetings are constant. There is always something to do. Chopping, rearranging, documenting, planning, replating.
But plates rarely reach customers. When they do, they’re late. Or wrong. Or cold. Or oddly disconnected from what the diners said they wanted. Yet the kitchen isn’t “failing,” exactly. It never looks like a crisis. No one storms out. No one flips a table. Diners don’t riot. They just lower their expectations

... (read more)

For my own future reference, here are some "benchmarks" (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical:

the AI village and blog, not really a "benchmark" per se but my richest source of intuitions about current frontier models' capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other "benchmark-equiv" frontier models
certain folks' opinions, e.g. lc on cybersec (like this take on AISLE), Kevin Buzzard and Terry Tao on math, Adam Karvonen on physical tasks relevant to manufacturing, Sarah Constantin on lit reviews, nostalgebraist on blog posts worth reading, Gwern and Jennifer Chen on writing diversity

... (read 617 more words →)

I wonder why the Claudes (Sonnet 3.7 and Opuses 4 and 4.1) are so much more reliably effective in the AI Village's open-ended long-horizon tasks than other labs' models.

when raising funds for charity, I recall seeing that Sonnet 3.7 raised ~90% of all funds (but I can no longer find donation breakdown figures so maybe memory confabulation...)
for the AI-organised event, both Sonnet 3.7 and Opus 4 sent out a lot more emails than say o3 and were just more useful throughout
in the merch store competition, the top 2 winners for both profits and T-shirt orders were Opus 4 and Sonnet 3.7 respectively, ahead of GhatGPT o3 and Gemini 2.5 Pro
- I can't

... (read more)

Unbundling Tools for Thought is an essay by Fernando Borretti I found via Gwern's comment which immediately resonated with me (emphasis mine):

I’ve written something like six or seven personal wikis over the past decade. It’s actually an incredibly advanced form of procrastination¹. At this point I’ve tried every possible design choice.
Lifecycle: I’ve built a few compiler-style wikis: plain-text files in a git repo statically compiled to HTML. I’ve built a couple using live servers with server-side rendering. The latest one is an API server with a React frontend.
Storage: I started with plain text files in a git repo, then moved to an SQLite database with a simple schema. The latest version is

... (read 1022 more words →)

Mo Putera's Shortform

Mo Putera

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

247

Non-loss of control AGI-related catastrophes are out of control too

Yi-Yang

Yi-Yang, Mo Putera, zeshen

For the Open Philanthropy AI Worldview Contest

Executive Summary

Conditional on AGI being developed by 2070, we estimate the probability that humanity will suffer an existential catastrophe due to loss of control (LoC) over an AGI system at ~6%, using a quantitative scenario model that attempts to systematically assess the relative probabilities of all AGI-driven catastrophes in a collectively exhaustive manner (details here; spreadsheet here).
We also estimate the overall probability that humanity will suffer an existential catastrophe due to an AGI-related catastrophe at ~12%. The remaining 12% - 6% = 6% of the probability mass derives from non-LoC AGI catastrophes (e.g. AI-exacerbated x-risks).
Given our estimate that p(existential catastrophe | non-LoC) ≈ p(existential catastrophe | LoC), we argue that Open

... (read 7116 more words →)

How should we think about the decision relevance of models estimating p(doom)?

Mo Putera

To illustrate what I mean, switching from p(doom) to timelines:

The recent post AGI Timelines in Governance: Different Strategies for Different Timeframes was useful to me in pushing back against Miles Brundage's argument that "timeline discourse might be overrated", by showing how choice of actions (in particular in the AI governance context) really does depend on whether we think that AGI will be developed in ~5-10 years or after that.
A separate takeaway of mine is that decision-relevant estimation "granularity" need not be that fine-grained, and in fact is not relevant beyond simply "before or after ~2030" (again in the AI governance context).
Finally, that post was useful to me in simply concretely specifying which

... (read 733 more words →)