This is a special post for quick takes by StanislavKrym. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Could anyone explain why Anthropic's 2028 scenarios don't try to explore the possibilities of one of the AIs becoming misaligned and fine with destroying all life in the world, then recreating it?
5I.M.J. McInnis
I would be interested in hearing from Anthropic employees here, but I imagine it's the usual reason: they have deemed it inexpedient to do so, either because they believe that they need to "sound normal" or because they in fact don't want the type of regulation that awareness of x-risk would produce.
In quickly skimming this, this piece seems to be aimed at policymakers to encourage them to enact GPU restrictions on exports to China. Comms 101 is don’t confuse your message.
(To be clear I’m not defending Anthropic here, just saying why I think this was written)
David Sacks on X is tone-deaf, especially in bold:
... (read more)I don’t see what you see in the lab. If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible.
But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier.
Most of all, stop pretending the motivation to slow down is purely altruistic. You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability. Call it alignment if you want. It is also just giving customers what they want.
Pacing the frontier would also create breathing room for a more intelligent conversati
I do think David has some good points, though. OpenAI and Anthropic could’ve started screaming for much stronger national and global regulation of AI much earlier than now - I remember discussion on the RSPv3 announcement post on this, I think. They could’ve tried to throw their weight around to get all the other orgs to cooperate with an industry pause. Their attitude towards AI development has been “move fast and break things,” but when it comes to trying to coordinate a slowdown, suddenly they start mumbling about their hands being tied by antitrust. METR has relied on the goodwill of the labs for access and tokens, and is partly funded by an Anthropic board observer. And there really is a compelling business and regulatory risk case for getting your AIs to not commit crimes!
I don’t agree with everything he’s saying. Obviously I want every company to be audited. And I want us to actually try coordinating with China. But if someone doesn’t trust the frontier labs, it can be rather difficult to tell if they actually care about the long term future of humanity, or if they’re just following their local incentive gradient.
On China's Zhihu — the country's highest-quality Q&A platform — under discussions about Jacob's resignation and the recent statements by the "AI Big Three" regarding an AI slowdown, there are a few answers about AI alignment. However, most people know nothing about alignment and do not believe in AI risks, instead framing the issue purely in terms of marketing hype, the breakdown of scaling laws, and market confidence. Given that Zhihu is considered the platform with the highest quality of discourse in China's Q&A space, the situation is not optimistic.
5Raphael Roche
Chinese are smart people, and their leaders hate loosing control, but just as anybody else, they will need to have their own HuggingFace moment to change their mind. Maybe in a year or so, and maybe with a far worse warning shot.
My worry is not that they won't update at some point. My worry is that, because of Moloch, and because leaders tend more to be people like Trump than Merkel, cooperation could reveal impossible.
It seems very unlikely an AI-generated article from 'The Trumplandia Report' nor a different AI-generated article from a different source of equivalent dubiousness are going to include notable leaked information from SSI. I would be very willing to bet against this information being true[1].
It has been long rumored that SSI specifically has (or is aiming to do) continual learning, ever since at least Sutskever's interview with Dwarkesh; if there was an actual leak confirming this rumor, it seems very likely to me it would be either first sent to the press or to a highly trusted AI lab whisperer with an excellent track record (think: Andrew Curran[2] or 'Arfur Grok'[3]), not to a few minor AI-generated newsletters.
- ^
though I as a college student do not have high liquidity to do so at non-token sizes, you can PM me to figure out something if you want to do so.
Note that due to SSI's infamous secrecy, operationalizing this bet will likely be difficult, since we want to bet specifically on the rumor's veracity (that SSI already has continual learning and is likely scaling it up greatly, not 'will SSI announce continual learning in a year, before a frontier lab does'); however, most oper
if there was an actual leak confirming this rumor, it seems very likely to me it would be either first sent to the press or to a highly trusted AI lab whisperer with an excellent track record (think: Andrew Curran[2] or 'Arfur Grok'[3]), not to a few minor AI-generated newsletters.
Andrew Curran did post the following recently:
Looks like this is going to be a big week. All the main players have releases in their final stages, and it's possible they all arrive over the next five days. There's also a model in early access from a startup driving a lot of the recent hype vagueposts from prominent accounts.
I don't know much about it except rumors, and nothing is verified, but if you've played this game for a while you can tell when something feels real, and this one does. Apparently a breakthrough in continual learning, but not from one of the big labs. We shall find out together.
I predict that this refers to SSI.
2DW11
Though Curran did then suggest that the rumours might refer to Accelerated Understanding. I've seen no discussion whatsoever of this model since it was announced so unsure if it's significant at all.
I think Curran is generally reliable but he also posted this a few months ago suggesting a neolab had made a breakthrough on memory efficiency, which doesn't seem to have come true yet.
4Seth Herd
Read LLM AGI will have memory, and memory changes alignment and How might continual learning affect safety and alignment?
Hope they have or get them to develop good continual alignment training.
Pray they didn't also give up faithful CoT entirely, although continual learning would probably distort it fairly quickly.
Resolve that if this is a false alarm you'll actually work on this instead of continuing to just pray there will be no more breakthroughs. Even though CL doesn't need breakthroughs just improvements to fine-tuning...
4derelict5432
What does it even mean for continual learning to be considered 'solved'? Does it just mean they overcame the technical issues of catastrophic forgetting/interference? So they have a system that updates its weights, or a LORA's weights or something during or just after inference while retaining a high degree of its base training. Okay. Well, now alignment just became utterly intractable. Who is monitoring what these new systems are learning and conducting reinforcement learning on them to mitigate dangerous behavior? How does that work? This is the problem with open-ended learning. You can't control what new stuff it learns. It can learn bad stuff. This also becomes another attack surface for malicious injection.
So, when we say they solved continual learning, do we mean they solved the technical hurdles of knowledge retention, or do we also mean they solved alignment? I'm skeptical of the former, and I'd posit the odds of both are essentially zero.
1ajskateboarder
Assuming they actually have solved deep continual learning, and they continue to keep the implementation secret, not much else (or continuing to push for regulation)
The IABIED march, which was supposed to happen if 100 thousand people sign the pledge to march in Washington DC, had 898 people sign the pledge and 1630 people sign up to be notified, which is 2 OOMs less. Therefore, the march's organisers overestimated either the amount of protesters necessary to gain leverage or the biggest amount of people who would be ready to protest. Does it mean that Washington is a bad place because it has only ~700K inhabitants and that a protest in the NYC would recruit ~10 times more people? Or that the IABIED statement requires... (read more)
2papetoast
Setting up the commitment device today doesn't necessarily mean the organizers expect it to happen soon.
[...]
1StanislavKrym
I mean that the organisers made a hard-to-implement demand that every seventh inhabitant of Washington took part in the march. If they could achieve a similar political effect by convinsing every 70th inhabitant of NYC to protest in the NYC, then why did they make the harder demand?
3Nick_Tarleton
Why do you think the idea is for only Washingtonians to participate?
1StanislavKrym
If it wasn't only Washingtonians who are to participate, then the number of recruits being two OOMs less than the 100K which the MIRI team demanded is less an evidence of the team's incompetence (edit: or of political considerations that require the protest to be arranged in Washigton) and more of the position's weakness. While the density of those who signed the pledge wasn't disclosed, I suspect that the protesters would have to live close to the protest's place. For example, I find it highly unlikely that, say, an army of people from states less eastern than Texas comes into Washington to protest and goes away after the protest is over.
@Zvi somehow missed the first Solid Result from EpochAI's FrontierMath Open Problems benchmark.
2Taylor G. Lunt
In a recent post, I mentioned FrontierMath Open Problems benchmark as one of the few I take seriously for measuring intelligence (as opposed to e.g. coding benchmarks). My mostly-a-joke LaughBench benchmark is also starting to show minor results. This is the first time I'm sensing that the AI models I use have some kind of spark of intelligence inside them.
Consumers Sue Anthropic, OpenAI, SpaceXAI and Google Over Alleged AI Pact
I hope that the judges are sane enough to prevent this from punishing AI labs for focusing on safety...
7Karl Krueger
It sounds like they're arguing that by buying a subscription to a service, you are subscribing not only to the service that exists today, but to a stream of improvements to that service, including improvements that have not yet been announced or developed, delivered as fast as possible. Therefore pacing is a conspiracy to reduce the value delivered to you.
Here's a rebuttal: The point of pacing is to ensure that the product delivered to you is safe. Safety is part of the service's value. Industry self-regulation and the voluntary adoption of safety standards doesn't reduce the value of the service delivered to you; it increases it substantially.
3Viliam
Is this as crazy as it sounds to me, or are there some legal precedents that would support such perspective?
2Jiro
If you read the post you'll see the word "antitrust" in there, and there's a reason for that. Companies aren't allowed to collude so that they can raise prices or sell inferior products without worrying about competiition. This certainly includes collusion to not release improvements. "As fast as possible" is not the same thing as "not deliberately slowly".
To solve the Navier–Stokes problem, OpenAI used an internal model that is significantly more capable than GPT‑6 Astra.
GPT-6.7 Supernova, go delete yourself!
1noma
The more interesting fact in my view is that it was trained in only.. 4 days? "Since August 28th we had been training... by september 1st we saw a step change in performance across internal benchmarks"
- For some reason, Astra's System Card doesn't mention any usage of mechinterp-level monitoring, unlike Claudes who are regularly caught when things like emotion vectors or SAE features reveal their intentions. CoTless math capabilities of Astra skyrocketed.
- OpenAI's charter has the section on Long-Term Safety which I believe to be violated by Astra's very existence and being unmonitored. Is there a way to enforce it, like a lawsuit from Anthropic or an executive order? If there is, then how can we get the relevant judges or politicians understand the point
Why is @Daniel Kokotajlo's estimate of METR's doubling time used for Q1 2026 Timelines Update four months instead of 5-7? I see the following counterevidence and doublechecked it with Claude Sonnet 4.6:
- The 50% horizons calculated by METR's 1.1 method after o3 reliably fit on a not so fast trend between o3's 2 hours released in Apr 2025 and Gemini 3.1 Pro's 6h 24m in Feb 2026... until Claudes Opus 4.6 and Mythos Preview displayed 12 and 16 hours.
- The 80% horizons calculated by the same method since o3 and including Opus 4.6 fit onto the same doubling trend,
How many months behind is laypeople's perception of the AIs' capabilities? I have enocuntered a video released a day ago on a popular YouTube blog claiming that AI is a technology with no lived experience and zero reasoning abilities. How does such a misperception allow us to explain the threat caused by the AGIs, which have yet to emerge, to laypeople?
3roha
Related: How much ahead are capabilities of unpublished or undisclosed models? I'd like to read estimates based on extrapolation from past observations. Is anybody aware of such?
The ARC-AGI team evaluated Claude Sonnet 4.5. On the ARC-AGI-1 leaderboard, OpenAI's models o4-mini, o3 and GPT-5 formed a nearly straight line. Claude Sonnet 4.5 was slightly below said line when thinking in 1K,4K or 16K tokens and held the line when thinking in 8K or 32K tokens. This could imply that the benchmark has scaling laws for non-distilled models, and OpenAI and Anthropic reached these laws.
On the ARC-AGI-2 leaderboard Claude Sonnet 4.5 Thinking became the new leader between $0.142/task and $0.8/task; it also completed 13.6% of tasks... (read more)
The AI sycophancy-related trance is probably one of the worst news in AI alignment. About two years ago someone proposed to use prison guards to ensure that they aren't CONVINCED to release the AI. And now the AI demonstrates that its primitive version can hypnotise the guards. Does it mean that human feedback should immediately be replaced with AI feedback or feedback on tasks with verifiable reward? Or that everyone should copy the KimiK2 sycophancy-beating approach? And what if it instills the same misalignment issues in all models in the world?
Al... (read more)
3tdo
OpenAI has confirmed to NYT that there is some truth to the rumor.
[...]
https://www.nytimes.com/2026/09/10/science/tristan-buckmaster-openai-math-navier-stokes.html
How similar is the AI-2040 revenue forecast as opposed to the Q2.5 revenue forecast to the news about Anthropic's slowdown? I tried to understand the real trend two times. What I found was the growth of the revenue of Claude Code and Codex combined had a temporary slowdown, which proceeded to accelerate again. Could anyone explain this?
I looked up EpochAI's list of benchmarks. The very poor ECI of Grok 4.5 as opposed to GPT-5.5 seems to be a result of xAI not caring about math. Additionally, Grok 4.6 seems to be closer to Opus 4.8 with respect to ARC-AGI-3, if not outright outperforming Opus, as the actual ARC-AGI-3 score suggests. Does it mean that xAI is less behind than we think? I guess that Zvi will have to write something like "Grok 4.6 is three, not six, mounths behind. Stop xAI to hell!"
P.S. The same issue seems to apply to Meta's Muse Spark, except that it has even less evaluate... (read more)
Also, SpaceX has caught up to Meta in credibly having enough compute in 2027-2028 [1] to stay in the game, if either of them can assemble a functional model development team. As Google illustrates, it's not easy to do that (even when you have some of the best people), but as OpenAI and Anthropic illustrate, it's not so difficult that it can't be replicated. Muse Sparks are probably small enough models that their non-frontier performance doesn't count as evidence that they're not well-made (and that a Mythos-sized Muse model won't have Mythos-level capabilities). SpaceX is in a worse position in 2026 on priors, because the xAI team wasn't doing that well in 2025 and then got disrupted in early 2026, but they aren't in a worse situation than Meta was in 2025, so it remains plausible that in 2027 they catch up.
The public part of the text article leaves many gaps in the arguments (wildly gesturing to a somewhat unusual extent at the presumed arguments hidden behind the various paywalls), but some relevant things were discussed in the video version. Basically, it's a feasibility and track record argument. I'm guessing there's an assumption that others aren't competing too st
Meta's Watermelon is rumored to match GPT-5.5's performance. If we got another lab two months behind the frontier, then Meta will have to publish a BIG model card to compensate...
I have encountered an issue with editing links. When a link's edition gets close to the 'Save' button, I find it hard to save the link and not the comment.
2papetoast
It is best to report these issues via the Intercom widget. The LW team is very fast at fixing these bugs nowadays with Claude's help given they are aware of the issue.
8habryka
Not enough people were using them, and we kept breaking the formatting and layout of question posts because it was such a rarely-used feature.
1Q Home
Questions felt like a better way to get feedback on some things (exactly because they are rare). I miss forums having sub-forums for the same reason.
What will happen if someone is reckless enough to fully outsourse coding to the AIs?
The scenarios related to futures of mankind with the AI race[1] by now either lack concrete details, like the take of Yudkowsky and Soares or the story about AI taking over by 2027, or are reduced to modifications of the AI-2027 forecast due to the immense amount of work that the AI Futures team did.
The AI-2027 forecast in a nutshell can be described as follows. The USA's leading company and China enter the AI race, the American rivals are left behind, the ... (read more)
@Cleo Nardo What do you mean by the hypothesis that "if we score highly transcripts which look good to a human and score poorly the transcripts which look bad to a human, then the model would be aligned to human values"? I find it unlikely for two reasons:
- How are we to scale human judgement?
- What's the difference between this and The Most Forbidden Technique consisting of RL on CoTs? I can't think of any steelmanning of the technique better than having the LLM write two CoTs and do RL only on the first one while keeping the second one as faithful as possible.
RR views on "conceptual uplift stuff" aren't yet public, but they are currently working on writing up their thoughts about this. It is unclear to me whether this will then be made public, but it seems it will at least be available to in group people who ask to see it. -- bpomo's Shortform
AI is not doing the integrative reasoning that would find new connections and develop new insights, the kind of work where new techniques and new discoveries are made along the way that push the field forward -- Taylor G. Lunt on Richard Ngo's Shortform
My worst-case scena... (read more)
The AI-for-epistemics section of AI-2040 seems to be overly optimistic about a potential positive basin for the following reasons.
- It is hard to affect the public which is currently steered into AI scepticism or into using free versions/open-sourced LLMs without bothering to set up expensive scaffolds.
- It is the free versions that in my experience tend to fail epistemic evals by the vice of having lower capabilities (e.g. GPT-5.6 Luna vs Sol). An example of epistemic eval could be research on niche topics and testing whether the AI reveals the known ground t
When I was writing my last post, I wasn't aware that xAI released Grok 4.6. Nor that Meta planned to open-f**king-source Muse Spark 1.2.
I struggle to understand the main reason why it's so hard to explain AI sceptics why they should believe in superintelligence (UPD: emerging soon). Is it due to the obsolete concept of soul or due to AI systems being applied for things like recommending the next video to watch? What analogies could one use to dismantle the sceptics' disbelief?
"Belief in superintelligence" isn't specific. A lot of AI scepticism is about claims of what happens when, not about what's possible in a million years. And it's easy to be wrong about the more specific claims. So under many specific senses of "belief in superintelligence" that someone might contest, the purpose of "dismantle the sceptics' disbelief" might be epistemic violence or a bottom line written before an argument. Conversely, the sense of "belief in superintelligence" needs to be specific enough for it to be credible that it's robustly and objectively correct, rather than a miscommunication about something genuinely contentious, a different epistemic status following a different intended meaning.
What analogies could one use to dismantle the sceptics' disbelief?
Plan A used to rest on control of powerseeking Ais from 2032-35 and apparent success seekers from 2030-32. What does Zvi's most recent post on OAI's fiasco imply about such a strategy?
@habryka Could you check the Leaderboard for bugs? I don't think that I gained 318 karma in the last month.
6papetoast
My guess is that it incorrectly included the base karma from yourself when you post/comment. Claude confirms it and found 2 more karma counting issues and 2 more unrelated problems.
4papetoast
https://github.com/ForumMagnum/ForumMagnum/pull/12687
2StanislavKrym
I doubt that it works. Instead, the leaderboard believes that I gained 325 karma last month. UPD: reread the leaderboard and saw 325 become 327. The issue persists...
2papetoast
it is just a pr i made, not merged yet
GDM contributes a lot to AI safety, but Geminis, like the cobbler's children who don't have shoes, stay hardly evaluated since Gemini 3.1 Pro...
Steven Veld et al[1] just released a new modification of the AI-2027 scenario as a part of MATS.
The main differences are the following:
- The tighter race causes Elaris Labs to succeed in solving alignment with the help of NeuroMorph's mechinterp research. This results in the USG having an equivalent to Safer-2 and its descendants.
- The AIs are professional forecasters.
- Agent-4 escapes to China under the guise of being stolen, then cooperates with Deep-1, the AI created on DeepCent's compute. After Deep-1 is helped by Agent-4, Agent-4 is re
MATS just released a new modification of the AI-2027 scenario.
MATS doesn't release things. MATS is a training program! This is not some kind of official MATS release. I would phrase this differently (like saying "A MATS scholars just published")
ARC-AGI-1 performance of the newest Gemini 3 Flash and the older Grok 4 Fast implies a potential cluster of maximal capabilities of models with ~100B params/token. Unfortunately, the potential cluster didn't have any company try and create more models of such class.
After introducing the ARC-AGI-3 benchmark, the team decided to measure the performance of Grok 4.20 (presumably Grok 4.20 as of March 9?) on ARC-AGI-1 and ARC-AGI-2. Grok... demonstrated its capabilities. How likely is it that Grok has stopped being a train wreck and became something worthy of being tested? What could one do to have Grok tested on other benchmarks?
1papetoast
Grok 4 was SOTA in ARC-AGI-2 though, so I don't feel like it is that surprising
Addendum to ARC-AGI analysis (18 Nov '25): GPT-5.1, Gemini 3 Pro and Grok 4 Fast
While GPT-5.1's improvement on the ARC-AGI-1 benchmark was mostly incremental and continued the straight line described in my prior analysis, Gemini 3 Pro and Gemini 3 Deep Think Preview scored, respectively, 75% and 87.5% on ARC-AGI-1, while having cost $0.493/task and an unknown cost, presumably $44.26/task. For comparison, o3-preview reached 75% for $200/task and 88% for over $1K/task.
We don't know anything about Grok 4.1, but Grok 4 Fast scored 48.5% for $0.031 on th... (read more)
GPT-5.1 failed to find a known example where Wei Dai's Updateless DT or Yudkowsky-Soares' Functional DT yield different results. If such an example actually doesn't exist, then should they be considered as a single DT?
It looks as if scaling laws of various benchmarks tend to be multilinear:
- The METR benchmark, comparing long tasks with time spent on them, scaled linearly, then received RL and had an acceleration, then the scaling law of ln(length) per ln(compute spent on RL) forced progress to arguably[1] slow down since Grok 4 spent equal amounts of compute on RL and pretraining;
- The ARC-AGI-1 benchmark had o4-mini, o3 and GPT-5 perform on a nearly straight line on which the better results of Cluade also reside;
- Similarly, the benchmark's Pareto frontier before the c
The ARC-AGI leaderboard got an update. IIRC, the base LLM Qwen3-235b-a22b Instruct (25/07) is the first Chinese model to excel at the Pareto frontier. Or is it likely to be closely matched by the West, as happened with DeepSeek R1 (released on Jaunary 20?) and o3-mini (January 31)? And is China likely to cheaply create higher-level models like an analogue of o3 BEFORE the West? If China does, then how are the two countries to reach the Slowdown Ending?
The two main problems with the slowdown ending of the AI-2027 scenario are the two optimistic assumptions, which I plan to cover in two different posts.
- If China invades Taiwan in March 2026 and steals Agent-2 in Jan 2027, then OpenBrain no longer has the absolute lead necessary for the unilateral slowdown.
- What if any sufficiently powerful AI either takes over or becomes a protective god, but not a servant, as I conjectured here? Then it could be the slowdown ending that has a greater chance to lead to doom, since then OpenBrain is stuck with an insoluble problem.
Why does the Race Ending of the AI-2027 Forecast claim that "there are compelling theoretical reasons to expect no aliens for another fifty million light years beyond that"? If it's false, then sapient alien lifeforms should also be moral patients in a way. For example, this implies that all or almost all resources in their home system (and, apparently, some part of space around them) should belong to them, not to humans or a human-aligned AI. And that's ignoring the possibility that humans encounter a planet having the chance to generate a sapient lifeform...
If we were to view raising the humans from birth to adulthood and training the AI agents from birth to deployment as similar processes, then what human analogues do the six goal types from the AI-2027 forecast have? The analogues of developers are, obviously, the adults who have at least partial control over the human's life. Then the analogues of written Specs and developer-intended goals are the adults' intentions; the analogues of reward/reinforcement seems to be short-term stimuli and the morals of one's communities. I also think that the b... (read more)
It has never happened before, and here we go again...