This is a special post for quick takes by I.M.J. McInnis. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Not loving that the doom AI in IABIED was named Sable, and Anthropic just released Fable. Like, what else was being considered? The Germinator? PAL-9000?
AI company heads should sign the Superintelligence Statement (by their professed values)
Elon Musk, Demis Hassabis, and Dario Amodei have now all said that they would slow down if they could (or in Dario's case, "wish we had more time"). They should sign the Statement on Superintelligence---there is not much daylight between their casual remarks in public and what that statement proposes. Employees should pressure their leadership to sign this; we need to make common knowledge so that it's not so easy for them to say "but oh that will never happen."
I don't know how sincere they are about this desire. They sure don't talk about it much, and when they do, they are so insistent that coordination could never possibly happen, which is an awfully convenient belief. "It is difficult to get a man to understand something when his [billion$ and global status] depends upon his not understanding it."
Sam Altman and Mark Zuckerberg should sign it too, of course. As should everyone! But it seems much easier to get Musk, Hassabis, and Amodei to do so.
If you're reading this and can exert any pressure on them to do so (or generally promoting signing it within your company), please please do so.
Unfortunately people dislike seriously discussing (or calling for) hypotheticals they see as unlikely, which is a big problem for coordination to make them more likely. Instant-runoff (ranked-choice) voting is one device to extract preferences about hypotheticals perceived as unlikely, making them more likely. But that doesn't fit here, doesn't obviously make it more convenient to declare what seems better.
We call for a prohibition on the development of superintelligence ...
Another issue is calling for collective action without intending to engage in it unilaterally. This at least must be a more explicit part of the declaration (rather than just a reference to a "prohibition"), there must be a more visible conditional stating that any implied commitments only apply in the hypothetical where the world succeeds in coordinating on this. The memes/norms associated with hypocrisy create friction in this context, thwarting efforts towards coordination. Alternatively, it could be an expression of preference rather than a call for action, but then there's the above issue with eliciting preferences about perceived-unlikely hypotheticals.
I saw this message without context in my mail box and thought to write that this was an unsolved problem[1], that things that simply are not true can't stand up very well in a world model, but this seems like something an intelligent human like Amodei or Musk should be able to do. A 99% "probability" (guess by a human) on ¬ai_doom should not be able to fix enough detail to directly contradict reasoning on the counterlogical/counterfactual where doom instead happens. Any failure to carry out this reasoning task seems like a simple failure of reasoning in logic and EUM, not an encounter with a hard (unsolved) decision theory/counterlogical reasoning problem.
At a human level of intelligence, the level of trapped priors required to get yourself into an actual unsolved problem in the context of predicting future AI developments seems to be past the point where you would claim to have a good guess on the doom-causing AI's name and well on the way to describing the Vingean reflection process of the antepenultimate ASI on priors alone.
- ^
https://www.lesswrong.com/posts/wXbSAKu2AcohaK2Gt/udt-shows-that-decision-theory-is-more-puzzling-than-ever?commentId=xdWttBZThtkyKj9Ts "PIBBSS Final Report: Logically Updateless Decision Making" footnote 12
How much should we update toward short timelines and hard treaty enforcement because of Flapping Airplanes' continued fundraising?
Flapping Airplanes, a neolab led by twentysomethings with a pitch of "we'll invent AI algorithms that are many OOMs more sample-efficient" raised at a valuation of $1.5B in January. They are now in talks to raise at $5B, from both the $1.5B-round lead (Index Ventures, a prominent generalist firm) and a well-regarded deeptech firm (Lux Capital). (COI note: the latter invested in my company.)
A naive interpretation is: insiders have increased their estimation of FA's success (in creating vastly more sample-efficient AI) 3.3x. My best guess is that this would be justified (explicitly) by "they've hired more smart people" and "they claim technical progress," with perhaps a dash of "I would have invested earlier if they'd let me." Any technical progress is, from what Fable and I can find out, more or less secret. (Everything here is a mix of public info and reliable anonymous sources.)
Reasons to temper that naive interpretation
- "Recruitment of elite talent" can be a reason in itself, beyond increasing probability of success–––investors to whom I've spoken have insisted there's an "acquihire floor" not too far below [whatever most recent valuation at the time is]. That floor gets hire as you get a bigger and harder-to-replicate team.
- Prices in the VC market are far from rational. Most VC firms have sub-stock-market performance for super-stock-market risk and illiquidity. The high-performing VCs can best be modeled as brand rent extractors. Recently, Instinct, with a product still in private beta, raised a Series A at a valuation of $0.5B and then raised its series B at $2.5B. The investors at the second price were rushing to "get in," whatever the value. The investors at the first price were, mostly, better-connected. Not much changed about the company in those 3 weeks, other than its extensive public promotion by Series A investors. In short, VCs are not pricing, except in vague vibey terms, off (P(success) * E[payoff | success]).
- Flapping Airplanes is a particularly prominent example of prices decoupling from business fundamentals. They raised 4 rounds before the first publicly documented one, starting with a valuation of $40M. At each turn, there was a deal (in at least several cases explicit) that the main value add of the current VC was pumping the bag to the next buyer. It definitely helped before that all rounds were secret, but they could be doing it again! The founders are extraordinarily well-connected; their babysitter was Jeff Dean. One investor told me that he invested because the CEO was so good at fundraising. The element of adverse selection here can be outweighed by confidence that the fundraising ability will carry it to at least a "greater fool."
- Somewhat cutting against this greater-fool theory is the continued participation of the last round's investor (though we know not at what size–––and probably their participation is necessary, if they want this stake marked up).
- $5B is the 9Xth percentile privately-believed valuation, not the 50th percentile private valuation (as should roughly be true for large cap public stocks) because ≈nobody can sell stock.
- Though so was $1.5B. So 3.3x isn't the 9Xth percentile increase, but the increase of the 9Xth percentile. Since over time uncertainty should decrease, you might expect this to imply that the median increased by a greater factor, but I would disagree–––conditional on having a 3.3x increase in the 9Xth percentile outcome, uncertainty has probably increased.
- Still, if you posit some fraction of gullible people, then perhaps Flappy could bid up this high on a fugazi. The secrecy of their innovations helps: the investors evaluating this deal might be forbidden from sharing technical proof with anyone who could audit it for the smell of BS.
- "The category" (of AI lab) has gotten massively bid up, and revenue has taken off past expectations, which could lead investors to increase their estimates of the size of the prize.
- If it'll be so darn sample-efficient, you wouldn't need so many FLOPs, would you?
- This one's a little snarky, but in the extremes there's something to it. Surely when doing "sample-efficient AI research" the ratio of researchers to compute should be higher than elsewhere!
Things I could learn that would untemper this
- Credible (i.e. backed by technical understanding and minimal personal interests) claims that they have made meaningful technical progress toward sample-efficient AI.
- Existing shareholders broadly declining chances to sell in a secondary offering and/or buying more.
- The same thing happening at other neolabs, particularly ones that haven't had hype-raises.
How much should it be tempered?
An Mx increase in Flappy's odds of success implies an Nx increase in the odds of our getting OOMs-more-sample-efficient AI in the next (say) 10 years, with N < M. I think strongly that we should believe 1 < M < 3.3 and 1 < N, but I'm not sure what to set as M, let alone N.
Directional implications
From a brutish, scaling-oriented perspective: many OOMs of sample efficiency should (holding compute equal) catapult us forward in effective scaling. Moreover, it makes it more likely, later on, that a single "rogue actor" could develop comprehensively superhuman machine intelligence without a large GPU footprint, making treaties harder to enforce.
I see AI research right now as much more compute-bottlenecked than data-bottlenecked. Yes, I know they trade off against each other, but my impression–––correct me if I'm wrong!–––is that current LLMs are being trained with the optimal (for LLMs) quantity of data to minimize usage of compute. Yes, they're pouring lots of money into higher-quality data, but it's pennies next to the money spent on compute.
From this perspective, then, it has the same effect as some quantum of "algorithmic improvement." And depending on how the data and compute trade off against each other in this fancy new architecture (I haven't run the numbers on how e.g. a 1,000,000x sample-efficiency improvement would play out with Chinchilla laws, because I ought to be working right now, but someone should.)
From a more qualitative perspective: Sample efficiency is a commonly-cited difference between humans/AGI and current AIs! If AIs get continual learning, it will need to be sample-efficient to be truly transformative. This should not only pull forward timelines, but also shorten the gap between "the last AI humans built" and "the first comprehensively superhuman AI." For folks who are less alignment-doomer than I am may draw some encouragement from the second, depending on your theories.
So. What's it all mean? Am I just being distracted by a shiny object? Or is this a big deal?
A fun riddle I was shocked to see the gippities solve without extended thinking or much yapping, even. I gave up on the third!
Here's the riddle, as stated to Opus 4.6 on an empty context window. "Consider single-word-name countries. An inclusive pair is when the name of one country is contained in another wholly. There are three such pairs. Find them."
I was surprised not only by how quickly it solved it, but also by the lack of thinking tokens. Gemini 3.1 Fast also did it. And by the unusual order in which the solutions were produced, in the exact reverse order most people find them.
(Answer below)
.
.
.
.
.
.
.
.
PSA: don't draft longform things in the comment / quicktake window, if you're a fool who neglects to press save as draft.