This is a special post for quick takes by Adam Scholl. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Senator Bernie Sanders is planning to introduce legislation that would ban the construction of new AI data centers. You can find his video announcement here, and here is the transcript:
...Thanks very much for joining me. I will soon be introducing legislation calling for a moratorium on the construction of new data centers.
Now, as a result, I've been called a luddite, anti-innovation, anti-progress, pro-Chinese, among many other things. So why am I doing that? Why am I calling for a moratorium on the construction of new data centers?
Bottom line: We are at the beginning of the most profound technological revolution in world history. That's the truth. This is a revolution which will bring unimaginable changes to our world. This is a revolution which will impact our economy with massive job displacement. It will threaten our democratic institutions. It will impact our emotional well-being, and what it even means to be a human being. It will impact how we educate and raise our kids. It will impact the nature of warfare, something we are seeing right now in Iran.
Further, and frighteningly, some very knowledgeable people fear that that what was once seen as science fiction could soon beco
15
4
2
2
1
1
1It's the kind of action that when universalized does indeed end the AGI death race! That is in an important sense proposing an end to the AGI death race.
8
It's also the kind of action that's within the Overton window and if passed moves the window.
3
It slows it down a bit, perhaps.
If everyone stopped building datacenters you really have made a lot of progress towards stopping the death race (and of course, the algorithm that produces banning datacenter construction would probably not stop there).
I think this is a common misconception. I'm pretty sure algorithmic progress will eventually reach a point where what currently takes a datacenter will be possible on a single machine for a slightly longer training period. If that same algorithm runs on a datacenter it would produce something superhuman, but cutting down to only single-gpu training would then not be enough to completely stop. Algo progress is a slow slog of "grad student descent", so it likely takes quite a bit longer, and maybe it takes enough longer to figure out alignment. But it doesn't stop the death race, it just slows it down. Actual stopping would require shredding all silicon big enough to even run the fully trained AI, which doesn't seem to be in the cards. I'm not saying datacenter construction is good or should continue, or that this won't buy time, but I think people are wishful-thinking about how much time it buys.
Agree qualitatively (and possible quantitatively). However, there's a quite large knock-on effect, which is a strong bundle of signals of "AGI is bad, don't make AGI". These signals move in various directions between different entities, carrying various messages, but they generally push against AGI. (E.g. signaling legitimacy of the Stop position; the US signaling to other states; society signaling to would-be capabilities researchers; Congress self-signaling "we're trying to ban this whole thing and will continue to add patches to ban dangerous stuff"; etc.)
29
I mean, sure, eventually. The key question is how much of algorithmic progress is downstream of hardware scaling. My sense is around 50% of it, maybe a bit more, so that if you cut scaling, progress now happens at around 1/4th of the speed, which is of course huge and makes things a lot better.
Thinking this through step by step in the framework of the AI Futures Model:
First, I'll check what the model says, then I'll reconstruct the reasoning behind why it predicts that.
By default, with Daniel's parameters, Automated Coder (AC) happens in 2030 and ASI happens in 2031 1.33 years later.
If I stop experiment and training compute growth at the start of 2027, then the model predicts Automated Coder in 2039 rather than 2030. So 4x slower in calendar time (exactly matching habyrka's guess). It also looks to have well over a 5 year takeoff from AC to ASI as opposed to the default of 1.33 years.
I got this by plugging in this modified version of our time series to this unreleased branch of our website.
However, this is highly sensitive to the timing of the compute growth pause, because it's a shock to the flow rather than the stock. e.g. if I instead stop growth at the start of 2029 as in this worksheet, then AC happens in Mar 2031, taking ~2.2 years instead of ~1.2, so slowing things down by <2x. It does still slow down takeoff from AC to ASI to 4 years, so by ~3x (and this is probably at least a slight underestimate because we don't model hardware R&D automation).
Now I'll re...
13
My current default assumption is that, yes, someone will build something that obsoletes human intellectual and physical labor. In those futures, the best-case scenario is "Maybe the AIs like keeping humans as pets (and won't breed us the way we breed pugs)." [1] And the other alternative futures go sharply downhill from there.
So I think of delay in terms of survivor curves, like someone with terminal cancer. How much time can we buy the human race? Can the children alive today get to enjoy a decent lifetime for however long we all have? So I heavily favor delay, in much the same way that I'd favor cancer remission.
A global AI halt might even buy us quite a bit of time.
1. If I had to bet on a specific model as liking humans and being a responsible "pet owner", then I currently suspect we might have the best odds with a descendant of Claude. I do actually think that "enculturation" and building morally thoughtful models that like humans gives us non-zero chance of a more acceptable outcome. But I would still prefer humans to control their own destiny. ↩︎
2
Do you expect algorithmic progress to never hit diminishing returns?
1
"Eventually", sure, but I don't think that's operative here. If we had the ASI recipe and could study it safely for ten years, we'd find a way to implement it in a single datacenter. But discovering it in a single data center is much harder. There is actually something missing from current LLMs, there's a part of intelligence they just don't have, and the only thing that seems to mitigate that issue is model size, so without ever-increasing model size and analysis of their training dynamics, I think any attempts to get the missing piece are throwing darts with the lights off. (To be fair I have pretty unusual timelines compared to most of LW so maybe what's convincing to me shouldn't be to you.)
0
I agree with the general concern, but it'd be clearly a move in the right direction on that front?
With this kind of proposal I'm more worried that it could lead to a unilateral slowdown just after having animated China to be much more aggressive on AI.
4
agreed, I'm not saying don't do it. I'm replying to habryka saying it proposes an end to the race.
[...]
Doesn't the USA have enough datacenters to stay ahead of china in the race to fully human-replacing AI for quite a while, even with a unilateral hardware pause? I'm not sure of this claim, but it's currently my impression that the compute ratio is pretty dramatic.
I expect there's already enough silicon in place to produce overwhelmingly superhuman AI that still has idiot-savant going on; completely stopping CPU and GPU production worldwide seems to me like it'd be a timeline-extending move, but not by more than a few years. Somewhere between 1.3x and 3x.
2
As of June 2025 the US has 5x as much compute as China, I'd expect the gap has grown with substantially more American than Chinese data centers coming online in the past ~9 months
https://epoch.ai/data-insights/ai-supercomputers-performance-share-by-country
1
I dont see how. If AGI/ASI is powerful, then the existing deployed compute will suddenly become more powerful when AGI happens. If that isn't X risk, then a few more data centers won't change things. This only matter in long timelines where more data centers are required to get AGI. I don't think that is the case. I think copying the mammal neocortex etc will get us there and that doesn't need more compute.
This is cool! I'm sad he spends so much of his time criticising the good part (AI doing tonnes of productive labour). I say this not because I want to demand every ally agree with me on every point, but because I want to early disavow beliefs that political expediency might want me to endorse.
It seems to me a meaningfully open question whether automating all human labor will end up net benefiting humans, even assuming we survive; of course it might, but I think much more dystopian outcomes also seem plausible. Markets tend to benefit humans because the price signals we send tend to correlate with our relative needs, and hence with our welfare; I think it is not obvious that this correlation will persist once humans become unable to generate economic value.
1
What would AI labs do differently if this were made law? Couldn't they build datacenters outside the US?
1
While I'm glad that these things are starting to be seriously discussed in the open by those in positions of power, this sounds like a blanket ban on all new datacenters? This seems too broad to me. The ideal moratorium would be on large GPU clusters, or large AI specific infrastructure, right?
On another note, I'm somewhat concerned that banning large compute infrastructure will concentrate pressure on algorithmic progress if there is no "low effort" [1] outlet of just scaling the compute. Maybe regulating the rate at which large scale compute infrastructure is constructed will still allow researchers to be "intellectually lazy" [2] by offloading progress onto compute scaling instead of innovating on algorithms.
1. I don't actually think the researchers are intellectually lazy or that scaling is low effort, these just seem like the most succinct terms to express a lack of pressure to adapt in a specific direction. ↩︎
2. Same as above. ↩︎
2
I don't think people are currently being intellectually lazy. They might be rationally spending effort in ways that produce less insight per compute but faster insight per month than they would if they had less compute. I do think that despite the way compute limitation would make people try harder, things are still worse than they would be with less compute. But not as much worse as it naively seems, because of what you've mentioned here.
It seems the pro-Trump Polymarket whale may have had a real edge after all. Wall Street Journal reports (paywalled link, screenshot) that he’s a former professional trader, who commissioned his own polls from a major polling firm using an alternate methodology—the neighbor method, i.e. asking respondents who they expect their neighbors will vote for—he thought would be less biased by preference falsification.
I didn't bet against him, though I strongly considered it; feeling glad this morning that I didn't.
73
2I don't remember anyone proposing "maybe this trader has an edge", even though incentivising such people to trade is the mechanism by which prediction markets work. Certainly I didn't, and in retrospect it feels like a failure not to have had 'the multi-million dollar trader might be smart money' as a hypothesis at all.
12
Why do you focus on this particular guy? Tens of thousands of traders were cumulatively betting billions of dollars in this market. All of these traders faced the same incentives.
Note that it is not enough to assume that willingness to bet more money makes a trader worth paying more attention to. You need the stronger assumption that willingness to bet n times more than each of n traders makes the single trader worth paying more attention to than all the other traders combined. I haven’t thought much about this, but the assumption seems false to me.
7
Because I saw a few posts discussing his trades, vs none for anyone else's, which in turn is presumably because he moved the market by ten percentage points or so. I'm not arguing that this "should" make him so salient, but given that he was salient I stand by my sense of failure.
4
Mmh, if there is no reason to take that particular trader seriously, but just the mere fact that his trades were salient, I don’t see why one should experience any sense of failure whatsoever for not having paid more attention to him at the time.
Still, my main point was about the reasons for taking that particular trader seriously, not the sense of failure for not having done so, and it seems like there is no substantive disagreement there.
Knowing now that he had an edge, I feel like his execution strategy was suspect. The Polymarket prices went from 66c during the order back to 57c on the 5 days before the election. He could have extracted a bit more money from the market if he had forecasted the volume correctly and traded against it proportionally.
On one hand, I feel a bit skeptical that some dude outperformed approximately every other pollster and analyst by having a correct inside-view belief about how existing pollster were messing up, especially given that he won't share the surveys. On the other hand, this sort of result is straightforwardly predicted by Inadequate Equilibria, where an entire industry had the affordance to be arbitrarily deficient in what most people would think was their primary value-add, because they had no incentive to accuracy (skin in the game), and as soon as someone with an edge could make outsized returns on it (via real-money prediction markets), they outperformed all the experts.
On net I think I'm still <50% that he had a correct belief about the size of Trump's advantage that was justified by the evidence he had available to him, but even being directionally-correct would have been sufficient to get outsized returns a lot of the time, so at that point I'm quibbling with his bet sizing rather than the direction of the bet.
311Norvid on Twitter made the apt point that we will need to see the actual private data before we can really judge. Not unusual for lucky people to backrationalize their luck as a sure win.
48
I can proudly say that though I disparaged the guy in private, I not once put my money where my mouth was, which means outside observers can infer that all along I secretly agreed with his analysis of the situation.
2
I think it can be both rational to doubt his edge and not trade on it.
5
Yes. https://www.lesswrong.com/posts/tDkYdyJSqe3DddtK4/alexander-gietelink-oldenziel-s-shortform?commentId=JqDaYkRyw2WSAZLDg
I think it is probably possible in principle to train superintelligence on a laptop, and I worry that this inconvenient fact is often elided in discourse about halting AI. It is extremely helpful that for now, AI training is so absurdly inefficient that non-proliferation strategies roughly as light-touch as the IAEA—e.g., bans on AI data centers, or powerful GPUs—might suffice to seriously slow AI progress. And I think humanity would be foolish not to take advantage of this relatively cheap temporary opportunity to slow AI progress, so that we can buy as much time as we can to figure out how to improve our chances of surviving the creation of superintelligence. But I do think superintelligence is likely to be created eventually regardless, at least absent non-proliferation regimes drastically more costly/invasive than the IAEA; relatedly, I do expect that the long-term survival of life will still probably require solving the alignment problem eventually.
2
21I think the global treaty has additional hope, which further wants to be capitalized on, in that it strongly signals "we don't want to make AGI". This can ramify through society, academic institutions, parties with big-money philanthropists, student groups, etc. Quoting from here:
A professor doing cutting-edge domain-nonspecific AI research should read in the paper that this is very bad; then should have students stop signing up for classes and research; and have student protests; and should be shunned by colleagues; and should have administration pressure them to switch areas; and then they should get their government funding cut. It should feel like what happens if you announce "Hey everyone! I'm going to go work in advertising for a bunch of money, convincing teenagers to get addicted to cigarettes!", but more so.
Regarding
the long-term survival of life will still probably require solving the alignment problem eventually.
Does it? What if we just don't, and find other robust ways to prevent the creation of AGI, and other ways to have a very hopeworthy future?
18
Well, if there are robust ways to prevent the creation of AGI, that's a crux for me. But it seems pretty hard to do if it is possible in principle to train superintelligence on a laptop.
7
To expand a bit more:
* If you're going to safe the world by making aligned AGI, that works by then doing something with the AGI; whatever that action is, humans could also do it (though it could be much harder).
* Social pressure and social avoidance can be quite powerful. Like, imagine that every person knows for sure that if anyone knew that ze is thinking about how to make AGI, they would hate / shun zer. So now it's logistically very difficult / infeasible to talk with other people about your AGI ideas. That makes it way way way harder to make long term progress on the technical problem of making AGI.
* The motive to make AGI can be sapped, perhaps to a very large degree, by clarifying how dangerous it is and by offering hopeworthy alternatives.
* You could have technical surveillance, a la Critch (IIUC) as presented in his interview with Shapira. Sounds hard but could be feasible.
2
I think it is conceivable that humanity could learn to somehow screen enough of the computations occuring on each laptop, enough of the thoughts occuring in each brain, etc., for danger in a way that is not obviously more dystopian/rights-violating than the IAEA. I do not however currently consider it obvious that this is possible, nor that humanity will try to (and succeed at) figuring out how in time, and on my models this is roughly the degree of screening that seems likely necessary to prevent the creation of superintelligence for centuries, rather than years or low-decades.
3
Well I agree it's not super obvious? I mean I'm not claiming that I have a bunch of inferences that you don't, or that I'm confident it's doable. But like, society can make an intellectual project way way way harder just by being hostile toward it.
7
I vaguely agree, but also I think it's not so implausible to get a pretty robust stop. I don't have a specific way, but think for example how, for now, it's hard to make a global pandemic from your garage, even though that's probably feasible in principle. Some combination of, no one would do that, and if they would, no one would help them, and without help it would just be hard. I mean, that's not very reassuring; largely I'm like "IDK, humanity (including HIA people) might figure it out!".
6
I think it is far easier to mistakenly/motivatedly conclude that actually superintelligence is a grand thing to aim at, and that the risks are minimal etc., than it is for creating global pandemics. As such I think the latter requires people who are deluded and/or evil to a quite unusual degree, whereas the former strikes me as requiring only people deluded and/or evil to an exceedingly common degree. Which makes me more worried that social pressure will not suffice to prevent it.
7
I kinda agree, and in general I definitely agree it's probably much harder than in the case of pandemics. Partly I think this is something we can change (by explaining risks and offering alternatives). On the other hand, as Critch points out, we do have quite strong and functionally operative intuitions around cutting down the tall poppies. In many cases this is a bad intuition, but it's not 100% wholly without merit in general, and in this case it would fit well.
2
There's a possibility that even in (or aiming at) an AGI-less very hopeworthy future, we'd need to figure out a bunch of stuff that would be contained in The Textbook From The Future, e.g., regarding how to ensure that strong intelligence amplification (or whatever other strong intentional changes to how the human mind works) doesn't change humans' values in undesirable ways, or how to create a governance system that actually robustly respects and nourishes the agency of its constituents, while also preventing that agency from overflowing catastrophically.
6
I qualitatively agree, but maybe quantitatively disagree? Not sure what "a bunch" means in relative terms. I'd say the lion's share isn't necessary to solve in human(e)-only futures. You just get so many advantages with human(e) minds vs. alien minds. But yeah, I think figuring out good governance is important for building a hopeworthy future, especially for building one that is legibly so.
9
I broadly agree, but I think it's important to specify what exactly one means by
[...]
To give an analogy, is it possible to create [a strain of Streptococcus with some specific property] in one's garage laboratory? Yeah, maybe, but you need many inputs, and one of those inputs is the initial Streptococcus strain.
I don't know whether this applies to you, but: it seems to me that when someone claims that in principle it should be possible to train a superintelligence on a laptop, they roughly base this on the fact that the learning algorithms of the human brain are much more efficient than those for which people need to keep building bigger and bigger datacenters and keep feeding them more and more data. And this is true, but a big reason why human brain algorithms are so efficient is that evolution did a lot of precomputation beforehand, so a large part of the learning is already done, pre-computed in the genome.
To get to the point: it's very unobvious to me that there's a way to go from randomly initialized weights to ASI using laptop-level computing power. But I think it is plausible that one can gather various pre-computed pieces together on one's laptop and apply a few weeks to months of training to get to a superintelligence.
Very rough BOTEC, corrections welcome:
A thing to keep in mind:
- laptop: 10^13 flops or something?
- future tricked out laptop: 10^15 flops or something?
- current big AI training cluster: 10^20 flops or something?
- future big AI training cluster: 10^23 flops or something?
That's a sizable gap but not all that crazy in the grand scheme of things? Intuitively one would have to have a pretty detailed (and accurate) understanding of things to (calibratedly) predict "yes train on datacenter, no train on laptop". Does that make any sense?
3
I mean, I'm not super confident. I think "it's very unobvious to me that there's a way ..." indicated mild skepticism / request for more reasons to think why it should be possible.
Are you saying that it's plausible that those 8 OoMs can be bridged by various algorithmic tricks or something?
5
I wasn't directly contradicting your statement, I think. I meant to say "If you're confident you can do it with a datacenter, then you should probably mostly guess that you can do it on a laptop, unless you have some pretty strong specific model here.". IDK whether you're confident you can do it with a datacenter.
I do think it's very likely you can do it on a laptop, though it's a bit hard to express why, and it's reasonable to treat it as unobvious. I think it's coming from various intuitions like "the info density of the human brain isn't crazy high" and "the effective flops needed for much of what human brains do day to day isn't that high" and "well, you could do a bunch of clever caching and just-in-time compressing/uncompressing and reordering of various parallel-ish computations, and thereby fit things into tiny amounts of data". As another intuition pump, consider the demoscene: https://en.wikipedia.org/wiki/Demoscene where they put impressive visual displays in very small amounts of data (though IDK how much compute is involved) https://www.youtube.com/watch?v=R-4wHUw_OdE&list=PLMuQbRD9kQr5hsbu9uyFSBA7kOrowz6Xo
6
I disagree for reasons discussed here.
6
I strongly agree. The whole focus on banning AI data centers seems somewhat myopic & misguided
8
I think the case for banning AI data centers seems strong—it would likely buy us additional years, perhaps even decades, to solve the problem! I expect this would greatly increase our odds of survival/flourishing, even if such bans seem unlikely to prevent the creation of superintelligence indefinitely.
8
if you’re a few years from the end of the game, myopia may not be a mistake?
5
I think this either stretches the definition of 'train' to a point where your training algorithm's priors on superintelligence would themselves constitute superintelligence[1] or stretches the definition of 'laptop' beyond the reasonable meaning of the word[2].
Assuming that there's any relation at all between what we know now and what form superintelligence is likely to take, RLVR seems like the best way to get there, and that entails a lot of quite expensive rollouts followed by some very expensive gradient passes.
1. ^
(To obtain significantly better efficiency than the RLVR-on-very-hard-cognitive-tasks training we're doing now, given that supervised learning is right out due to lack of superintelligent sample data)
2. ^
A computer powerful enough to run this kind of training at scale doesn't have any other outstanding practical use.
5
It is true that it does require doing so "eventually", but like, we would be talking about 100+ years here. If Moore's law threatens to get into the territory where the only choice you have is total surveillance state or continuing Moore's law then... you just stop Moore's law. We can just like, not continue to make ever more efficient transistors and computer and AI algorithms.
If you notice that in a few years algorithmic progress is going to be so widely distributed that you can start developing extremely dangerous systems on consumer hardware then... you just don't do the algorithmic progress research.
If you notice that in a few decades individuals can cause enormous harm using their consumer machines, then you use those few decades to build verified computing hardware that is privacy-preserving while still preventing you from running dangerous code on it.
I don't understand this worldview where for some reason you can't use technological progress to make it harder to unilaterally deploy dangerous AI systems. There are technological solutions to preventing people from doing dangerous stuff with their computers without requiring a total surveillance state.
I’m not sure what you’re getting at with all your “just”s. Like, it doesn’t seem like we can “just” get a data centre ban. Why would these other bans be easier? probably you don’t mean that they would be, but I’m confused what you do mean.
Similarly, I don’t understand which worldview “where you can’t use technological progress to make it harder to unilaterally deploy AI” you’re talking about. In particular, I don’t see such a worldview expressed in the comment you’re replying to. I’d guess you think it’s a consequence of the “drastically more costly/invasive” qualifier, but the connection is a little remote for me to follow.
12
Huh, I thought my reasoning was relatively clear. To expand, the OP says:
[...]
and:
[...]
I am proposing a set of non-proliferation regimes that are not "drastically more invasive" than the IAEA (the matter of "costliness" feels confusing given that there is an unclear counterfactual, so I'll footnote that instead[1]).
The "just" here means "you can just do X using non-proliferation regimes that are not drastically more invasive than the IAEA". For example the IAEA has heavily curtailed research into how to build nuclear weapons more cheaply and efficiently, which seems like it applies pretty straightforwardly to algorithmic progress. A simple code-signing regime where high-performance chips are limited to only run signed code seems like it would also not be "drastically more invasive than the IAEA". Of course the IAEA has drastically curtailed buildout of nuclear energy and weapons production, and indeed nuclear energy has been getting more expensive per watt over time, which seems like it implies a reversal of Moore's law would also be less invasive than the IAEA.
I am not saying this means it will be easy for decision-makers to get buy-in for any of these policies, but in as much as we are evaluating policies on the dimension of "how privacy/control invasive do these policies have to be in order to achieve their goals of non-proliferation even over the course of decades", I think there are many entries in that set of policies that are less invasive than what we've done with nuclear reactors and nuclear weapons that we could "just" implement.
Of course this is ignoring most of the political tradeoffs for many of these policies, and I am quite uncertain which things will actually end up in the political Overton window, and be feasible. But I am pretty sure that "we will have no options that do not involve very serious violation of civil liberties for getting this done" will not actually be the reason this fails, which I feel like the OP is pretty clearly ar
For example the IAEA has heavily curtailed research into how to build nuclear weapons more cheaply and efficiently, which seems like it applies pretty straightforwardly to algorithmic progress.
IIUC, it’s legal everywhere on Earth to do basic research that might eventually lead to a new, much more inexpensive and hard-to-monitor method to enrich uranium to weapons grade.
I’m thinking mainly of laser isotope enrichment, which was first explored in the 1970s. No super-inexpensive method has turned up, thankfully. (The best-known approach seems to be in the same ballpark as gas centrifuges in terms of cost, specialty parts etc., or if anything somewhat worse. Definitely not radically simpler and cheaper.) But I think there’s a big space of possible techniques, and meanwhile people in academia keep inventing new types of lasers and new optical excitation and separation paradigms. I don’t think there’s any general impossibility proof that kg-scale uranium enrichment in a random basement with only widely-available parts can’t ever get invented someday by this line of research.
(If it did, it probably wouldn’t be the death of nonproliferation because you can still try to monitor and control ...
14
Yep, that's roughly my model. So the equivalent here would be that if you are doing related work in computer science or whatever, nobody would show up at your door, but if you tried to release a paper being like "we improved training performance by 2x using this fancy convolution" the authorities would show up at your door.
I think this is sufficient to prevent most large-scale investment into this kind of research (at least within a nation, internationally this all is a bit trickier and my models are more uncertain).
4
Seems plausible. At a guess, the effect of that would be to barely or at best somewhat slow down the actual core research leading to AGI, while greatly slowing down any visible impacts and last mile research. (I'm unsure because IDK how much of the current influx of resources gets directed to [real capabilities research, as defined presuming my mainline model where learning programs as of 2026 aren't all that relevant to AGI], and I'm unsure what the effect on research is of current big labs blundering around and showing more clearly what limits are of current learning programs.)
Ot1h, you would get more warning of AGI coming in the sense that people who deeply understand the related fields (e.g. let's say, vaguely, cognitive science, "technical philosophy/epistemology", algorithms) might be able to call that we're actually getting most of the relevant understanding. Ototh, you would get less warning in the sense that you wouldn't be getting big new coalescences of economic productivity or scary demos.
Cf. Red vs. Blue here: https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts#Red_vs__Blue_AGI_capabilities_research
4
I certainly think your world model in which learning programs as of 2026 are not very relevant to AGI should predict different things here! I do think that's likely the underlying crux.
9
I assume very few people are interested in doing independent research into improving nuclear weapons. If institutional AI algorithmic research were effectively banned, all else equal, I assume many more people would be interested in independently researching it, which would require more in-practice restriction on speech to curtail. (Based on your tweets I'm guessing you think that curtailing independent research wouldn't be necessary and aren't considering it here; but this may be a background disagreement with people saying invasive restrictions would be needed.)
[...]
Controlling widely-used hardware and only allowing approved code to run on it does seem drastically more invasive, sufficiently obviously so that I have no idea where you're coming from here. If this only applied to the largest supercomputers I might not call it more invasive, but the whole premise of this thread is not-that.
3
Yeah, my current guess (with like 85% confidence?) is that if you curtail large industrial-scale investment you would get there. I don't think independent research would get there any time soon. You would need some kind of industrial level of investment at some point in the coming decades.
I agree there is more motivation to build more efficient AI algorithms than for people to build more effective nuclear reactor or nuclear weapon designs, though I think a lot of that is downstream of a societal stigma for nuclear weapons, on which I think there is a lot of uncertainty on how it will develop in the case of AI.
A proper treatment of this topic would start from the top about the role of social stigma and how this will affect talent allocation and resource investment, and how it interfaces with regulations. I think the default outcome here is that there won't be much of a social stigma around AI development, this will consistently keep investment into developing more competent AI systems high, and also be itself the force that prevents regulation from interfering with that investment.
In the worlds where you actually have widespread buy-in to do something IAEA-like, you probably have quite a bit of stigma, and this will be largely responsible for keeping the level of investment low (though the regulations itself will also help). And if you manage to keep it that way, then I think you are probably fine for a very long time. Of course the stigma might break at any point, similar to how nuclear agreements might break at any point, and then you are back on a clock (and the clock will of course be shorter because you will have made at least some relevant progress in other scientific fields). All my statements here are about how if you keep the stigma and regulations up, you will be fine. In most worlds the stigma and regulations will break at some point, and you will have much less time, but this isn't because the alternative would be some kind of Orwellian or invasive
4
It sounds like I have more expectation of a much more efficient paradigm (a la e.g. Steven Byrnes) being feasibly discovered through purely theoretical work (though not necessarily single-2026-laptop efficient, or discovered on any particular schedule), which is coloring my takes here.
I agree that stigma is important and would reduce the level of intervention needed to shut down independent research. It's only very recently that I've seen any discussion of stigma as load-bearing in pause scenarios, so I wasn't thinking of it.
[...]
I was thinking of it as more invasive in affecting (by limiting what code they can run) far more actors (as opposed to, what, reactor operators and uranium handlers?, in the nuclear case). If unrestricted general-purpose CPUs are still readily available, it does seem like nothing much would change in practice & the important freedoms would be preserved; combined with only a few chipmakers actually being liable for compliance, I can see calling this not more invasive.
[...]
(I do think it's probably meaningful that these aren't legal mandates, and more meaningful that unrestricted platforms are also readily available.)
4
Hundreds of years seems far too strong even if the only driver were Moore's Law, which it isn't. You also have:
* Improvements in algorithms, data and software
* Increased optimization for ML workloads on laptops for running local models for autocomplete, transcription, simple tasks etc
You also don't need to get into actual 'run on laptop' range, even 2-3OOMs above that would allow you to train on things that would barely register as datacenters. 10-15 years seems more likely for training > current frontier models on not-really-a-datacenters especially if you want a buffer to account for the uncertainty that there could be discrete breakthroughs in training efficiency
6
I agree that if you take a factor of 1,000 and play forward current algorithmic progress speeds by 10-15 years, then you get there, but it really doesn't seem very hard to just not have algorithmic progress continue that long, and it doesn't require a total surveillance state to prevent people from acquiring $1M+ of frontier computing hardware, so I feel like both of these are just much better levers to pull on.
4
If you can prevent algorithmic progress then I agree somewhat, though experiments to make this sort of progress should be doable on small volumes of compute so you'd need to suppress the research or publishing.
I do think that not being able to acquire, say, $1M worth of matmul-adapted compute is a higher bar than you imply here. Being able to do large numbers of matmuls is an extremely useful property for like a zillion reasons beyond AI - iirc Google poured at least hundreds of millions into building TPUs based only on the projected demand for very simple NLP algorithms. LLM-optimized matmul machines are helpful but you can use anything if you're willing to adapt your algorithms and software. I would expect rendering farm or basically any serious cluster at all in 15y to be able to train >current models.
2
Yep, and I think all you need to do is to have a code-signing regime for running code on that compute. This would be reasonably annoying, but I don't think would make a huge difference economically for non-AI applications.
[...]
I think we probably disagree on how insights from small experiments scale to big experiments, but I agree you would need to suppress at least some research or publishing, similar to what we do with nuclear weapons and nuclear reactors.
2
I agree that this is the big "if" here. I totally think there could be discrete breakthroughs in training efficiency, and it might be worth giving up substantial civil liberties to hedge against that case. I might advocate for such policies after we have done the very basics of keeping things safe in the worlds where there aren't enormous discrete breakthroughs in training efficiency which currently seems more likely to me.
4
Which discourse are you referring to? This consideration is front-and-center in every discussion I've been in about the topic. In particular, there's explicit discussion of "how long can we maintain a pause?" given various background parameters.
It seems like the answer isn't "indefinitely", because it gets easier to develop a misaligned superintelligence over time. (Though I admit that, as other commenters raise, there may be ways to halt or dramatically slow the process driving that change as well.)
4
I'm glad to hear it; it seems to me very relevant for thinking sensibly about strategy here. I mean just that I personally have noticed little mention of this possibility, in the discourse about pausing/halting AI that I have encountered—e.g., that surrounding Nate and Eliezer's book (though the book itself does mention it!) Perhaps this suggests merely that I have have been sampling badly, e.g. from dumb corners of this discourse.
4
What kind of algorithm and what kind of training set could fit on a laptop and still produce superintelligence? It sounds like the algorithm or its designer might need to be superintelligent already for that to happen.
3
Steven Byrnes wrote similarly in Jan '23:
[...]
3
Though note the difference between 'single-PC compute after more expensive experimental work' (what it sounds like Steven is predicting, and Habryka is assuming) and 'single-PC compute without that' (what Adam is predicting).
Nah, my model allows ASI without massive compute at any point in the process, see “Foom & Doom 1: ‘Brain in a box in a basement’” (esp. §1.3), and maybe also “The nature of LLM algorithmic progress” §4.
2
It's more dignified to try to stop AI, have someone create a superintelligence on a laptop and die anyway, than it is not to try at all.
Arguments criticizing the FDA often seem to weirdly ignore the "F." For all I know food safety regulations are radically overzealous too, but if so I've never noticed (or heard a case for) this causing notable harm.
Overall, my experience as a food consumer seems decent—food is cheap, and essentially never harms me in ways I expect regulators could feasibly prevent (e.g., by giving me food poisoning, heavy metal poisoning, etc). I think there may be harmful contaminants in food we haven't discovered yet, but if so I mostly don't blame the FDA for that lack of knowledge, and insofar as I do it seems an argument they're being under-zealous.
1Criticizing FDA food regulations is a niche; it is hard to criticize 'the unseen', especially when it's mostly about pleasure and the FDA is crying: 'we're saying lives! Won't someone thinking of the children? How can you disagree, just to stuff your face? Shouldn't you be on a diet anyway?'
But if you go looking, you'll find tons of it: pasteurized cheese and milk being a major flashpoint, as apparently the original unpasteurized versions are a lot tastier. (I'm reminded of things like beef tallow for fries or Chipotle - how do you know how good McDonald's french fries used to taste before an overzealous crusader destroyed them if you weren't there 30+ years ago? And are you really going to stand up and argue 'I think that we should let people eat fries made with cow fat, because I am probably a lardass who loves fries and weighs 300 pounds, rather than listen to The Science™'?) There's also the recent backfiring of overzealous allergy regulations, which threatens to cut off a large fraction of the entire American food supply to people with sesame & peanut allergies, due solely to the FDA. (Naturally, of course, the companies get the blame.) Similarly, I read food industry peop...
46
Those are great examples, thanks; I can totally believe there exist many such problems.
Still, I do really appreciate ~never having to worry that food from grocery stores or restaurants will acutely poison me; and similarly, not having to worry that much that pharmaceuticals are adulterated/contaminated. So overall I think I currently feel net grateful about the FDA’s purity standards, and net hateful just about their efficacy standards?
2
God bless Robert F. Kennedy Jr.
9
The 'Food' and the 'Drug' parts behave very differently. By default food products are allowed. There may be purity requirements or restaurant regulations but you don't need to run studies or get approvals to serve an edible product or a new combination. By default drugs are banned.
I think the FDA is under zealous about heavy metals and other contaminants. But the FDA does a decent job of regulating food. However the 'drug' side is a nightmare. But the two situations are de facto handled in very, very different ways. So its not obvious why an argument would cover both of them.
9
Have you ever visited a country without zealous food safety regulations? I think it's one of those things where it's hard to realize what the alternative looks like (plentiful, cheap, and delicious street food available wherever people gather, so that you no longer have to plan around making sure you either bring food or go somewhere with restaurants, and it is viable for individuals to exist without needing a kitchen of their own).
4
What countries are you imagining? I know some countries have more street food, but from what I anecdotally hear most also have far more food poisoning/contamination issues. I'm not sure what the optimal tradeoff here looks like, and I could easily believe it's closer to the norms in e.g. Southeast Asia than the U.S. But it at least feels much less obvious to me than that drug regulations are overzealous.
(Also note that much regulation of things like food trucks is done by cities/states, not the FDA).
2
Mexico and Chile are the most salient examples to me. But also I've only ever gotten food poisoning once in my life despite frequent risky food behavior.
Strong agree that the magnitude of the overzealousness is much higher for drugs than for food.
4
Basically, because food is a domain where there are highly negative tail effects, but not highly positive tail effects, conditioning on eating food at all, and thus it's an area where you can afford to be restrictive, and this is notably not the case for medicine, where high negative and positive tail effect effects exist, so you need to be more lenient on your standards.
4
I think one exception may be salmonellosis. In USA, you get 1.2 million illnesses, 23000 hospitalizations, and 450 deaths every year. To compare, in EU, selling contaminated chicken products is illegal, and when hundreds of people get sick, it becomes a scandal.
2
You have to refrigerate eggs in the us while you don't have to in the EU because of FDA regulations about washing the eggs.
There's a strain of pro-sugar and anti-fat policies in regards to food for which the FDA shares some of the blame. Through that they might have contributed to the obesity crisis.
It's possible that the regulations on farmers markets reduce the amount of farmers markets (which might be central to lower obesity levels in France and better health).
When asked to compare French regulations for farmers markets with the US regulations, Claude told me:
Overall, while both countries prioritize food safety, the French system tends to be more flexible for small producers and traditional methods, whereas the US system is more uniformly applied regardless of scale. The French approach often allows for more diverse and traditional products at markets, while the US system provides more consistent safety standards across diverse regions.
You likely could argue that the FDA shares part the blame for the obesity epidemic by setting bad incentives for low-fat and high fructose corn syrup foods while at the same time making it harder for farmers markets to actually sell healthy food.
Apart from that you do have people who oppose forced labeling of GMO products which is also an FDA rule.
[...]
Do you see microplastics as "harmful contaminants that haven't been discovered yet"? It would be possible to have regulations that limit the amount of microplastic in plastic drinking bottles but those currently don't exist.
1
I'd be interested in an article looking at whether the FDA is better at regulating food safety. I do expect food is an easier area, because erring on the side of caution doesn't really lose you much — most food products have close substitutes. If there's some low but not extremely low risk of a chemical in a food being bad for you, then the FDA can more easily deny approval without significant consequences: Medicine has more outsized effects if you are slow to approve usage.
Yet, perhaps this has led to reduced variety in food choices? I notice less generic or lesser-known food and beverage brands relative to a decade ago, though I haven't verified whether my that background belief is accurate. I'd be curious also for an investigation in such an article about the extent of the barriers to designing a new food product; especially food products that aren't doing anything new, purely a mixture of ingredients already considered safe (or at least, considered allowed). Would there be more variety? Or notably cheaper food?
I was surprised to find a literature review about probiotics which suggested they may have significant CNS effects. The tl;dr of the review seems to be: 1) You want doses of at least or CFU, and 2) You want, in particular, the strains B. longum, B. breve, B. infantis, L. helveticus, L. rhamnosus, L. plantarum, and L. casei.
I then sorted the top 15 results on Amazon for "probiotic" by these desiderata, and found that this one seems to be best.
Some points of uncertainty:
- Probiotic manufacturers generally don't disclose the strain proportions of their products, so there's some chance they mostly include e.g. whatever's cheapest, plus a smattering of other stuff.
- One of the reviewed studies suggests L. casei may impair memory. I couldn't find a product that didn't have L. casei but did have at least CFU of each other recommended strain, so if you take the L. casei/memory concern seriously your best option might be combining this and this.
[This comment is no longer endorsed by its author]
For convenience, here's a slightly edited-for-clarity version of the abstract:
38 studies (all randomized controlled trials) were included: 25 in animals and 15 in humans (2 studies were conducted in both). Most studies used Bifidobacterium (eg, B. longum, B. breve, and B. infantis) and Lactobacillus (eg, L. helveticus, and L. rhamnosus), with doses between and 10^10 colony-forming units for 2 weeks in animals and 4 weeks in humans.
These probiotics showed efficacy in improving psychiatric disorder-related behaviors including anxiety, depression, autism spectrum disorder (ASD), obsessive-compulsive disorder, and memory abilities, including spatial and non-spatial memory.
Because many of the basic science studies showed some efficacy of probiotics on central nervous system function, this background may guide and promote further preclinical and clinical studies. Translating animal studies to human studies has obvious limitations but also suggests possibilities. Here, we provide several suggestions for the translation of animal studies. More experimental designs with both behavioral and neuroimaging measures in healthy volunteers and patients are needed in the future.
Possibly another good example of scientists failing to use More Dakka. The mice studies all showed solid effects, but then the human studies used the same dose range (10^9 or 10^10 CFU) and only about half showed effects! Googled for negative side effects of probiotics and the healthline result really had to stretch for anything bad. Wondering if, as much larger organisms, we should just be jacking up the dosage quite a bit.
5
On the other hand: half of mouse studies working in humans is an extremely good success rate. We should be quite suspicious of file-drawer effects and p-hacking.
5
I agree the effect is consistent enough that we should be suspicious of file drawer/p-hacking—although that's also what you'd expect to see if the effect were in fact large—but note that they were different studies, i.e. the human studies mostly weren't based on the non-human ones.
4
I was initially very concerned about this but then noticed that almost all the tested secondary endpoints were positive in the mice studies too. The human studies could plausibly still be meaningless though.
Has anyone (esp you Jim) looked into fecal transplants for this instead, in case our much longer digestive system is a problem?
In the early 1900s the Smithsonian Institution published a book each year, which mostly just described their organizational and budget updates. But they each also contained a General Appendix at the end, which seems to have served a function analogous to the modern "Edge" essays—reflections by scientists of the time on key questions of interest. For example, the 1929 book includes essays speculating about what "life" and "light" are, how insects fly, etc.
Apparently Otzi the Iceman still has a significant amount of brain tissue. Conceivably memories are preserved?
3
Another (unlikely, but more likely than almost all other ancient people) candidate for partial future revival: During the 79 AD eruption of Vesuvius, part of this man's brain was vitrified.
I found LinkedIn's background breakdown of DeepMind employees interesting; fewer neuroscience backgrounds than I would have expected.