Except that Yudkowsky had actually made the predictions in public. However, he didn't know in advance that the AIs would be trained as neural networks that are OOMs less efficient at keeping context[1] in mind. Other potential mispredictions are Yudkowsky's cases for the possibility to greatly increase the capabilities starting from a human brain simulation[2] or to simulate a human brain working ~6 OOMs faster:
Yudkowsky's case for a superfast human brain
T hefastest observed neurons fire 1000 times per second; the fastest axon fibers con duct signals at 150 meters/second, a half-millionth the speed of light; each synaptic op eration dissipates around 15,000 attojoules, which is more than a million times the ther modynamicminimumforirreversible computations at room temperature (kT300 ln(2) = 0003 attojoules per bit). It would be physically possible to build a brain that computed a million times as fast as a human brain, without shrinking the size, or running at lower temperatures, or invoking reversible computing or quantum computing. If a human mind were thus accelerated, a subjective year of thinking would be accomplished for ev ery 31 physical seconds in the outside world, and a millennium would fly by in eight and a half hours. Vinge (1993) referred to such sped-up minds as “weak superhumanity”: a mind that thinks like a human but much faster.
However, as Turchin points out in his book[3] written in Russian, simulating a human brain requires[4] just 1e15 FLOP/second, or less than 1e22 FLOP/month.
Turchin's argument in Russian
Для создания ИИ необходимо, как минимум, наличие достаточно мощного компьютера. Сейчас самые мощные компьютеры имеют мощность порядка 1 петафлопа (10 операций с плавающей запятой в секунду). По некоторым оценкам, этого достаточно для эмуляции человеческого мозга, а значит, ИИ тоже мог бы работать на такой платформе. Сейчас такие компьютеры доступны только очень крупным организациям на ограниченное время. Однако закон Мура предполагает, что мощность компьютеров возрастёт за 10 лет примерно в 100 раз, т. е., мощность настольного компьютера возрастёт до уровня терафлопа, и понадобится только 1000 настольных компьютеров, объединённых в кластер, чтобы набрать нужный 1 петафлоп. Цена такого агрегата составит около миллиона долларов в нынешних ценах – сумма, доступная даже небольшой организации. Для этого достаточно реализовать уже почти готовые наработки в области многоядерности (некоторые фирмы уже сейчас предлагают чипы с 1024 процессорами ) и уменьшения размеров кремниевых элементов.
ChatGPT's translation into English
To create AI, at the very least, a sufficiently powerful computer is required. Currently, the most powerful computers have a performance of about 1 petaflop (10¹⁵ floating-point operations per second). According to some estimates, this is enough to emulate the human brain, which means that AI could also run on such a platform. At present, such computers are available only to very large organizations for limited periods of time. However, Moore’s Law suggests that computer performance will increase roughly 100-fold over the next 10 years. That is, the performance of a desktop computer will reach the level of a teraflop, and only 1,000 desktop computers connected in a cluster would be needed to achieve the required 1 petaflop. The cost of such a system would be about one million dollars at today’s prices—a sum affordable even for a small organization. To achieve this, it is enough to implement the nearly completed developments in multicore technology (some companies are already offering chips with 1,024 processors) and in reducing the size of silicon elements.
- ^
- ^
A case against the existence of an architecture more efficient than a human brain is found in Jacob Cannel's post. But it doesn't exclude a human brain trained for millions of years.
- ^
Unfortunately, the book's official translation into English has too low quality .
- ^
Fortunately, the simulation requires OOMs more dynamic memory.
1
[epistemic status: the points I make are IMO real and important, but there are also various counterpoints; I'm not settled on an overall opinion here, and the categories I draw are probably kind of dumb/misleading]
Many people who are concerned about existential risk from AI spend their time advocating for radical changes to how AI is handled. Most notably, they advocate for costly restrictions on how AI is developed now and in the future, e.g. the Pause AI people or the MIRI people. In contrast, I spend most of my time thinking about relatively cheap interventions that AI companies could implement to reduce risk assuming a low budget, and about how to cause AI companies to marginally increase that budget. I'll use the words "radicals" and "moderates" to refer to these two clusters of people/strategies. In this post, I’ll discuss the effect of being a radical or a moderate on your epistemics.
I don’t necessarily disagree with radicals, and most of the disagreement is unrelated to the topic of this post; see footnote for more on this.[1]
I often hear people claim that being a radical is better for your epistemics than being a moderate: in particular, I often hear people say that moderates end up too friendly to AI companies due to working with people from AI companies. I agree, but I think that working as a moderate has a huge advantage for your epistemics.
The main advantage for epistemics of working as a moderate is that almost all of your work has an informed, intelligent, thoughtful audience. I spend lots of time talking to and aiming to persuade AI company staff who are generally very intelligent, knowledgeable about AI, and intimately familiar with the goings-on at AI companies. In contrast, as a radical, almost all of your audience—policymakers, elites, the general public—is poorly informed, only able or willing to engage shallowly, and needs to have their attention grabbed intentionally. The former situation is obviously way more conducive to maintaining good epistemics.
I think working as a moderate has a bunch of good effects on me:
Many people I know who work on radical AI advocacy spend almost all their time thinking about what is persuasive and attention-grabbing for an uninformed audience. They don't experience nearly as much pressure on a day-to-day basis to be well informed about AI, to understand the fine points of their arguments, or to be calibrated and careful in their statements. They update way less on the situation from their day-to-day work than I do. They spend their time as big fish in a small pond.
I think this effect is pretty big. People who work on radical policy change often seem to me to be disconnected from reality and sloppy with their thinking; to engage as soldiers for their side of an argument, enthusiastically repeating their slogans. I think it's pretty bizarre that despite the fact that LessWrongers are usually acutely aware of the epistemic downsides of being an activist, they seem to have paid relatively little attention to this in their recent transition to activism. Given that radical activism both seems very promising and is popular among LessWrongers regardless of what I think about it, I hope we try to understand the risks (perhaps by thinking about historical analogies) of activism and think proactively and with humility about how to mitigate them.
I'll note again that the epistemic advantages of working as a moderate aren’t in themselves strong reasons to believe that moderates are right about their overall strategy.
I will also note that I work as a moderate from outside AI companies; I believe that working inside AI companies carries substantial risks for your epistemics. But IMO the risks from working at a company are worth conceptually distinguishing from the risks resulting from working towards companies adopting marginal changes.
In this post, I mostly conflated "being a moderate" with "working with people at AI companies". You could in principle be a moderate and work to impose extremely moderate regulations, or push for minor changes to the behavior of governments. I did this conflation mostly because I think that for small and inexpensive actions, you're usually better off trying to make them happen by talking to companies or other actors directly (e.g. starting a non-profit to do the project) rather than trying to persuade uninformed people to make them happen. And cases where you push for minor changes to the behavior of governments have many of the advantages I described here: you're doing work that substantially involves understanding a topic (e.g. the inner workings of the USG) that your interlocutors also understand well, and you spend a lot of your time responding to well-informed objections about the costs and benefits of some intervention.
Thanks to Daniel Filan for helpful comments.
Some of our difference in strategy is just specialization: I’m excited for many radical projects, some of my best friends work on them, and I can imagine myself working on them in the future. I work as a moderate mostly because I (and Redwood) have comparative advantage for it: I like thinking in detail about countermeasures, and engaging in detailed arguments with well-informed but skeptical audiences about threat models.
And most of the rest of the difference in strategy between me and radicals is downstream of genuine object-level disagreement about AI risks and how promising different interventions are. If you think that all the interventions I'm excited for are useless, then obviously you shouldn't spend your time advocating for them.