Shulman and Yudkowsky on AI progress
38johnswentworth
52Matthew Barnett
29RomanS
9Owain_Evans
8RomanS
5Matthew Barnett
3RomanS
4RobertM
26Rob Bensinger
18Sammy Martin
16Gram Stone
16lsusr
14calef
12Veedrac
10DanielFilan
-12mary coca
New Comment
I read this line:
I also correctly bet that vaccine approval and deployment would be historically unprecedently fast and successful due to the high demand).
... and I was like "there is no way in hell that this was unprecedentedly fast". The first likely-counterexample which sprang to mind was the 1957 influenza pandemic, so I looked it up. The timeline goes roughly like this:
The first cases were reported in Guizhou of southern China, in 1956[6][7] or in early 1957.[1][3][8][9] They were soon reported in the neighbouring province of Yunnan in late February or early March 1957.[9][10] By the middle of March, the flu had spread all over China.[9][11]
The People's Republic of China was not a member of the World Health Organization at the time (not until 1981[12]), and did not inform other countries about the outbreak.[11] The United States CDC, however, states that the flu was "first reported in Singapore in February 1957".
[...]
The microbiologist Maurice Hilleman was alarmed by pictures of those affected by the virus in Hong Kong that were published in The New York Times. He obtained samples of the virus from a US Navy doctor in Japan. The Public Health Service released the virus cultures to vaccine manufacturers on 12 May 1957, and a vaccine entered trials at Fort Ord on 26 July and Lowry Air Force Base on 29 July.[20]
The number of deaths peaked the week ending 17 October, with 600 reported in England and Wales.[18] The vaccine was available in the same month in the United Kingdom.
So it sure sounds like they were faster in 1957. (Though that article does note that the 1957 vaccine "was initially available only in limited quantities". Even so, it sounds like it took about 5 months to do what took us about 11 months in the 2020 pandemic.)
My understanding is that the correct line is something like, "The COVID-19 vaccines were developed and approved unprecedentedly fast, excluding influenza vaccines." If you want to find examples of short vaccine development, you don't need to go all the way back to the 1957 influenza pandemic. For the 2009 Swine flu pandemic,
Analysis of the genetic divergence of the virus in samples from different cases indicated that the virus jumped to humans in 2008, probably after June, and not later than the end of November,[38] likely around September 2008... By 19 November 2009, doses of vaccine had been administered in over 16 countries.
And more obviously, the flu shot is modified yearly to keep up-to-date with new variants. Wikipedia notes that influenza vaccines were first successfully distributed in the 1940s, after developement began in 1931.
When considering vaccines other than influenza shot, this 2017 EA forum post from Peter Wildeford is informative. He tracks the development history of "important" vaccines, as he notes,
This is not intended to be an exhaustive list of all vaccines, but is intended to be exhaustive of all vaccines that would be considered "important", such as the vaccines on the WHO list of essential medicines and notable vaccines under current development.
His bottom line:
Taken together and weighing these three sources of evidence evenly, this suggests an average of 29 years for the typical vaccine.
No vaccine on his list had been researched, manufactured, and distributed in less than one year. The closest candidate is the Rabies vaccine, which had a 4 year timeline, from 1881-1885.
so, like, if I was looking for places that would break upward, I would be like "universal translators that finally work"
The story of DeepL might be of relevance.
DeepL has created a translator that is noticeably better than Google Translate. Their translations are often near-flawless.
The interesting thing is: DeepL is a small company that has OOMs less compute, data, and researchers than Google.
Their small team has beaten Google (!), at the Google's own game (!), by the means of algorithmic innovation.
Interesting. Can you point to a study by external researchers (not DeepL) that compares DeepL to other systems (such as Google Translate) quantitatively? After a quick search, I could only find one paper, which was just testing some tricky idioms in Spanish and didn't find significant differences between DeepL and Google. (Wikipedia links to this archived page of comparisons conducted by DeepL but there's no information about the methodology used and the difference in performance seem too big to be credible to me.)
The primary source of my quality assessment is my personal experience with both Google Translate and DeepL. I speak 3 languages, and often have to translate between them (2 of them are not my native languages, including English).
As I understand, making such comparisons in a quantitative manner is tricky, as there are no standardized metrics, there are many dimensions of translation quality, and the quality strongly depends on the language pair and the input text.
Google Scholar lists a bunch of papers that compare Google Translate and DeepL. I checked a few, and they're all over the place. For example, one claims that Google is better, another claims that they score the same, and yet another claims that DeepL is better.
My tentative conclusion: by quantitative metrics, DeepL is in the same league as Google Translate, and might be better by some metrics. Which is still an impressive achievement by DeepL, considering the fact that they have orders-of-magnitude less data, compute, and researchers than Google.
My tentative conclusion: by quantitative metrics, DeepL is in the same league as Google Translate, and might be better by some metrics. Which is still an impressive achievement by DeepL, considering the fact that they have orders-of-magnitude less data, compute, and researchers than Google.
Do they though? Google is a large company, certainty, but they might not actually give Google Translate researchers a lot of funding. Google gets revenue from translation by offering it as a cloud service, but I found this thread from 2018 where someone said,
Google Translate and Cloud Translation API are two different products doing some similar functions. It is safe to assume they have different algorithms and differences in translations are not only common but expected.
From this, it appears that there is little incentive for Google to improve the algorithms on Google Translate.
We could try to guesstimate how much money Google spends on the R&D for Google Translate.
Judging by these data, Google Translate has 500 million daily users, or 11% of the total Internet population worldwide.
Not sure how much it costs Google to run a service of such a scale, but I would guesstimate that it's in the order of ~$1 bln / year.
If they spend 90% of the total Translate budget on keeping the service online, and 10% on the R&D, they have ~$100 mln / year on the R&D, which is likely much more than the total income of DeepL.
It is unlikely that Google is spending ~$100 mln / year on something without a very good reason.
One of the possible reasons is training data. Judging by the same source, Google Translate generates 100 billion words / day. It means, Google gets about the same massive amount of new user-generated training data per day.
The amount is truly massive: thanks to Google Translate, Google gets the Library-of-Congress worth of new data per year, multiplied by 10.
And some of the new data is hard to get by other means (e.g. users trying to translate their personal messages from a low-resource language like Basque or Azerbaijani).
I think your estimate for the operating costs (mostly compute) of running a service that has 500 million daily users (with probably well under 10 requests per user on average, and pretty short average input) is too high by 4 orders of magnitude, maybe even 5. My main uncertainty is how exactly they're servicing each request (i.e. what % of requests are novel enough to actually need to get run though some more-expensive ML model instead of just returning a cached result).
5 billion requests per day translates to ~58k requests per second. You might have trouble standing up a service that handles that level of traffic on a 10k/year compute budget if you buy compute on a public cloud (i.e. AWS/GCP) but those have pretty fat margins so the direct cost to Google themselves would be a lot lower.
Note: I've written up short summaries of each entry in this sequence so far on https://intelligence.org/late-2021-miri-conversations/, and included links to audio recordings of most of the posts.
Compare this,
[Shulman][22:18]
We're in the Eliezerverse with huge kinks in loss graphs on automated programming/Putnam problems.
Not from scaling up inputs but from a local discovery that is much bigger in impact than the sorts of jumps we observe from things like Transformers.
[Yudkowsky][22:21]
but, sure, "huge kinks in loss graphs on automated programming / Putnam problems" sounds like something that is, if not mandated on my model, much more likely than it is in the Paulverse. though I am a bit surprised because I would not have expected Paul to be okay betting on that.
to this,
[Rohin] To the extent this is accurate, it doesn't seem like you really get to make a bet that resolves before the end times, since you agree on basically everything until the point at which Eliezer predicts that you get the zero-to-one transition on the underlying driver of impact.
Eliezer does (at least weakly) expect more trend breaks before The End (even on metrics that aren't qualitative measures of impressiveness/intelligence, but just things like model loss functions), despite the fact that Rohin's summary of his view is (I think) roughly accurate.
What explains this? I think it's something roughly like, part of the reason Eliezer expects a sudden transition when we reach the core of generality in the first place is because he thinks that's how things usually go in the history of tech/AI progress - there's also specific reasons to think it will happen in the case of finding the core of generality, but there are also general reasons. See e.g. this from Eliezer:
well, the Eliezerverse has more weird novel profitable things, because it has more weirdness
I take 'more weirdness' to mean something like more discoveries that induce sudden improvements out there in general.
So I think that's why his view does make (weaker) differential predictions about earlier events that we can test, not because the zero-to-one core of generality hypothesis predicts anything about narrow AI progress, but because some of the beliefs that led to that hypothesis do.
We can see there's two (connected) lines of argument and that Eliezer and Paul/Carl/Richard have different things to say on each - 1 is more localized and about seeing what we can learn about AGI specifically, and 2 is about reference class reasoning and what tech progress in general tells us about AGI:
- Specific to AGI: What can we infer from human evolution and interrogating our understanding of general intelligence (?) about whether AGI will arrive suddenly?
- Reference Class: What can we infer about AGI progress from the general record of technological progress, especially how common big impacts are when there's lots of effort and investment?
My sense is that Eliezer answers
- Big update for Eliezer's view: This tells us a lot, in particular we learn evolution got to the core of generality quickly, so AI progress will probably get there quickly as well. Plus, Humans are an existence proof for the core of generality, which suggests our default expectation should be sudden progress when we hit the core.
- Smaller update for Eliezer's view: This isn't that important - there's no necessary connection between AGI and e.g. bridges or nukes. But, you can at least see that there's not a strong consistent track record of continuous improvement once you understand the historical record the right way (plus the underlying assumption in 2 that there will be a lot of effort and investment is probably wrong as well). Nonetheless, if you avoid retrospective trend-fitting and look at progress in the most natural (qualitative?) way, you'll see that early discoveries that go from 0 to 1 are all over the place - Bitcoin, the Wright flyer, nuclear weapons are at least not crazy exceptions and quite possibly the default.
While Paul and Carl(?) answer,
- Smaller update for Paul's view: The disanalogies between AI progress and Evolution all point in the direction of AI progress being smoother than evolution (we're intelligently trying to find the capabilities we want) - we get a weak update in favour of the smooth progress view from understanding this disanalogy between AI progress and evolution, but really we don't learn much, except that there aren't any good reasons to think there are only a few paths to a large set of powerful world affecting capabilities. Also, the core of generality idea is wrong, so the idea that Humans are an existence proof for it or that evolution tells us something about how to find it is wrong.
- Big update for Paul's view: reasoning from the reference class of 'technologies where there are many opportunities for improvement and many people trying different things at once' lets us see why expecting smooth progress should be the default. It's because as long as there are lots of paths to improvement in the underlying capability landscape (which is the default because that's how the world works by default), and there are lots of people trying to make improvements in different ways, the incremental changes add up to smooth outputs.
So Eliezer's claim that Paul et al's trend-fitting must include
doing something sophisticated but wordless, where they fit a sophisticated but wordless universal model of technological permittivity to bridge lengths, then have a wordless model of cognitive scaling in the back of their minds
is sort of correct, but the model isn't really sophisticated or wordless.
The model is: as long as the underlying 'capability landscape' offers many paths to improvements, not just a few really narrow ones that swamp everything else, lots of people intelligently trying lots of different approaches will lead to lots of small discoveries that add up. Additionally, most examples of tech progress look like 'multiple ways of doing something that add up', this is confirmed by the historical record.
And then the model of cognitive scaling consists of (among other things) specific counterarguments to the claim that AGI progress is one of those cases with a few big paths to improvement (e.g. Evolution doesn't give us evidence that AGI progress will be sudden).
[Shulman]
As I work through sectors and the rollout of past automation I see opportunities for large-scale rollout that is not heavily blocked by regulation...[Long list of examples]
[Yudkowsky]
so... when I imagine trying to deploy this style of thought myself to predict the recent past without benefit of hindsight, it returns a lot of errors. perhaps this is because I do not know how to use this style of thought...
..."There are many possible regulatory regimes in the world, some of which would permit rapid construction of mRNA-vaccine factories well in advance of FDA approval. Given the overall urgency of the pandemic some of those extra-USA vaccines would be sold to individuals or a few countries like Israel willing to pay high prices for them, which would provide evidence of efficacy and break the usual impulse towards regulatory uniformity among developed countries..."
On Carl's view, it sure seems like you'd just say something like "Healthcare is very overregulated, there will be an unusually strong effort anyway in lots of countries because Covid is an emergency, so it'll be faster by some hard to predict amount but still bottlenecked by regulatory pressures." And indeed the fastest countries got there in ~10 months instead of the multiple years predicted by superforecasters, or the ~3 months it would have taken with immediate approval.
The obvious object-level difference between Eliezer 'applying' Carl's view to retrodict covid vaccine rollout and Carl's prediction about AI is that Carl is saying there's an enormous number of potential applications of intermediately general AI tech, and many of them aren't blocked by regulation, while Eliezer's attempted operating of Carl's view for covid vaccines is saying "There are many chances for countries with lots of regulatory barriers to do the smart thing".
The vaccine example is a different argument than AI predictions, because what Carl is saying is that there are many completely open goals for improvement like automating factories and call centres etc. not that there are many opportunities to avoid the regulatory barriers that will block everything by default.
But it seems like Eliezer is making a more outside view appeal, i.e. approach stories where big innovations are used wisely with a lot of scepticism because of our past record, even if you can tell a story about why it will be quite different this time.
Now that we clarified up-thread that Eliezer's position is not that there was a giant algorithmic innovation in between chimps and humans, but rather that there was some innovation in between dinosaurs and some primate or bird that allowed the primate/bird lines to scale better
Where was this clarified...? My Eliezer-model says "There were in fact innovations that arose in the primate and bird lines which allowed the primate and bird lines to scale better, but the primate line still didn't scale that well, so we should expect to discover algorithmic innovations, if not giant ones, during hominin evolution, and one or more of these was the core of overlapping somethingness that handles chipping handaxes but also generalizes to building spaceships."
If we're looking for an innovation in birds and primates, there's some evidence of 'hardware' innovation rather than 'software.'
For a speculative software innovation hypothesis, there seem to be cognitive adaptations that arose in the LCA of all anthropoids for mid-level visual representations e.g. glossiness, above the level of, say, lines at a particular orientation, and below the level of natural objects, which seem like an easy way to exapt into categories, then just stack more layers for generalization and semantic memory. These probably reduce cost and error by allowing the reliable identification of foraging targets at a distance. There seem to be corresponding cognitive adaptations for auditory representations that strongly correlate with foraging targets, e.g. calls of birds that also target the fruits of angiosperm trees. Maybe birds and primates were under similar selection for mid-level visual representations that easily exapt into categories, etc.
Whenever birds are an outlier I ask myself "is it because birds fly?" Bird cells (especially bird mitochondria) are intensely optimized for power output because flying demands a high power-to-weight ratio. I think bird cells' individually high power output produces brains that can perform more (or better) calculations per unit volume/mass.
For what it's worth, the most relevant difficult-to-fall-prey-to-Goodheartian-tricks measure is probably cross entropy validation loss, as shown in this figure from the GPT-3 paper:
Serious scaling efforts are much more likely to emphasize progress here over Parameter Count Number Bigger clickbait.
Further, while this number will keep going down, we're going to crash into the entropy of human generated text at some point. Whether that's within 3 OOM or ten is anybody's guess, though.
I have some comments on things Eliezer has said. I don't expect these disagreements are very important to the main questions, because I tend to agree with him overall despite it.
my worldview also permits but does not mandate that you get up to the chimp level [...]
A naïve model of intelligence is a linear axis, that puts everything on a simple trendline with one-dimensional distances. I assume most people here understand that this is a gross oversimplification. Intelligence is made of multiple pieces, which can have unique strengths. There is still such a thing as generality of intelligence, that at some point you have enough tools to dynamically apply your reasoning to a great many more things than those tools were originally adapted for. This ability does seem to have some degree of scale, in that a human is more general than a chimp is more general than a mouse, though it also seems to be fairly sharp-edged, in that the difference in generality between a human and a chimp seems much greater than between a chimp and a mouse.
Because of the great differences between computer systems and biological ones, the individual components of computer intelligence (whether necessary for generality or not) when measured relative to a human tend to jump quickly between zero, when the program doesn't have that ability, and effectively infinite, when the program has that ability. There is also a large set of relations between capabilities whereby one ability can substitute for another, typically at the cost of some large factor reduction in performance.
A traditional chess engine has several component skills, like searching moves and updating decision trees, that it does vastly better than a human. This ability feeds down into some metrics of positional understanding. Positional understanding is not a particularly strong fundamental ability of a traditional chess engine, but rather something inefficiently paid for with its other skills that it does have in excess. The same idea holds for human intelligence, where we can use our more fundamental evolved skills, like object recognition, to build more complex skills. Because we have a broad array of baseline skills, and the enough tools to combine them to fit novel tasks, we can solve a much wider array of tasks, and can transfer domains with generally less cost than computers can. Nonetheless, there exist cognitive tasks we know can be done well that are outside of human mental capability.
When I envision AI progress leading up to AGI, I don't think of a single figure of merit that increases uniformly. I think of a set of capabilities, of which some are ~0, some are ~∞, and others are derived quantities not explicitly coded in. Scale advances in NNs push the effective infinities to greater effective infinities, and by extension push up the derived quantities across the board. Fundamental algorithmic advances increase the set of capabilities at ~infinity. At some point I expect the combination of fundamental and derived quantities to capture enough facets of cognition to push generality past a tipping point. In the run up to that point, lesser levels of generality will likely make AI systems applicable to more and more extensions of their primary domains.
This seems to me like it's mostly, if not totally, a literal interpretation of the world. Yet, to finally get to the point, nowhere in my map do I have a clear interpretation of what “get up to the chimp level” means. The degree to which chimps are generally intelligent seems very specific to the base skill set that chimps have, and it seems much more likely than not that AI will approach it from a completely different angle, because their base skillset is completely different and will generalize in a completely different way. The comment that “chimps are not very valuable” does not seem to map onto any relevant comment about pre-explosion AI. I do not know what it would mean to have a chimp level AI, or even chimp level generality.
I would not be terribly surprised to find that results on benchmarks continue according to graph, and yet, GPT-4 somehow does not seem very much smarter than GPT-3 in conversation.
I would be quite surprised for a similar improvement in perplexity not to correspond to at least a similar improvement in apparent smartness, versus GPT-3 over GPT-2.
I would not be surprised for the perplexity improvement to level off, maybe not immediately but at least in some small count of generations, as it seems entirely reasonable that there are some aspects of cognition that GPT-style models can't purchase. But for perplexity to improve on cue without an apparent improvement in intelligence, while logically coherent, would imply some very weird things about either the entropy of language or the scaling of model capacity.
That is, either language has a bunch of easy to access regularity between what GPT-3 reached and what an agent with a more advanced understanding of the world could reach, distributed coincidentally in line with previous capability increases, or GPT-3 roughly caps out the semantic capabilities of the network, but extra parameters added on top are still practically identically effective at extracting a huge amount of more marginal non-semantic regularities at a rate fast enough to compete with prior model scale increases that did both.
Stuff coming uncorrelated that way, sounds like some of the history I lived through, where people managed to make the graphs of Moore's Law seem to look steady by rejiggering the axes, and yet, between 1990 and 2000 home computers got a whole lot faster, and between 2010 and 2020 they did not.
There is truth to this comment, in that Dennard scaling fell around the turn of the millenia, but the 2010s were deceptive in that home computers stagnated in performance despite Moore's Law improvements, because Intel sold you your CPUs, were stuck on an old node, being stuck on an old node prevented them from pushing out new architectures, and their monopoly position meant they never really needed to compete on price or core count either.
But Moore's Law did continue, just through TSMC, and the corresponding performance improvements were felt primarily in GPUs and mobile SoCs. Both of those have improved at a great pace. In the last three years competition has returned to the desktop CPU market, and Intel has just managed to get out of their node crisis, so CPU performance really is picking up steam again. This is true both for per-core performance, driven by architectures making use of the great many transistors available, and even moreso true of aggregate performance, what with core counts in the Steam Survey increasing from an average of 3.0 in early 2017 to an average of 5.0 in April this year.
You are correct that the scaling regimes are different now, and Dennard scaling really is dead for good, but if you look back at the original Moore's Law graphs from 1965, they never mentioned frequency, so I don't buy the claim that the graphs have been rejigged.
One thing Carl notes is that a variety of areas where AI could contribute a lot to the economy are currently pretty unregulated. But I think there's a not-crazy story where once you are within striking range of making an area way more efficient with computers, then the regulation hits. I'm not sure how to evaluate how right that is (e.g. I don't think it's the story of housing regulation), but just wanted it said.
Curated and popular this week