If I’m planning a holiday to New York (and I live pretty far from New York), it’s quite straightforward to get fellow travellers to agree that we need to buy plane tickets to New York. Which airport? Eh, whichever is more convenient, I guess. Alternatively, some may prefer a road...
Homo sapiens is a real puzzle. Many species on very different branches of the tree of life—corvids, parrots, elephants, dolphins, cephalopods and non-human primates—have converged to very similar limits of intelligence. Yet humans, despite being extremely similar genetically to other primates, have blown right past those limits. Or have they?...
That's alright. Would you be able to articulate what you associate with AGI in general? For example, do you associate AGI with certain intellectual or physical capabilities, or do you associate it more with something like moral agency, personhood or consciousness?
Thank you for the clarification!
Of course, it is much more likely to be predictable a couple of days in advance than a year in advance, but even the former may conceivably be quite challenging depending on situational awareness of near-human-level models in training.
Do I understand correctly that you think that we are likely to only recognize AGI after it has been built? If so, how would we recognize AGI as you define it?
Do you also think that AGI will result in a fast take-off?
What would you expect the world to look like if AGI < 2030? Or put another way, what evidence would convince you that AGI < 2030?
What do you make of feral children like Genie? While there are not many counterfactuals to cultural learning—probably mostly because depriving children of cultural learning is considered highly immoral—feral children do provide strong evidence that humans that are deprived of cultural learning do not come close to being functional adults. Additionally, it seems obvious that people who do not receive certain training, e.g., those who do not learn math or who do not learn carpentry, generally have low capability in that domain.
the genetic changes come first, then the cultural changes come after
You mean to say that the human body was virtually “finished evolving” 200,000 years ago, thereby laying the groundwork for cultural... (read more)
How do LLMs and the scaling laws make you update in this way? They make me update in the opposite direction. For example, I also believe that the human body is optimized for tool use and scaling, precisely because of the gene-culture coevolution that Henrich describes. Without culture, this optimization would not have occurred. Our bodies are cultural artifacts.
Cultural learning is an integral part of the scaling laws; the scaling laws show that indefinitely scaling the number of parameters in a model doesn't quite work; the training data also has to scale, with the implication that that data is some kind of cultural artifact, where the quality of that artifact determines the... (read more)
Current LLMs can only do sequential reasoning of any kind by adjusting their activations, not their weights, and this is probably not enough to derive and internalize new concepts à la C.
For me this is the key bit which makes me update towards your thesis.
This is indeed an interesting sociological breakdown of the “movement”, for lack of a better word.
I think the injection of the author’s beliefs about whether or not short timelines are correct distracting from the central point. For example, the author states the following.
there is no good argument for when [AGI] might be built.
This is a bad argument against worrying about short timelines, bordering on intellectual dishonesty. Building anti-asteroid defenses is a good idea even if you don’t know that one is going to hit us within the next year.
The argument that it’s better to have AGI appear sooner rather than later because institutions are slowly breaking down is an interesting one. It’s... (read more)
Ah, I see now. Thank you! I remember reading this discussion before and agree with your viewpoint that he is still directionally correct.
he apparently faked some of his evidence
Would be happy to hear more about this. Got any links? A quick Google search doesn’t turn up anything.
If I’m planning a holiday to New York (and I live pretty far from New York), it’s quite straightforward to get fellow travellers to agree that we need to buy plane tickets to New York. Which airport? Eh, whichever is more convenient, I guess. Alternatively, some may prefer a road trip to New York, but the general direction is obvious for everyone.
However, as the holiday gets closer in time or space, the question of what we actually mean by a holiday in New York becomes more and more contentious. Did we mean New York State or New York City? Did we mean Brooklyn or Broadway? Which Broadway theater? Which show? Which seats?
By... (read more)
Homo sapiens is a real puzzle. Many species on very different branches of the tree of life—corvids, parrots, elephants, dolphins, cephalopods and non-human primates—have converged to very similar limits of intelligence. Yet humans, despite being extremely similar genetically to other primates, have blown right past those limits. Or have they?
In recent years, interdisciplinary research has provided new insights into human intelligence, best described in Joseph Henrich's 2015 book The Secret of Our Success. Scott Alexander reviewed it in 2019 and its ideas have been mentioned here and there over the years, but the AI research community, including the AI alignment community, has neglected to appropriately absorb their implications, to its severe detriment.... (read 4425 more words →)
I’m glad you asked. I completely agree that nothing in the current LLM architecture prevents that technically and I expect that it will happen eventually.
The issue in the near future is practicality, because training models is currently—and will in the near future still be—very expensive. Inference is less expensive, but still so expensive that profit is only possible by serving the model statically (i.e., without changing its weights) to many clients, which amortizes the cost of training and inference.
These clients often rely heavily on models being static, because it makes its behavior predictable enough to be suitable for a production environment. For example, if you use a model for a chat bot... (read more)