I found this kinda hard to grok but gonna throw in my two cents anyway. Seems like priests (sensu stricto) obtain power by creating a body of 'consensually true' 'expertise' and hijacking human prestige bias adaptations, not really corresponding to reality because all theistic hypotheses are false, but not totally arbitrary because the 'knowledge' is culturally inherited. When someone tries to establish a new consensus, we call it a schism. This seems like one meta level up from what kings do and I'm not sure is faithfully captured by the obscure vs. arcane distinction. This also doesn't directly contradict my vague impression that wars of succession are way more common than schisms historically, each of these being special cases of the parade going somewhere else than where kings and priests tried to lead them, respectively. I think this fits pretty well with lawyers as modern warrior-priests too (priests sensu lato of course). Law isn't quite the same as theology I think, in that it's a little less contingent than theology but more than science/CS/math.
I really like how many hypotheses you generated here.
A few months ago my GM pulled me in kind of in the way where you think you might get fired, he wants you to feel like you could hear a pin drop you know? He's in his early fifties. Then he showed me a huge order where he had to ship a whole bunch of stuff separately to different locations and he was trying to calculate the weights and shipping costs of all these separate packages going to separate locations, and the non-AI solutions were failing him, so he copypasta'd into ChatGPT and it showed all its work, and he's like "Look at this!", and I just told him that he was having an appropriate emotional reaction because he's old and almost everyone younger than him is a boiling frog or as clueless as he was ten minutes ago. That kind of stuff gives me hope.
Particularly I wonder how you would shed all that heat in a vacuum? I keep seeing arguments that the cold of space would make data centers easier to cool, but this totally contradicts my observation that all the spacecraft I've read about have dedicated systems to shed excess heat produced by electronic systems, as opposed to, say, having dedicated heating systems for life support and never having to worry about electronic systems generating excess heat because the cold of space just takes care of it? E.g. Apollo as I understand it mostly relied on the waste heat of systems dedicated to other functions, the Apollo 13 crew only had to worry about cold because they eliminated virtually all sources of waste heat simultaneously to conserve electricity. I think Apollo's only dedicated systems for cabin temperature control were actually cooling systems. And this totally wouldn't apply in the data center scenario.
I feel like this quickly glosses over the hypothesis that gestural language evolved first, or that they evolved simultaneously with significantly more sophisticated gestural behavior evolving earlier. I believe gestural language is much older than 500 ka (up to, let's say, 2 Ma), which is consistent with the fossil evidence on vocalization adaptations.
It's undeniable that some of the cognitive changes that occurred during human evolution affected motivation; in fact, in my view, I think proto-curiosity and proto-patience would have been favored by selection quite early. On the other hand, in my view, sustainable, scalable joint attention and behaviorally modern imitation learning (e.g. overimitation) are more complex and would have required more than just motivational changes. In particular, I don't believe that most of the linguistic capability gap between chimps and humans can be explained as 'motivational hobbling.'
F5 in Old World monkeys is very likely homologous to Broca's area in humans, and although the gross neuroanatomy of humans and nonhuman primates is highly conserved, there are notable differences between the fine neuroanatomy of F5 in macaques and Broca's area. Chimp F5 has intermediate features, but the evidence here is limited since we don't do single-cell recordings in great apes anymore.
My own explanation for why there does not appear to be a derived gross language organ in humans is that F5 and Broca's area both generate and interpret hierarchical act strings as such. Such a scheme would have several continuous parameters responsive to selection, including hierarchy depth, hierarchy breadth, goal maintenance duration and goal switching speed. I think at various scales this system is general enough to generate and interpret (i.e. socially learn) act strings for flintknapping, gestural and vocal language, controlled fire use, etc. I think this explains why chimps can also learn to knap, but their tools are worse than habilis, and I think it also explains many of the specific linguistic limitations observed in apes using sign and lexigrams.
1
It seems like a bad sign that, even with maximally optimistic inputs, your model never falsely retrodicts intelligence explosions in the past.
Not as immediately obvious as reflexively avoiding a branch, but on my model, you ought to think of language (first gesture, then speech) as an exaptation of cognitive adaptations for flintknapping. F5 in Old World monkeys is an interface between abstract goal representations and concrete action sequences, and F5 is believed to be the homologue of Broca's area in humans. Looks like F5 generates and interprets hierarchical act strings as such (i.e. regardless of modality). You can hill climb on hierarchy depth, breadth, goal maintenance duration, switching speed (goals are active until completed, losing track of a goal being premature goal inactivation), which lends credibility to this idea. I think of Broca's area as F5 with more atomic action categories + big model smell.
I agree with this as an object-level observation on the usefulness of MAV3D itself, but also have a Lucas critique of the Bitter Lesson that ultimately leads me to different conclusions about what this really tells us.
I think of EURISKO, Deep Blue, and AlphaGo/Zero as slightly discordant historical examples that you could defy, but on my view they are subtle sources of evidence supporting microfoundations of cognitive returns on cognitive reinvestment that are inconsistent with Sutton's interpretation of the observations that inspired him to compose The Bitter Lesson.
EURISKO is almost a ghost story, but if the stories are true, then this doesn’t imply that N clever tricks would’ve allowed EURISKO to rule the world, or even that EURISKO is better classified as an AI as opposed to an intelligence augmentation tool handcrafted by Lenat to complement his particular cognitive quirks, but Lenat + EURISKO reached a surprising level of capability quite early. Eliezer seems to have focused on EURISKO as an early exemplar of cognitive returns on recursive self-improvement, but I don't think this is the only interesting frame.
It’s suggestive that EURISKO was written in Interlisp, as the homoiconic nature of LISPs might have been a critical unhobbling. That is to say, because Lenat’s engineered heuristics were LISP code, by homoiconicity they were also LISP data, an advantage that Lenat fearlessly exploited via macros, and by extension, domain specific languages. It also appears that Lenat implemented an early, idiosyncratic version of genetic algorithms. EURISKO was pretty close to GOFAI, except perhaps for the Search, but these descriptions of its architecture strongly suggest some intuitive appreciation by Lenat of something akin to the Bitter Lesson, decades before the coining of that phrase. It looks like Lenat figured out how to do Search and Learning in something close to the GOFAI paradigm, and got surprisingly high cognitive returns on those investments, although perhaps I have just made them seem a little less surprising. Of course, in my view, Lenat must not have fully appreciated the Lesson, as he spent the rest of his career working on Cyc. But for a little while at least, Lenat walked a fine line between the version of Engineering that doesn’t work, and the version that (kind of) does.
I would compare this distinction to the presence of folk semantics in all natural languages, and the absence of folk syntax. Parts of semantics are introspectively accessible to humans, so introspection permits veridical folk descriptions of semantics (informal, useful descriptions of what is true, possible, etc.), but the generator of syntax is introspectively inaccessible to humans, so generating veridical folk descriptions of syntax is much harder, if not impossible, via the same mechanism we applied in the case of folk semantics, thus successful computational modeling of syntax for the most part requires Science/Bayes (e.g. Linguistics). In my view, EURISKO was indeed mostly invented/discovered with Science/Bayes rather than Introspection, but this was hard for Lenat to tease out post mortem, and then he went way too far in the Introspection direction, failing to appreciate that most if not all of his cognitive returns came from mostly implicit, successful Science/Bayes (like mathematicians), which from the inside is hard to distinguish from successful Introspection. But Lenat's ostensible error does not explain away the cognitive returns observed in the case of EURISKO, if we have in fact observed any.
Deep Blue demonstrated significant cognitive returns from massively parallel alpha-beta pruning + engineered evaluation functions and opening/endgame heuristics. Arguably, these were functions as opposed to data, but if we maintain the LISP/FP mindset for a moment, functions and source code are data. I can squint at Deep Blue as an exemplar of ‘feature engineering’ ‘working’ i.e., large allocations of engineering effort on a ‘reasonable’ (to humans) timescale, in concert with unprecedentedly ambitious hardware allocation/specialization and parallelism, permitting cognitive returns on cognitive reinvestment to exceed a critical threshold of capability (i.e. beating Kasparov even once, possibly even on an off-day for Kasparov).
Crucially, not all engineered features are brittle (or even unscalable, with respect to a concrete capability target, which is my model of Deep Blue), and not all learned features are robust (or even scalable, again with respect to a concrete capability target, which is my model of how DQN didn't solve (i.e. meet the human expert capability threshold in the domain of) Go before AlphaGo (Go board state Search in ‘latent space’ was not ‘tractable’ with those permutations of compute, data, and (Learning) algorithm)), which might explain a thing or two about the weird competence profile of LLMs as well.
All of this to say, I don't think about cognitive returns in a way that demands a fundamentally sharp distinction between Learning and Engineering, even if it's been qualitatively pretty sharp under most historical conditions, nor do I think about cognitive returns in a way that forbids significant but reasonable amounts of pretty mundane engineering effort pushing capabilities past a critical threshold, and crucially, if that threshold is lethal, then you can die ‘without’ Learning.
As I hope might become especially clear in the case of AlphaGo/AlphaZero, I think the architectural incorporation of optimally specific representations can also significantly contribute to the magnitude of cognitive returns, as observed, I claim, in the cases of Deep Blue and EURISKO, where board states and domain specific languages respectively constituted strong priors on optimal actions when appropriately bound to other architectural modules (notably, Search modules in each case), and were necessary for the definition of evaluation functions.
I think a naive interpretation of the Bitter Lesson seems at first glance to be especially well-supported by the observed differences in capability and generality between the Alpha-series of architectures. You combine the policy and value networks into one, two-headed network, stop doing rollouts, and throw away all the human data, and it’s better, more capable and more general, it can perform at superhuman level in multiple perfect information, zero-sum, adversarial games besides Go (implicitly given their Rules in the form of the states, actions, and transition model of an MDP of course), and beat earlier versions of itself. But we also did the opposite experiments (e.g. policy head only Leela Chess Zero at inference time) and again an architecture with a Learning module but no Engineered Search module produced significantly lower cognitive returns than a similar architecture that did have an Engineered Search module.
Then MuZero wiped the board by tractably Learning the Rules in the latent space, but only four years after AlphaGo had reached the first target capability, and it still used Engineered Search (MCTS). To my knowledge, we are trying to build things that learn better search algorithms than we can engineer ourselves, but we still aren’t there. I’m not even claiming this wouldn’t work eventually, or that an engineered AGI architecture will be more general and more capable than a learned AGI architecture, I just think someone will build the engineered architecture first and then kill everyone, before we can learn that architecture from scratch and then kill everyone. On my model, returns on Search haven’t been a fundamental constraint on capability growth since right before EURISKO. On the other hand, returns on Engineered Game Rules (state space, action space, transition model), Compute, Data, and Learning have all been constraints under various historical conditions.
So I guess my model says that ‘merely static representations’ of semantic volumetric histories will constitute the first optimally specific board states of Nature in history, and we will use them to define loss functions so that we can do supervised learning on human games (recorded volumetric episodes) and learn a transition model (predictive model of the time evolution of recorded volumetric episodes, or ‘next-moment prediction’) and an action space (generative model of recorded human actions), then we will combine this with Engineered Search and some other stuff, then solve Go (kill everyone). Four years later something more capable and more general will discover this civilization from first principles and then solve Go, Chess, and Shogi with one architecture (kill everyone), and this trend will have been smooth.
If this isn’t pretty much exactly what Abram had in mind when he wrote:
Very roughly speaking, the bottleneck here is world-models. Game tree search can probably work on real-world problems to the extent that NNs can provide good world-models for these problems. Of course, we haven't seen large-scale tests of this sort of architecture yet (Claude Plays Pokemon is even less a test of how well this sort of thing works; reasoning models are not doing MCTS internally).
then I might have to conclude that he and I have come to similar conclusions for completely different reasons.
I feel like explicit versions of the microfoundations implied by naive interpretations of the Bitter Lesson would falsely retrodict that we couldn't beat Kasparov even once without ubiquitous adoption of ReLUs and late oughts/early teens amounts of compute, that DQN was sufficient to solve Go, and that EURISKO is a ghost story.
I didn’t talk about the neurobiology of constructive episodic simulation in humans at all, but would be willing to do so, and I think my model of that is also consistent with my microfoundations.
I view this from a heuristics and biases perspective. Mental health effects of LLMs seem well-explained as mediated by processing fluency as described by Schwarz et al. So it's in the halo effect, mere exposure, confirmation bias, availability bias, etc., cluster. If there is a meaningful uptick in mental illness that could be intervened upon, this view suggests some funny interventions like making LLM outputs harder to read with a blur or drop shadow, or chewing popcorn while interacting with LLMs. Also, Cognitive Reflection Test scores (which could change within lifetime, despite apparently being pretty stable) should negatively correlate with susceptibility to LLM-induced psychosis.
I'm excited to hear this, and I wanted to plug economist Gary Stevenson, who I think has the best combination of models, communications, and platform out of everyone I've seen speak or write on this issue. He's from the UK and has a weak focus there, but he talks about the US aplenty and recognizes that this is a global problem that will only get worse.