Thanks, good suggestion. I've updated the title.
I'm interested in estimating how many 'OOMs of compute' span the human range. There are a lot of embedded assumptions there, but let's go with them for the sake of a thought experiment.
Cortical neuron counts in humans have a standard deviation of 10 - 17%, depending on which source you use. Neuron counts are a useful concrete anchor that I can relate to AI models.
There are many other factors that account for intelligence variation among humans. I'd like to construct a toy model where those other factors are backed out. Put another way - if intelligence variation was entirely explained by neuron count differences, how much larger would the standard deviation of the... (read 276 more words →)
There is an old (2013) paper from Google here that mentions training an ngram model on 1.3T tokens: ("Our second-level distributed language model uses word 4-grams. The English model is trained on a 1.3 × 10^12 token training set"). An even earlier 2006 blog post here also references a 1T word corpus.
This number is 2x as big as MassiveWeb, more than a decade old, and not necessarily the whole web even back then. So I would be quite surprised if the MassiveWeb 506B token number represents a limit of what's available on the web. My guess would be that there's at least an order of magnitude more tokens available in a full... (read more)
Thanks for the feedback, exactly the kind of thing I was hoping to get from posting here.
I have thought about a multiplicative model for intelligence, but wouldn't the fact we see pretty-close-to-Gaussian results on intelligence tests tend to disconfirm it? Any residual non-Gaussianity seems like it can be explained by population stratification, etc, rather than a fundamentally non-Gaussian underlying structure. Also, like you say the polygenic evidence also seems to point to a linear additive model being essentially correct.