Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.
This is a linkpost for https://docs.google.com/document/d/1_GqOrCo29qKly1z48-mR86IV7TUDfzaEXxD3lGFQ8Wk/edit
Thanks for this! I think it is a well-written and important critique. I don't agree with it though, but unfortunately I am not sure how to respond. Basically you are taking a possibility--that there is some special sauce architecture in the brain that is outside the space of current algorithms & that we don't know how to find via evolution because it's complex enough that if we just try to imitate evolution we'll probably mess up and draw our search space to exclude it, or make the search space too big and never find it even with 10^44 flops--and saying "this feels like 50% likely to me" and Ajeya is like "no no it feels like 10% to me" and I'm like "I'm being generous by giving it even 5%, I don't see how you could look at the history of AI progress so far & what we know about the brain and still take this hypothesis seriously" But it's just coming down to different intuitions/priors. (Would you agree with this characterization?)
Thanks for the comment! I agree with this characterization. I think one of the main points I was trying to make in this piece was that as long as the prior for "amount of special sauce in the brain (or needed for TAI)" is a free parameter, the uncertainty in this parameter may dominate the timelines conversation (e.g. people who think that it is big may be basically unmoved by the Bio-Anchors calculation), so I'd love to see more work aimed at estimating it. (Then the rest of my post was an attempt to give some preliminary intuition pumps for why this parameter feels relatively big to me. But I think there are probably better arguments to be made (especially around whether TAI is in the space of current algorithms), and I'll try to write more about it in the future.)
(BTW, I'm really sorry for the slow reply! I still haven't figured out how to juggle replying to things on LW in a reasonable time frame with not getting too distracted by reading fun things whenever I log onto LW...)
Nice. I agree with your point about how uncertainty in this parameter may dominate timelines conversation. If you do write more about why the prior on special sauce should be large (e.g. whether TAI is in the space of current algorithms), I'd be interested to read them! Though don't feel like you have to do this--maybe you have more important projects to do.
(No rush! This sort of conversation doesn't have a time limit, so it's not hurting me at all to wait even months before replying. I'm glad you like LW. :) )
One small thing: When you first use the word "power", I thought you were talking about energy use rather than computational power. Although you clarify in "A closer look at the NN anchor", I would get the wrong impression if I just read the hypotheses:
... TAI will run on an amount of power comparable to the human brain ...
... neural network which would use that much power ...
Maybe change "power" to "computational power" there? I expect biological systems to be much more strongly selected to minimize energy use than TAI systems would be, but the same is not true for computational power.
This is a linkpost for a review of Ajeya Cotra's Biological Anchors report (see also update here) that I wrote in April 2022. It's since won a prize from the EA criticism and red-teaming contest, so I thought it might be good to share here for further discussion.
Here's a summary from the judges of the red-teaming contest:
Note that a link to a summary/review of the book Principles of Deep Learning Theory on page 8 has been moved here: More Recent Progress in the Theory of Neural Networks.