It's sad that agentfoundations.org links no longer work, leading to broken links in many decision theory posts (e.g. here and here)

I've been asked to clarify a point of fact, so I'll do so here:

My recollection is that he probed a little and was like "I'm not too worried about that" and didn't probe further.

This does ring a bell, and my brain is weakly telling me it did happen on a walk with Nate, but it's so fuzzy that I can't tell if it's a real memory or not.  A confounder here is that I've probably also had the conversational route "MIRI burnout is a thing, yikes" -> "I'm not too worried, I'm a robust and upbeat person" multiple times with people other than Nate.

In private correspondence, Nate seems to remember some actual details, and I trust that he is accurately reporting his beliefs.  So I'd mostly defer to him on questions of fact here.

I'm pretty sure I'm the person mentioned in TurnTrout's footnote.  I confirm that, at the time he asked me, I had no recollection of being "warned" by Nate but thought it very plausible that I'd forgotten.

What's "denormalization"?

Cool! It wrote and executed code to solve the problem, and it got it right.

Are you using chat-GPT-4?  I thought it can't run code?

Interesting, I find what you are saying here broadly plausible, and it is updating me (at least toward greater uncertainity/confusion).  I notice that I don't expect the 10x effect, or the Von Neumann effect, to be anywhere close to purely genetic.  Maybe some path-dependency in learning?  But my intuition (of unknown quality) is that there should be some software tweaks which make the high end of this more reliably achievable.

Anyway, to check that I understand your position, would this be a fair dialogue?:

Person: "The jump from chimps to humans is some combination of a 3x scaleup and some algorithmic improvements.  Once you have human-level AI, scaling it up 3x and adding a chimp-to-human-jump worth of algorithmic improvement would get you something vastly superhuman, like 30x or 1000x Von Neumann, if not incomparable."

Vivek's model of Jacob: "Nope. The 3x scaleup is the only thing, there wasn't much algorithmic improvement.  The chimp-to-human scaling jump was important because it enabled language/accumulation, but there is nothing else left like that.  There's nothing practical you can do with 3x human-level compute that would 30x Von Neumann^[1], even if you/AIs did a bunch of algorithmic research."

I find your view more plausible than before, but don't know what credence to put on it.  I'd have more of a take if I properly read your posts.

1. ^{^}

   I'm not sure how to operationalize this "30x-ing" though.  Some candidates:

   - "1000 scientists + 30 Von Neumanns" vs. "1000 scientists + 1 ASI"

   - "1 ASI" vs. "30 Von Neumanns"

   - "100 ASIs" vs. "3000 Von Neumanns"

In your view, who would contribute more to science -- 1000 Einsteins, or 10,000 average scientists?^[1]

"IQ variation is due to continuous introduction of bad mutations" is an interesting hypothesis, and definitely helps save your theory.  But there are many other candidates, like "slow fixation of positive mutations" and "fitness tradeoffs^[2]".

Do you have specific evidence for either:

1. Deleterious mutations being the primary source of IQ variation
2. Human intelligence "plateauing" around the level of top humans^[3]

Or do you believe these things just because they are consistent with your learning efficiency model and are otherwise plausible?^[4]

Maybe you have a very different view of leading scientists than most people I've read here?  My picture here is not based on any high-quality epistemics (e.g. it includes "second-hand vibes"), but I'll make up some claims anyway, for you to agree or disagree with:

• There are some "top scientists" (like Einstein, Dirac, Von Neumann, etc).  Within them, much of the variance in fame is incidental, but they are clearly a class apart from merely 96th percentile scientists.  1000 {96%-ile-scientists} would be beaten by 500 {96%-ile-scientists} + 100 Einstein-level scientists.
• Even within "top scientists" in a field, the best one is more than 3x as intrinsically productive^[5] as the 100th best one.

1. ^{^}

   I'm like 90% on the Einsteins for theoretical physics, and 60% on the Einsteins for chemistry

2. ^{^}

   Within this, I could imagine anything from "this gene's mechanism obviously demands more energy/nutrients" to "this gene happens to mess up some other random thing, not even in the brain, just because biochemistry is complicated".  I have no idea what the actual prevalence of any of this is.

3. ^{^}

   What does this even mean?  Should the top 1/million already be within 10x of peak productivity?  How close should the smartest human alive be to the peak?  Are they nearly free of deleterious mutations?

4. ^{^}

   I agree that they are consistent with each other and with your view of learning efficiency, but am not convinced of any of them.

5. ^{^}

   "intrinsic" == assume they have the same resources (like lab equipment and junior scientists if they're experimentalists)

Human intelligence in terms of brain arch priors also plateaus

Why do you think this?