This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In particular, humans prominently do reinforcement learning.
Humans care about lots of things in reality, not just certain kinds of cognitive-update-signals.
"RL -> high chance of caring about reality" predicts this observation more strongly than "RL -> low chance of caring about reality"
This seems pretty straightforward to me, but I bet there are also pieces of your perspective I'm just not seeing.
But in particular, it doesn't seem relevant to consider selection pressures from evolution, except insofar as we're postulating additional mechanisms which evolution found which explain away some of the reality-caring? That would weaken (but not eliminate) the update towards "RL -> high chance of caring about reality."
Humans also engage in within-lifetime learning, so of course you see all kinds of indicators of that in brains.
I don't see how this point is relevant. Are you saying that within-lifetime learning is unsurprising, so we can't make further updates by reasoning about how people do it?
Both of those observations have high probability, so they aren't significant Bayesian evidence for "RL tends to produce external goals by default."
I'm saying that there was a missed update towards that conclusion, so it doesn't matter if we already knew that humans do within-lifetime learning?