This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Animals were selected over millions of generations to effectively pursue external goals. So yes, they have external goals.
Humans also engage in within-lifetime learning, so of course you see all kinds of indicators of that in brains.
Both of those observations have high probability, so they aren't significant Bayesian evidence for "RL tends to produce external goals by default."
In particular, for this to be evidence for Richard's claim, you need to say: "If RL tended to produce systems that care about reward, then RL would be significantly less likely to play a role in human cognition." There's some update there but it's just not big. It's easy to build brains that use RL as part of a more complicated system and end up with lots of goals other than reward. My view is probably the other way---humans care about reward more than I would guess from the actual amount of RL they can do over the course of their life (my guess is that other systems play a significant role in our conscious attitude towards pleasure).