This is a special post for quick takes by daijin. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I recently did an RL competition on kaggle. With enough time spent in RL and teaching (the former, a month, the latter, over 7 years on a weekly basis), children (and adults who learn) start to look like RL agents.
I noticed long reward horizons (periods where you are heading to a long term reward with no rewards in between) as a barrier to progress in my RL competition, then a friend mentioned she wanted to build a career but it felt unfathomable to commit that huge amount of time towards it, even putting aside uncertainty.
Long reward horizons show up elsewhere. To oceanfare, you have fish, but to go to space you have to endure huge expanses of no rewards.
Intrinsic rewards don't solve this - being motivated to explore for exploration's sake leads to perverse outcomes. The closest thing that solves this is persistent out-of-band memory of trajectories to reach the edge of ones knowledge. In geography this is a map. In knowledge space this is a pedagogical map.