Now observe that human children can learn to make very reliable predictions. So they must be doing some sort of science.
Danger! You're not looking at the whole system. Children's knowledge doesn't just come from their experience after birth (or even conception), but is implicitly encoded by the interplay between their DNA, the womb, and certain environmental invariants. That knowledge was accumulated through evolution.
So children are not, in their early development, using some really awesome learning algorithm (and certainly not a universally applicable one); rather, they are born with a huge "boost", and their post-natal experiences need to fill in relatively little information, as that initial, implicit knowledge heavily constrains how sensory data should be interpreted and gives useful assumptions that help in modeling the world.
If you were to look only at what sensory data children get, you would find that it is woefully insufficient to "train" them to the level they eventually reach, no matter what epistemology they're using. It's not that there's a problem with the scientific method (though there is), or that we have some powerful Bayesian algorithm to learn from childhood development, but rather, children are springboarding off of a larger body of knowledge.
You seem to be endorsing the discredited "blank slate" paradigm.
A better strategy would be to look at how evolution "learned" and "encoded" that data, and how to represent such assumptions about this environment, which is what I'm attempting to do with a model I'm working on: it will incorporate the constraints imposed by thermodynamics, life, and information theory, and see what "intelligence" means in such a model, and how to get it.
(By the way, I made essentially this same point way back when. I think the same point holds here.)
Re: "If you were to look only at what sensory data children get, you would find that it is woefully insufficient to "train" them to the level they eventually reach, no matter what epistemology they're using."
That hasn't been demonstrated - AFAIK.
Children are not blank slates - but if they were highly intelligent agents with negligible a-priori knowledge, they might well wind up eventually being much smarter than adult humans. In fact that would be strongly expected - for a sufficiently smart agent.
You know what to do.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.