If you teach an AI to fish, it might optimize its performance within a narrow scope. Teach it to teach itself to fish, and you've created a recursively self-improving AGI that is unaligned with human values by default and will most likely end up killing us all.
-- Eliezer, probably
It reminds me of what I told my kids about mathematicians being lazy and inventing ways of skipping work. For example, instead of repeated addition, find tricky ways to do multiplication. Or that fractions are just delayed divisions - we postpone doing the division as long as possible - including many multiplications on the way too (canceling down). Or think of summation rules.
If you teach a man to fish, he’ll be eating fish for a lifetime. Teach him to teach himself to fish, and you’ve given him the fundamental life skill of figuring things out on his own.