When you are presented with a new concept, the first step is to "mechanically" learn it. At that point, you are able to solve only questions that closely match what you were taught.
The next step is to really understand the meaning of the concept, in a deeper sence. In school, this is usually achieved by providing excercises that are progressively harder and harder. Harder in this case means that the questions diverges more from the learnt material and more and more requires deeper understanding of the concept.
If you only go so far as learning the "mechanical" part of the concept, you usually will forget about it rather quickly. If you, on the other hand, really understand the meaning behind the concept, the knowledge will stick much longer. And not only that, it is okey to forget about the mechanical steps and only keep the basic understanding as you can quickly and easily look up the mechanical part when you recognize a problem.
I think this is all a normal learning process, practised everyday in school.
You touch into a topic that is all too common in the first half, and that is the problem of definitions. It is not unusual to find people having an argument over something, without first doing a clear definition of the question.
For example, what is intelligence, and what is conscience? There are lots of discussions about the possibility or impossibility to create artificial versions of these, without first having a common definition. Such an argument is almost a waste of time, except for the situation where it may lead to better understanding of definition.
Another famous example is the question of the meaning of life. Define life first, and I think the question will be easier.
Yet another example is the Chinese Room (a human not understanding Chinese will take a chinese question and use a set of written rules to produce an answer in chinese). The question is then, can this construction be considered to understand chinese? This is a very hot discussion, but I see no attempt to first define what you mean with "understand chinese".
I can't help thinking about the the computer that produced the answer 42, because the question wasn't exact enough. While quite funny the first time I read it, people still ask questions that way.
I have two observations, one personal and one general:
Once, I tried to apply artificial neural nets on the task to evaluate positional situations in the game of Go. I did a very basic error, which was to train the net only on positive examples. The net quickly learned to give high scores for these, but then I tested on bad situations it still reported high scores. Maybe a little naive mistake, but you have to learn sometimes.
A very common example is testing of software. Usually, people pay much attention on testing the positive cases, and verifying that they work as they should. Less time is spent on testing things that should not work, sometimes resulting in programs that generates answers when it should not. The problem here is that testing the positive cases usually consists of a limited set, while the negative cases are almost infinite.