Humans assume that humans continue a story based on deeper ideas, not surface words. So for example, if we started a story these two different ways:
"a girl named little red riding hood lived with her mother and father. one day she baked some cookies and decided to bring them to her grandmother who lived on the other side of the deep dark forest" ...
"once upon a time, a little girl lived with her parents. one day she made some food to bring to her mother's mother who lived on the other side of a large, ominous woods" ...
Our bias is that a human understands both prompts as being fairly equivalent in terms of the ideas they contain, so they would complete them in fairly similar ways. Humans might expect other humans to complete the story based on the general ideas, and not in a way that is highly sensitive to how the prompt is worded. This might or might not be true for GPT.
A tangible way to control for this kind of human bias, would be to try multiple rewordings/rewritings of the same prompt when examining the way that GPT generates the text after the prompt.
Humans assume that humans continue a story based on deeper ideas, not surface words. So for example, if we started a story these two different ways:
"a girl named little red riding hood lived with her mother and father. one day she baked some cookies and decided to bring them to her grandmother who lived on the other side of the deep dark forest" ...
"once upon a time, a little girl lived with her parents. one day she made some food to bring to her mother's mother who lived on the other side of a large, ominous woods" ...
Our bias is that a human understands both prompts as being fairly equivalent in terms of the ideas they contain, so they would complete them in fairly similar ways. Humans might expect other humans to complete the story based on the general ideas, and not in a way that is highly sensitive to how the prompt is worded. This might or might not be true for GPT.
A tangible way to control for this kind of human bias, would be to try multiple rewordings/rewritings of the same prompt when examining the way that GPT generates the text after the prompt.