Epistemic status: Shower thoughts / I have an idea, but not much knowledge
While the the smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads, the largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.
I would expect that with increased model size it will be possible to increase the attention field by a lot without much need for additional AI insight.
In a guide for AI dungeon there's a description of a pin item that serves as the context for the story. GPT-3 seems to be good at understanding the goals that are set in the pin item to the point that it tries to achieve them faster then the person who wrote the guide desires.
If the attention field would be larger it would allow for multiple pin items and each pin item could be even larger.
In AI dungeon GPT-3 can't influence the content of the pin items but it would be possible to give the GPT-3 the ability to use console commands to write into pin items to be able to have memory abilities similar to the short-term and middle term memory that humans have.
The ability to interact with the unix console was already shown Natural Language Shell example on the OpenAI website.
At the beginning a resulting agent could be mentored like AI dungeon has mentoring of the AI. If the AI is able to query Google via the console, I would imagine that it could be effective at many tasks.
A lot of newspaper articles could be written by such an agent who scouts the internet for other information that's published on the topic and synthesizes available information. Cyberwar could also be done by such agents.
I'd be happy to hear what other people think about such an agent.
To engage more directly with what you're suggesting: You seem to be suggesting that the AI could read text, figure out the most important part, and feed it to itself to improve further outputs. Without a highly supervised step that not only figures out the most important part but optimizes it to be used effectively by the AI, I don't think that would be effective. Already the AI tends to get "stuck"--to pick up on a particular phrase or pattern as very important and keep repeating it--in situations what that's unexpected or unwanted.
Epistemic status: Shower thoughts / I have an idea, but not much knowledge
While the the smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads, the largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.
I would expect that with increased model size it will be possible to increase the attention field by a lot without much need for additional AI insight.
In a guide for AI dungeon there's a description of a pin item that serves as the context for the story. GPT-3 seems to be good at understanding the goals that are set in the pin item to the point that it tries to achieve them faster then the person who wrote the guide desires.
If the attention field would be larger it would allow for multiple pin items and each pin item could be even larger.
In AI dungeon GPT-3 can't influence the content of the pin items but it would be possible to give the GPT-3 the ability to use console commands to write into pin items to be able to have memory abilities similar to the short-term and middle term memory that humans have.
The ability to interact with the unix console was already shown Natural Language Shell example on the OpenAI website.
At the beginning a resulting agent could be mentored like AI dungeon has mentoring of the AI. If the AI is able to query Google via the console, I would imagine that it could be effective at many tasks.
A lot of newspaper articles could be written by such an agent who scouts the internet for other information that's published on the topic and synthesizes available information. Cyberwar could also be done by such agents.
I'd be happy to hear what other people think about such an agent.