Authors, Lawyers, Teachers, Researchers, Doctors, Coders. These are just some people who've found that their work can be performed dramatically better with the right implementation of GPT-3/ Large Transformer models. Due to this, I poist that access to the best AI models will become as fundamental to productivity of most...
Imagine typing the following meta-question into GPT-4, a revolutionary new 20 Trillion parameter language model released in 2021: "I asked the superintelligence how to cure cancer. The superintelligence responded __" How likely are we to get an actual cure for cancer, complete with manufacturing blueprints? Or will we get yet...
Authors, Lawyers, Teachers, Researchers, Doctors, Coders. These are just some people who've found that their work can be performed dramatically better with the right implementation of GPT-3/ Large Transformer models. Due to this, I poist that access to the best AI models will become as fundamental to productivity of most jobs, as access to a computer and internet is today.
Unlike computers, large transformers cost millions to train and run- thus there is a far greater risk of centralisation, control and surveillance. OpenAI has announced that they will be closely monitoring access and cancelling access to anyone indulging in harmful usage. Yet this kind of power to exclude somebody from a basic tool... (read 461 more words →)
Artificial eXpert Intelligence and Artificial Super Intelligence. Sorry for not being clear- edited title to be more obvious.
What I'm going for here is that the few shot examples show GPT-3 is a sort of DWIM AI (Do What I Mean): something it has implicitly learned from examples of humans responding to other human's requests. It is able to understand simple requests (like unscramble, use word in sentence, etc.) with about as much data as a human would need: understanding the underlying motive of the request and attempting to fulfill it.
On firefighters, the instruction would work even with children who had never seen a fire, but only heard its attributes verbally described ("hot... (read more)
Imagine typing the following meta-question into GPT-4, a revolutionary new 20 Trillion parameter language model released in 2021:
"I asked the superintelligence how to cure cancer. The superintelligence responded __"
How likely are we to get an actual cure for cancer, complete with manufacturing blueprints? Or will we get yet another "nice sounding, vague suggestion" like "by combining genetic engineering and fungi based medicine"- the sort GPT-2/3 is likely to suggest?
The response depends on whether GPT focuses on either:
1. What GPT thinks humans think that the superintelligence would say; or
2. Using basic reasoning, solve for what the character (an actual superintelligence) would say if this scenario were playing out in real life.
If GPT takes... (read 1001 more words →)
The big jump in performance between the zero shot and few shot setting in arithmetic and other non-linguistic reasoning tasks[esp. 3D- & 3D+] is why I think it is almost certain #2 is true. Few shot inference relies on no further training [unlike fine tuning], so the improvement in 'pattern recognition' so to speak is happening entirely at inference. It follows that the underlying model has general reasoning abilities, i.e. the ability to detect and repeat arbitrary patterns of ever increasing complexity, that occur in its input (conditioning) data.
Interestingly, the model fails to completely learn 4D and 5D arithmetic, where its zero-shot scores were really low. However few shot inference does show... (read 353 more words →)
I feel the few shot arithmetic results were some of the most revolutionary results we've received this decade. Learning math from nothing but verbal examples shows that we have an agent that can reason like us. The progression to complex algebra seems inevitable, with nothing but more parameters.
Assumption #2 is entirely correct, which is why few shot examples matter. The system is literally smart enough to figure out what addition is with those 50 examples. I bet most of us took longer than 50 examples.
" But #2 is wild: it would represent a kind of non-linguistic general intelligence ability which would be remarkable to find in a language model."
Yes, it does represent... (read more)
The data for GPT-2 has been replicated by the open source OpenWebText project. To my knowledge the same dataset was utilised for GPT-3, so accessing it is not a problem.
The parallelizability of GPT-3 is something I've been looking into. The current implementation of zero-2 seems like the best way to memory-optimally train a 170B parameter transformer model.