I think that taking a theoretical computer science perspective to GPT-3 might help clarify some part of the debate.

My background is in theoretical computer science, and that's my prefered perspective to make sense of the world. So in this post, I try to rephrase the task we ask GPT-3 to solve in terms of search and optimization problems, and I propose different criteria for solving such a task. I thus consider GPT-3 as a black-box, and only ask about how to decide if it is good at its task or not.

Epistemic status: probably nothing new, but maybe an interesting perspective. This is an attempt to understand the intuitions behind different statements about whether GPT-3 "solves some task".

The $P r o m p t_{p}$ Search Problem

Let's fix a language (English for example). Then a very basic description of the task thrown at GPT-3 is that it's given a prompt $p$ and it needs to give a follow-up text $x$ such that $x$ is a good answer to the prompt. For example, $p$ might be a question about parenthesis or addition or a story to complete, and $x$ is GPT-3's answer. I thus define $P r o m p t_{p}$ to be this search problem for a given prompt $p$ .

Now, what does it mean for an algorithm $A$ to solve $P r o m p t_{p}$ ? Multiple possibilities come to mind:

(Deterministic Solution) For every execution $t$ of $A$ , such that the value of $A$ for prompt $p$ at the end of $t$ is $A_{t} (p)$ , we have $A_{t} (p) \in P r o m p t_{p}$ . This criterion has the advantage for simplicity, but it's also completely unrealistic -- even a human might fail to give a decent answer to a prompt.
(Nondeterministic Solution) There is an execution $t$ of $A$ such that satisfies $A_{t} (p) \in P r o m p t_{p}$ . Here, we only require that $A$ might give a correct answer, without any other guarantees. This is obviously too weak, as a monkeys bashing on keyboards will eventually give a good answer -- yet are not "solving it".
(Probabilistic Solution) For a meaningful distribution on the executions of $A$ (for example a uniform distribution or a distribution based on simplicity), we have $P r [A_{t} (p) \in P r o m p t_{p}] > l$ , for some $l \in] 0, 1 [$ . This looks more realistic, as it allows for a margin of error. What about the value of $l$ ? I think we can go with anything $> 1 / 2$ , just like in Probabilistic Turing Machines: it ensures that we can make the probabilty of success as high as we want by repeating the call to $A$ a polynomial number of times (So a po

...

Computer Science