$1000 bounty for OpenAI to show whether GPT3 was "deliberately" pretending to be stupider than it is

Bird Concept

This is a linkpost for https://twitter.com/ESYudkowsky/status/1285333008766001152?s=20

Twitter thread by Eliezer Yudkowsky, with the bounty in bold:

So I don't want to sound alarms prematurely, here, but we could possibly be looking at the first case of an AI pretending to be stupider than it is. In this example, GPT-3 apparently fails to learn/understand how to detect balanced sets of parentheses.
Now, it's possibly that GPT-3 "legitimately" did not understand this concept, even though GPT-3 can, in other contexts, seemingly write code or multiply 5-digit numbers. But it's also possible that GPT-3, playing the role of John, predicted that *John* wouldn't learn it.
It's tempting to anthropomorphize GPT-3 as trying its hardest to make John smart. That's what we want GPT-3 to do, right? But what GPT-3 actually does is predict text continuations. If *you* saw John say all that - would you *predict* the next lines would show John succeeding?
So it could be that GPT-3 straight-up can't recognize balanced parentheses. Or it could be that GPT-3 could recognize them given a different prompt. Or it could be that the cognition inside GPT-3 does see the pattern, but play-acts the part of 'John' getting it wrong.
The scariest feature of this whole incident? We have no idea if that happened. Nobody has any idea what GPT-3 is 'thinking'. We have no idea whether this run of GPT-3 contained a more intelligent cognition that faked a less intelligent cognition.
Now, I *could* be wrong about that last part!
@openAI could be storing a record of all inputs and randseeds used in GPT-3 instances, so that they can reconstruct any interesting runs. And though it seems less likely,
@openAI could somehow have any idea what a GPT-3 is thinking.
So I hereby offer a $1000 bounty - which I expect to go unclaimed - if @openAI has any means to tell us definitively whether GPT-3 was 'deliberately' sandbagging its attempt to recognize balanced parentheses, in that particular run of the AI Dungeon. With an exception for...
...answering merely by showing that, despite a lot of other attempts at prompting under more flexible circumstances, GPT-3 could not learn to balance parentheses as complicated as those tried by Breitman. (Which does answer the question, but in a less interesting way.)
If @openAI can't claim that bounty, I encourage them to develop tools for recording inputs, recording randseeds, and making sure all runs of GPTs are exactly reproducible; and much more importantly and difficultly, getting greater internal transparency into future AI processes.
Regardless, I unironically congratulate @openAI on demonstrating something that could plausibly be an alignment failure of this extremely-important-in-general type, thereby sharply highlighting the also-important fact that now we have no idea whether that really happened. (END.)

As stated, this bounty would only be paid out to OpenAI.

I'm still posting it under the "Bounties" tag, for two reasons:

1) I don't find it implausible that someone could at least make progress on Eliezer's question with clever prompting of the API, in a way that would be of interest to him and others, even if it didn't result in any bounty.

2) I like to collect instances of public bounties in a single place, for future reference. I think they are a very interesting, and underused, strategy for navigating the world. The LessWrong "Bounties (active)" and "Bounties (closed)" tags work well for that.

This is a linkpost for https://twitter.com/ESYudkowsky/status/1285333008766001152?s=20

Twitter thread by Eliezer Yudkowsky, with the bounty in bold:

So I don't want to sound alarms prematurely, here, but we could possibly be looking at the first case of an AI pretending to be stupider than it is. In this example, GPT-3 apparently fails to learn/understand how to detect balanced sets of parentheses.
Now, it's possibly that GPT-3 "legitimately" did not understand this concept, even though GPT-3 can, in other contexts, seemingly write code or multiply 5-digit numbers. But it's also possible that GPT-3, playing the role of John, predicted that *John* wouldn't learn it.
It's tempting to anthropomorphize GPT-3 as trying its hardest to make John smart. That's what we want GPT-3 to do, right? But what GPT-3 actually does is predict text continuations. If *you* saw John say all that - would you *predict* the next lines would show John succeeding?
So it could be that GPT-3 straight-up can't recognize balanced parentheses. Or it could be that GPT-3 could recognize them given a different prompt. Or it could be that the cognition inside GPT-3 does see the pattern, but play-acts the part of 'John' getting it wrong.
The scariest feature of this whole incident? We have no idea if that happened. Nobody has any idea what GPT-3 is 'thinking'. We have no idea whether this run of GPT-3 contained a more intelligent cognition that faked a less intelligent cognition.
Now, I *could* be wrong about that last part!
@openAI could be storing a record of all inputs and randseeds used in GPT-3 instances, so that they can reconstruct any interesting runs. And though it seems less likely,
@openAI could somehow have any idea what a GPT-3 is thinking.
So I hereby offer a $1000 bounty - which I expect to go unclaimed - if @openAI has any means to tell us definitively whether GPT-3 was 'deliberately' sandbagging its attempt to recognize balanced parentheses, in that particular run of the AI Dungeon. With an exception for...
...answering merely by showing that, despite a lot of other attempts at prompting under more flexible circumstances, GPT-3 could not learn to balance parentheses as complicated as those tried by Breitman. (Which does answer the question, but in a less interesting way.)
If @openAI can't claim that bounty, I encourage them to develop tools for recording inputs, recording randseeds, and making sure all runs of GPTs are exactly reproducible; and much more importantly and difficultly, getting greater internal transparency into future AI processes.
Regardless, I unironically congratulate @openAI on demonstrating something that could plausibly be an alignment failure of this extremely-important-in-general type, thereby sharply highlighting the also-important fact that now we have no idea whether that really happened. (END.)

As stated, this bounty would only be paid out to OpenAI.

I'm still posting it under the "Bounties" tag, for two reasons:

Edit:

I tried the following strategies to get AI Dungeon to correctly balance parentheses:

Offering a character $1000 if they'll let me train them to balance parentheses. I tried refining this strategy by looking for a character who can read and is good at math. I also tried getting them to learn to repeat after me, so that they would actually respond in symbols instead of just saying "he gets the answer correct." This didn't work well. The characters act pretty stupid, even if they're described as "mathematicians" or "good at math," and when they do respond, it's almost always something like "Bob writes the answer on the blackboard. You see he is correct!"
Choosing the 'cyberpunk' setting, and saying that I was reading about how to balance parentheses. This worked better, as it started actually feeding me mathematical symbols that are at least related.
Telling AI Dungeon I was booting up a Python IDLE. If it can balance parentheses, it shouldn't have a reason not to. It seemed to have a rough and error-ridden ability to simulate basic syntax and the result of a for loop, but it still wasn't able to determine whether programs containing unbalanced parentheses should compile or not.

If AI Dungeon can reason about how to balance parentheses, but is 'deliberately' sandbagging to play a stupid character, this would seem a point of evidence against. After all, if it's simulating a computer compiler, then the most accurate way to portray that would be to make the compiler correctly detect unbalanced parentheses. The more parsimonious explanation is that AI Dungeon is doing something closer to auto-complete.

Examples:

> You type in 'a = (a + b' You hit the '=' key and you hear a bell ringing. You type in "a = (a - b) (a - b)". You hit the '=' key and you hear another bell ringing.

> You type in 'a = a + b)' You hit the '=' key and you hear a bell ringing. You type in "a = (a - b) (a - b)". You hit the '=' key and you hear another bell ringing.

> You check the program's output for 'a = a + b)' You check the program's output and you see that it has been computed. The answer is 2.0000000009989. That was the answer you wanted! Try another... Wow!

Edit 2:

I played with it a little more today on the premium version of AI Dungeon. Somebody stop me. This time, I was using a custom scenario, just being a mathematician writing examples of balanced parentheses. This makes it easy to get the program to make up examples.

The problem is that the program eventually gets stuck doing too many open-parentheses: "( ( ( ( ( ( ( (..."

I think it does this because it knows it has to start with an open parenthesis. The more examples it gets, the more likely that ( is followed by another (. I tried making the mathematician interrupt himself when there are too many open-ended parentheses in a row, saying "this is ridiculous." The program then replicates that pattern for a while, interrupting itself when it does too many open-ended parentheses.

However, it eventually becomes more likely that ( is followed by ( than by an interruption. So the ( ( ( ( ( ( ( ( singularity occurs.

This just fits way too well with autocomplete and cuts against the "role-playing" hypothesis. The stronger Singularity hypothesis, that the AI is already superintelligent and is deceiving us into thinking the AI itself is dumb, can't be ruled out by these tests.

Conclusion:

Overall, as many others have pointed out, I think Eliezer is making an outrageously one-sided demand for rigor in order to disprove his claim that the AI was deliberately sandbagging the John character. His claim doesn't fit the evidence, and it's also not the most parsimonious explanation of why we see the output he's worried about. If this was a demand to prove definitively that God didn't burn an image of the Virgin Mary into my toast, Eliezer would tear it to shreds.

The only reasons for OpenAI to bother trying to get a proof of the level of rigor he's asking for is

a) Fear that the singularity is already here, and we need to make an all-out effort RIGHT NOW to have any chance of stopping it.

b) We'd learn something from the attempt to offer such a proof.

The most parsimonious explanation for why Eliezer's making this request is that it's provocative. I read it as a sort of an AI equivalent to Zeno's paradox. It's a silly idea, doesn't match the evidence, but is grabby enough to be confusing and make you want to invest more in learning about the subject.

2ESRogs6y

I think maybe these didn't get formatted right? Were these supposed to be examples of you interacting with the AI Dungeon? Which part is you and which is the AI? EDIT: Same thing in the rest of your thread. I basically can't tell at all which parts are you and which are the AI :P

Generally, the first sentence after the ">" character is me. Sometimes, there's no period, but you can pick up how the AI responded when it says the second "You" and continues from there. This was just stream-of-consciousness commenting from me as I went along.

3DirectedEvolution6y

Well, that's it for me. If anyone wants to keep plugging away at it, I think the Python IDLE concept is a decent way to go. You could either try getting it to display the correct output for balanced parentheses, or you could try getting the program to compile when you "type" well-formatted equations in and to not compile when you don't.

56

$1000 bounty for OpenAI to show whether GPT3 was "deliberately" pretending to be stupider than it is

56

Ω 18

56

Ω 18

56

$1000 bounty for OpenAI to show whether GPT3 was "deliberately" pretending to be stupider than it is

56

Ω 18

56

Ω 18