Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Epistemic status: Speculation with some factual claims in areas I’m not an expert in.

Thanks to Jean-Stanislas Denain, Charbel-Raphael Segerie, Alexandre Variengien, and Arun Jose for helpful feedback on drafts, and thanks to janus, who shared related ideas.

Main claims

  • GPTs’ next-token-prediction process roughly matches System 1 (aka human intuition) and is not easily accessible, but GPTs can also exhibit more complicated behavior through chains of thought, which roughly matches System 2 (aka human conscious thinking process).
  • Human will be able understand how a human-level GPTs (trained to do next-token-prediction) complete complicated tasks by reading the chains of thought.
  • GPTs trained with RLHF will bypass this supervision.

System 2 and GPTs’ chains of thought are similar

A sensible model of the human thinking process

Here is what I feel like I’m doing when I’m thinking:

Repeat

  1. Sample my next thought from my intuition
  2. Broadcast this thought to the whole brain[1]

When you ask me what is my favorite food, it feels like some thoughts “pop” into consciousness, and the following thoughts deal with previous thoughts. This is also what happens when I try to prove a statement: ideas and intuitions come to my mind, then new thoughts about these intuitions appear.

This roughly matches the model described in Consciousness and the Brain by Stanislas Dehaene, and I believe it’s a common model of the brain within neuroscience.[2]

How GPTs “think”

Autoregressive text models are performing the same kind of process when they generate text. Sampling text is using the following algorithm:

Repeat:

  1. Do a forward pass, and sample the next token from the output distribution
  2. Add the generated token to the input. It makes it part of the input for the next forward pass, which means it can be used by lots of different attention heads, including specialized heads at earlier layers.

This looks similar the human thinking process, and the rest of the post will be exploring this similarity and draw some conclusion we might draw from this.

System 2 and GPTs’ chains of thought have similar strengths

A theoretical reason

The sample + broadcast algorithm enables the execution of many serial steps. Neurons take about 1ms to fire[3], which means that if a thought takes 200ms to be generated, the brain can only do 200 serial operations. This is similar to the hundredths of serial matrix multiplications GPT-3 does in its 96 layers[4]. But by sampling and thought and broadcasting it, both GPT-3 using chains of thought[5] and the brain using System 2 can in principle implement algorithms which require much more serial steps.

More precisely, in a Transformer, the number of serial steps used to generate one token is  (no matter the prompt length), whereas the number of serial steps used to generate  tokens is  (where one step is one path trough a block made of an attention layer, an MLP, and a residual connection).

As you can see in the figure below, each path the information can take to generate the first token is exactly  long, but paths to generate the last token can be up to  long.

(Connections skipping more than 2 positions were omitted to increase readability.)

The actual number of FLOPs used is the same when generating  tokens from an  token prompt and when generating  tokens from and  token prompt because a large prompt enables more parallel computations to take place[6]. But parallel computations are not always the bottleneck, and in particular, I think that planning to deceive humans and taking control is the kind of task which requires a large number of serial steps.

Empirical evidence

  • Chains of thought helped complete tasks which human need System 2 for: for example, humans struggle to answer quantitative reasoning questions with System 1, and succeed with System 2. Similarly, Minerva was greatly helped by chain of thought prompting.
  • Chains of thought didn’t help much in tasks human do well with System 1 alone: for example, the best human translators translate using entirely System 1. Similarly, I’m not aware of anyone having success in making LLM’s translation abilities better through chain of thought prompting

GPT-3 is currently struggling to solve some task with chain of thought prompting that human can easily do with System 2, such as adding together 20 digit numbers together (with a scratch pad). But I would be surprised if this kind of limitations weren’t drastically reduced by using chains of thought on bigger GPTs. I also expect GPTs without chains of thought to remain really bad at the of tasks which human can’t to do at all with System 1, even with a lot of training.

On the other hand, I expect GPTs to get better at tasks humans solve with System 1 alone, and that these kinds of tasks won’t be solved more efficiently with chain of thought prompting.[7]

I’m open to ideas about how to make precise bets out of these vague predictions.

Human-level GPTs trained to do next-token-prediction will probably have transparent thoughts

Deception and escaping supervision are easier to do with System 2

As a human, trying to deceive five-year olds about your intention would probably be easy. Deceiving them if they can watch every thought of yours right from the beginning would probably be extremely challenging, because your evil intentions would be spotted if you ever thought of ways to escape supervision.

In general, I believe that escaping supervision and deception are the kind of task which is much easier to do with System 2 than with System 1. A reason beyond the intuition given by the scenario with 5-year-olds with a mind reader is that generating complex counterfactuals requires many serial steps, something only System 2 can do well.

If the models around AGI look remotely like GPTs, given that System 2 and chains of thought have similar strength, I expect models around AGI to be able to think about deception “in plain sight” (using chains of thought), but to be too weak to plan about deceiving humans without actually putting any deception-relevant information inside the tokens in the chains of thought.

Chains of thought of human-level GPTs will be transparent by default

Definition

A human-level GPT is a model trained to do next-token-prediction that is around human performance at all text-related tasks when prompted to use chains of thought. In particular, the process of generating one token will probably be much weaker than the process of generating an answer after a 1000-token long chain of thought, so a human-level GPT might be not that good at reasoning within the generation of one token.

Claim

Because deception with System 1 only is so hard, I think that inside human-level GPTs, there won’t be parts of the network able to plan about way to deceive human or build a way to encode hidden information inside the chains of thought in a single forward pass.

In the framework of Simulators, the smartest simulacra inside a human-level GPT will probably be too dumb to generate and immediatly start acting uppon a dangerous plan without using the power of the sample + broadcast algorithm.

Argument

With a model only trained to do next-token-prediction, the model can only use the logic and patterns of the data it has been trained on, which doesn’t contain ways to carry out chains of thought not understandable by human beings trying hard to decipher them.[8]

Therefore, hiding information inside the chain of thought requires the model to find a reason and a way to hide information.

Because understanding why and how to hide information in the chains of thought requires many serial steps, it is very hard to do in a single forward pass, and thus, human-level GPTs won’t be able to start the process of using sampling and broadcasting to hide information in chains of thoughts, and won’t be able to generate great deceptive plans.

GPTs will reach AGI before being able to build a deceptive plan in one forward pass

A reasonable assumption is that GPTs will find it easier to do good Alignment Research with chains of thought and the power of sample + broadcast, than to understand that hiding information in the chains of thought is useful (and build a scheme to do steganography[9]) in one forward pass.

Then you could use the first GPTs to reach AGI - too dumb to plan about ways to murder you without actually writing thoughts about it - to do Alignment research, and hopefully solve Alignment in the general case where you don’t have easy access to models’ thoughts.

You would then use this first AGI while monitoring the generated chains of thought (though scaling this oversight would be challenging). If your first AGI had once in a while thoughts about murdering you or hiding information in chains of thoughts in order to bypass your supervision, you could just restart the generation process. If it was too frequent, you would have good evidence that it is a good idea to shut it off (and this evidence would enormously help to convince all governments to prevent anyone from building AGI as long as we suck at Alignment).

Reinforcement learning makes thoughts opaque

(A caricature of) How humans learn to think

It seems that the brains use a mixture of reinforcement learning and self-supervised learning to learn to do things[10]. I can see how this might work for learning to think (using System 2): you hear people speak, and thereby build an inner representation of language which allows you to speak yourself (this is like GPTs self-supervised learning). But I expect other sources of feedback to shape how we think:

  1. sometimes, people give us feedback about what we say after we have thought a little bit about what to say in our head. This leads us to speak in a way that differs from the distribution of language we read and listen to. This is close to some form of RLHF with a hidden scratch pad.
  2. sometimes, some negative consequence happens because we have thought about something. If my understanding of the brain is correct, this should lead the “thought sampling procedure” to change its output based on how useful thoughts are. This is close to some form of direct RLHF (without a hidden scratch pad).

RL in the human brain contributes to make human thoughts hard to decipher

Self-deception[11] is what happens when your unconscious processes “know something” (or act as if they know it, it’s hard to tell what it means for an unconscious process to “know something), but your conscious self doesn’t. If someone had a mind reader, you wouldn’t be able to lie, but self-deception could still help you to take adversarial actions. If you managed to escape the supervision of 5-year-olds with a mind reader, I would bet you had done so using self-deception.

However, self-deception rarely appears through self-supervised learning alone. It can happen (for example, if you unconsciously imitate people who underestimate how late they will be) but it intuitively seems that “self-lies” learned through self-supervised learning are limited in scope and strength. Therefore, I expect that a human-level GPT trained with self-supervised learning would have an even harder time escaping human supervision than a human would escaping 5-year-olds with a mind reader.

On the other hand, reinforcement learning is a natural culprit for the existence of self-deception: self-deception is useful if you are a human because it allows you to not waste energy lying to people on purpose and makes you a better liar, thus it will be reinforced by reinforcement learning. Moreover, reinforcement learning could encourage hidden meaning in thoughts if the reward was given based on what the mind reader detected.

RLHF could make GPTs’ thoughts hard to decipher

Applying reinforcement learning to mimic of human think it tempting, but by using RLHF, you lose the nice properties described above which make oversight of the thinking process of GPTs possible:

  1. Hidden scratch pads don’t need to be human understandable: even human can’t always understand each other scratch pads, and it might be much worse with entities with a radically different architecture;
  2. Even if the feedback is done directly on the chains of thought, the model has incentives to hide information inside the output tokens because being able to use the sample + broadcast algorithm on information the oversight process doesn’t want the AI to use will be rewarded (if some of the forbidden information is useful). Hiding information could also be useful to compress the amount of thoughts present in the chains of thought. Thus, RLHF is likely to teach GPT the use of steganography.

When it comes to successful AI oversight, the “cherry on the cake” might be deadly.

Training on text generated by models trained with RLHF is enough to make GPTs’ thoughts opaque

If you train the next GPT model, you might have a lot of data generated by other GPTs trained with RLHF in your training data, either because there is Instruct GPT data on the web, or you because you added it on purpose to add many examples of helpful completions.

This is very likely to make the chains of thought of your model opaque because if the model trained with RLHF learned steganography, your model will need to learn it in order to predict what it will say. This removes most of the safety guarantees which I believe you have with a GPT model trained only on human data: even if the RLHF model didn’t use its steganographic abilities to conquer the world, maybe the larger model you are training will generalize the use of steganography beyond its original use if prompted in a way which hints towards it. This means it will be able to use the sample + broadcast algorithm to think about deceptive plans without being spotted, by hiding information in its generations.[12]

This is not mitigated by removing harmful completions from the training corpus: if there is steganography in current models trained with RLHF, it might be present even in completions which look helpful, honest and harmless. The only way to mitigate this I know of is to not include data generated by models trained with RLHF in the training data of new GPTs.

How these claims could be wrong

Here is an incomplete list of how the claims above could be wrong. I don’t think the strength of any of them is strong enough to discard the validity of the argument above, but the conjunction of them makes me 

  • GPTs trained with some kind of next-token-prediction task might not scale to AGI, which would make this analogy not relevant for AGI safety.
  • GPTs with chains of thought might not be as strong as I expect it: next-token-prediction might not be a good enough task to teach an AI to use sample + broadcast, making these systems much worse at using System 2 that humans are (given a System 1 of similar capabilities).[13]
  • Human might suck at prompt engineering: maybe GPTs are able to match System 2 in principle, but we might not be able to find the prompts to make the most of it.
  • GPTs might hide significant information in prompts by default.[14]
  • It could be easy for simulacras inside GPTs to figure out a great way and a good motivation to use steganography within a single forward pass.
  • It could be easy for simulacras inside GPTs to figure out a great way and a good motivation to take over the world within a single forward pass.
  • There could be reasons for GPTs trained with self-supervised to learn how to do steganography I didn’t think of.
  • There could be ways to prevent GPTs fine-tuned with RLHF to learn steganography.

I’m excited about attempts to figure out which of the above are right, or attempts to make some of the above wrong.


 

  1. ^

    Communication with senses and your motor controls happens somewhere in step 1, either without conscious control, or as a direct consequence of the broadcasting step that preceded. I also ignore other ways in which parts of the brain can interact, such as hormonal communication.

  2. ^

    It also looks like this post by Kaj_Sotala, which describes System 2 as being composed of System 1 processes.

  3. ^

    See synaptic delay, Britannica.

  4. ^

    Given that the number of layer in GPTs grows roughly as the fourth square root of compute, the order of magnitude won’t change within the next ten year if the architecture remains the same. This matters because I’m not sure why chain of thoughts would be inherently more powerful than 1-token prediction with a very large prompt if models were not bottleknecked by the number of serial operations.

  5. ^

    In this post, “chains of thought” also include answers which include many tokens, for which you can think while you answer. The equivalent of “Pure System 1” in GPTs would be a single token answer.

  6. ^

    This is why the OpenAI API charges you an amount of money proportional to prompt length + number of tokens generated.

  7. ^

    I expect GPTs to benefit from chain of thought prompting in areas where human use System 1 with parts of the brain GPTs autoregressive training didn’t built, such as vision. For example, I wouldn’t be surprised if chain of thought prompting helped GPTs understand ASCII art.

  8. ^

    Of course, some internet text is like that. But I expect it to be such a minority of internet’s text that it wouldn’t matter in practice (GPTs needs a lot of text to justify investing is special “hide reasoning in text” machinery, right?). I’m interested if anyone works on either detecting this kind of text to remove it from training corpuses, or investigating if the claim that GPTs needs a lot of text to justify investing resources into a complicated algorithm is true.

  9. ^

    Here is a description of what steganography is and what it could look like in the context of LLMs, by A Ray.

  10. ^

    See here for a deeper dive about RL and the brain by lukeprog.

  11. ^

    More details about self-deception in these articles by Paul Christiano: If you can't lie to others you must lie to yourself, and The Elephant in the brain

  12. ^

    This is also much worse than examples of steganography given by humans because there can be much more RLHF-generated text, and because the context in which steganography by human is present is very different from the context in which future GPTs will be used, while the prompts with which Instruct GPT is used are very close to the prompts with which future GPTs will be.

  13. ^

    For examplen System 2 seems to have a “working memory”, which doesn’t have an equivalent object in GPTs. Maybe GPTs need something more powerful than chains of thought to match human System 2 performance. 

    I would love to know an example of a task GPTs can’t do because of this lack of working memory, or a task GPTs can do much better than humans because they aren’t bound by a working memory of small and fixed size.

  14. ^

    For example using the start of a completions to figure out what the context is, and then not using as much processing power to figure it out is a way of using chains of thought which is not completely transparent to humans, and it could be the case there are more significant ways in which GPTs hide information in chains of thought by default.

63

Ω 32

New Comment
17 comments, sorted by Click to highlight new comments since: Today at 4:52 PM

I respect that you said at the start this was speculation.

You establish that aspects of human cognition and GPT are roughly similar, but then begin making strong conclusions about GPT using this lossy analogy.

This is akin to me trying to do mechanistic interpretability on a transformer by studying how humans "pay attention" to things.

The argument isn't "human = GPT, human use information X in chains of thought, therefore GPT use information X in chains of thought", which is what you seem to suggest.

I think I can make a version of this argument which doesn't rely on the human analogy at all:

  1. You can't do advance deception and mesa optimization in <1k serial steps
  2. The reason why chains of though works at all is not because GPTs put information there on purpose, but rather because humans put useful information there, GPTs imitate this, and then it can use information it has put there. It would be surprising if it encoded "by accident" information in a way humans can't understand
  3. You can do human level AI research with <1k serial step per forward pass with chains of thought
  4. Therefore, GPTs will be able to do human level AI research with GPTs not powerful enough to do hidden deception

Step 2 is not as crisp as I would like it, but it's definitely not an argument as weak as "GPT=humans".

Edited: Tried to stay more on point.

With respect to your argument.

I am assuming this is an LLM that hasn't been trained with RL.

  1. Ok, you probably can't do advanced deception with a bounded automata, agreed.

  2. There could be any number of mechanisms that mean that GPT is getting information from words and context that isn't readily apparent to a human overseer.

  3. I don't actually know if this true but it is plausible.

  4. I do not buy this conclusion at all.

The first statement was a claim about "advanced" deception in systems without CoT. Once a system is powerful enough to perform human level AI research it is not at all apparent it isn't going to be capable of doing some form of steganography, which is not necessarily "advanced" but is still good enough to fool humans.

(Forgive how long this is, have a lot of thoughts on this subject!)

The first statement was a claim about "advanced" deception in systems without CoT. Once a system is powerful enough to perform human level AI research it is not at all apparent it isn't going to be capable of doing some form of steganography, which is not necessarily "advanced" but is still good enough to fool humans.

It seems plausible to me that, although a GPT model that is capable[1] of learning to use chains-of-thought to do human-level AI research would be capable of learning to use chains-of-thought to deceive oversight[2] if it were trained on those kinds of chains-of-thought (or maybe if it were trained with RL), this highly-specific pattern of behavior[3] will probably not in fact be entrained when doing supervised learning on webtext alone or on webtext + human-level AI research chains-of-thought.

If that is true, if such a GPT model did not learn how to use the generated tokens to support (N x #layers)-step deception, then it would be limited to #layers-step deception. According to Assumption 1, this is not sufficient to actually pull off advanced deception. So unless for some reason it deploys its human-level AI research chain-of-thought process towards discovering[4] effective constant-time "hidden" deception mechanisms, and unless it additionally uses some method to incorporate[5] those discovered mechanisms back into its weights, I don't see a viable path for this GPT model to end up deceptive based on Assumptions 1-3.

EDIT: I'm realizing there's a lingering question along the lines of "What if the model learns to do steganography without CoT?". The reason is that for training a GPT model, we teacher-force the token targets, so the inputs to the model at the next position are screened off from the influence of the outputs from the model at the current position. So there's no way for the model to "benefit" from steganography during training. Plus, passing information that way creates a massive internal coordination problem: the "decoder" circuits need to change in lockstep with the "encoder" circuit, whereas for non-steganographic communication between circuits, this happens automatically through the unembedding+embedding matrices.

  1. ^

    Pretend that we know the model has requisite background knowledge, was trained on data containing lots of explicit multistep reasoning, and has a bunch of spare capacity.

  2. ^

    Doing hidden multi-step cognition that is not "in plain sight" via the outputted tokens.

  3. ^

    Similarly, I wouldn't be surprised if you could fine-tune a GPT model that was pretrained on webtext to decode ROT13 strings using chain-of-thought, but I would be very surprised if it picked up the behavior of deploying CoT for decoding ROT13 strings based on the webtext pretraining alone.

  4. ^

    Why is it looking for these, in the first place? What was the cognitive stacktrace that produced this behavior from the circuits that were entrained by the training task?

  5. ^

    How would this happen? A GPT model can generate the string "I should set my output logit bias parameters to -2", but that doesn't thereby actually set its own output logit bias parameters to -2, which is what it'd need to do to translate a discovered constant-time "hidden" deception mechanism into its weights. I don't know of any existing or proposed setups where this is supported.

Interesting post!

  • Human will be able understand how a human-level GPTs (trained to do next-token-prediction) complete complicated tasks by reading the chains of thought.

What about LMs trained with say, in-fill objectives (like UL2, GLM, T5, etc.?

I'm curious what you think about instruction finetuning or RLHF via expert distillation, both of which can be done using a next-token prediction objective - do you think this would still lead to intepretable cognition? Would this be closer to NPT on all text or RL?

If so, what's the difference between RLHF and instruction finetuning or expert distillation? Why can't we cast all RL tasks into "imitate the good trajectories using next token prediction"?

What about BoN selection? As in, take a reward function, sample N trajectories, and then select the best? If you sample best of N, the expected cost of this policy is at most N times the expected cost of the base policy. At comparable levels of competence, do you expect sampling BoN on a human imitator to be safer than an RLHF'ed or expert distilled model? 

Where does prompt engineering fall into the spectrum between RL and NPT on web corpora?


I also have a few questions/confusions about the headline claim:

With a model only trained to do next-token-prediction, the model can only use the logic and patterns of the data it has been trained on, which doesn’t contain ways to carry out chains of thought not understandable by human beings trying hard to decipher them.[8]

Human text output contains not just the thoughts of people, but the results of significant amounts of thinking. For example, an answer sheet for a multiple choice math test does not contain the cognition used to generate the answers. Shouldn't this create an incentive for sequential foundation models (like GPTs) to embody more and more thoughts that humans do via system 2 into its "system 1", so to speak?

I also expect GPTs without chains of thought to remain really bad at the of tasks which human can’t to do at all with System 1, even with a lot of training.

I agree that it seems unlikely that GPTs in the near future will do zero shot stenography, but why do you expect that GPTs will fail to incorporate system 2 logic into its system 1? Is this mainly a competitiveness reason? (I.e. if GPT-5 can do a task that requires system 2 capabilities without CoT, then GPT-4 can do the same task with CoT, and so the first dangerous AIs built via GPT + CoT  will rely on CoT to do the system 2 tasks?)

However, self-deception rarely appears through self-supervised learning alone. It can happen (for example, if you unconsciously imitate people who underestimate how late they will be) but it intuitively seems that “self-lies” learned through self-supervised learning are limited in scope and strength. Therefore, I expect that a human-level GPT trained with self-supervised learning would have an even harder time escaping human supervision than a human would escaping 5-year-olds with a mind reader.

Given that GPT is imitating humans who do deceive each other and themselves, including many who are much more sophisticated liars than a 5 year old + mind reader, I'm not sure why you're so confident that we'd get so much transparency by default.
 


My main contention with the post is that I'm not sure that chain of thought necessarily works the way it intuitively does for humans, in the same way that prompting may not work the same way that prompting humans does! For example, a recent paper did automatic CoT prompts for a bunch of datasets, and some of the CoT examples from the AQuA dataset are pretty horrifying:

Q: A car finishes a journey in 20 hours at the speed of 60 km/hr. If the same distance is to be covered in 10 hours, how much speed does the car gain? Answer Choices: (A) 80 kmph (B) 50 kmph (C) 120 kmph (D) 70 kmph (E) 80 kmph 
A: Let’s think step by step. The car is currently travelling at 60 km/hr. In order to travel the same distance in 10 hours, the car would need to travel at 6 times the speed, or 360 km/hr. Therefore, the car would need to gain 300 km/hr. The answer is C. The answer is C. 

Q: Mark told John "If you give me half your money I will have Rs.75. John said, "if you give me one third of your money, I will have Rs.75/-. How much money did John have ? 
Answer Choices: (A) 22 (B) 60 (C) 28 (D) 26 (E) 18 A: Let’s think step by step. If Mark has Rs.75, then John has Rs.75 as well. If Mark gives John half his money, then Mark will have Rs.37.50 and John will have Rs.112.50. However, this isn’t one of the answer choices. If Mark gives John one third of his money, then Mark will have Rs.50 and John will have Rs.100. This is one of the answer choices, so John must have had Rs.100 to begin with. The answer is D.

A prompt including these two incorrect reasoning example can have better performance over some human engineered CoT prompts

 

[EDIT: see also gwern's comments below]

A prompt including these two incorrect reasoning example can have better performance over some human engineered CoT prompts.

Yes. An even better example would be a different paper showing that you can simply shuffle all the answers in the few-shot prompt to improve over zero-shot. This was tweeted with the snark 'how is few-shot learning "meta-learning" if the answers don't need to be right?' But of course, there's no problem with that. All meta-learning is about is about solving the problem, not about respecting your preconceived notions of how the model 'ought' to compute. (Many optimal strategies, such as exact ones obtained by dynamic programming, can look bizarre to humans, so that's not a good criteria.)

This is because contrary to OP, a GPT model is not doing something as simple as reasoning explicitly through text or System I pattern-matching. (Or at least, if it is, the meaning of those two things are much broader and more opaque than one would think and so merely replaces an enigma with a mystery.) A GPT model doing few-shot learning is doing meta-learning: it is solving a (very large) family of related tasks using informative priors to efficiently and Bayes-optimally infer the latent variables of each specific problem (such as the agent involved, competence, spelling ability etc) in order to minimize predictive loss. Including example questions with the wrong answers is still useful if it helps narrow down the uncertainty and elicit the right final answer. There are many properties of any piece of text which go beyond merely being 'the right answer', such as the a priori range of numbers or the formatting of the answer or the average length or... These can be more useful than mere correctness. Just as in real life, when you see a confusing question and then see an incorrect answer, that can often be very enlightening as to the question asker's expectations & assumptions, and you then know how to answer it. ('Bad' prompts may also just stochastically happen to tickle a given model in a way that favors the 'right' family of tasks - think adversarial examples but in a more benign 'machine teaching' way.)

Prompts are not 'right' or 'wrong'; they are programs, which only have meaning with respect to the larger family of prompts/tasks which are the context in which they are interpreted by the neural computer. Since you still don't know what that computer is or how the prompt is being interpreted, you can't say that the baseline model is reasoning how you think it is based on a naive human-style reading of the prompt. It obviously is not reasoning like that! RLHF will make it worse, but the default GPT behavior given a prompt is already opaque and inscrutable and alien.

(If you don't like the meta-learning perspective, then I would point out the literature on LMs 'cheating' by finding various dataset biases and shortcuts to achieve high scores, often effectively solving tasks, while not learning the abilities that one assumed was necessary to solve those tasks. They look like they are reasoning, they get the right answer... and then it turns out you can, say, remove the input and still get high scores, or something like that.)

I broadly agree, though I haven't thought enough to be certain in either view. 

Yes. An even better example would be a different paper showing that you can simply shuffle all the answers in the few-shot prompt to improve over zero-shot.

Yeah, I thought about this result too, though I couldn't find it quickly enough to reference it. 

What's in your view the difference between GPTs and the brain? Isn't the brain also doing meta-learning when you "sample your next thought"? I never said System 1 was only doing pattern matching. System 1 can definitely do very complex things (for example, in real time strategy game, great players often rely only on System 1 to take strategic decisions). I'm pretty sure your System 1 is solving a (very large) family of related tasks using informative priors to efficiently and Bayes-optimally infer the latent variables of each specific problem (but you're only aware of what gets sampled). Still, System 1 is limited by the number of serial steps, which is why I think our prior on what System 1 can do should put a very low weight on "it simulates an agent which reasons from first principles that it should take control of the future and finds a good plan to do so". 

If your main point of disagreement is "GPT is using different information in the next than humans" because it has been found that GPT used information humans can't use, I would like to have a clear example of that. The one you give doesn't seem that clear-cut: it would have to be true that human do worse when they are given examples of reasoning in which answers are swapped (and no other context about what they should do), which doesn't feel obvious. Humans put some context clues they are not consciously aware of in text they generate, but that doesn't mean that they can't use them.

Btw, this framing is consistent with the fact that humans have personalities because they are "tuned with RL": they experienced some kind of mode collapse very similar to the one seen in Instruct GPT, which lead to certain phrasing and thoughts to get reinforced. Human personality depends on how you have been raised, and is a bit random, like mode collapse. (But it's postdiction, so not worth many Bayes points.)

Thanks for your detailed feedback!

 

I don't see a strong conceptual difference between GPTs and models trained with in-fill objectives. In-fill objectives have some advantages for some tasks (like the tasks used to pretrain them), but they have the same properties when it comes to the number of sequential steps in one in-fill step vs multiple sequential in-fills. Like GPTs, they aren't easy to understand in one in-fill step, but I expect sequential in-fills to be close to human thoughts, though I haven't seen exotic examples of sequential in-fills which make use of the fact that you can fill text in any order. (Fill-in models are also harder to train at scale, right? Do some people still expect these kinds of model to play a role in the road to AGI?)

Instruction fine-tuning is fine-tuning a prompt across many questions, right? I've heard that they are often not human-understandable, which is not very reassuring. (The prompt might encode increasingly detailed reasons and ways to escape supervision, just as RL could hypothetically learn a hard-coded deception strategy? To be honest, I thought more about the power of GPTs at inference time than at training time, where the framing in terms of sequential steps breaks down.)

What do you mean by RLHF via expert distillation?

The big difference I see between NTP and RL is that in RL the model has opportunities to create "coordination with itself", which can take the form of hard-coded deception plans which don't require much thought, or steganography techniques (which is really challenging if you never had a chance to plan for it in advance, but really easy if you are trained with RL with wrong incentives).

At comparable levels of competence, I expect sampling BoN on a human imitator to be much safer than an RLHF'ed or expert distilled model, because there isn't this opportunity for the model to create "coordination with itself".

Same for prompt engineering, especially if your search process looks like "look by hand for things that would help humans" and not like "use gradient descent to find a good prompt", where you have way less control over the information that is used by the model (adversarial attacks that push the right knobs inside the model look might be the way to get the most performance, but it also means you don't know what these knobs are). Still looks better than pure RL since it only helps a little to create "coordination with itself".

But yeah, there is a big spectrum between pure NTP and pure RL, and I don't have enough conceptual clarity to draw a clear boundary.

 

I agree that it seems unlikely that GPTs in the near future will do zero shot stenography, but why do you expect that GPTs will fail to incorporate system 2 logic into its system 1? Is this mainly a competitiveness reason?

The core claim only holds for competitiveness reason. I would expect a "Dyson Sphere GPT" to be far beyond humans at everything using System 1 alone. But I still need the assumption that "things hard for human's System1 are hard for GPTs' System 1". This one is wrong for some task where GPTs' have ungodly amount of training (like "a vs the" or pure NTP), but I expect it to hold for all actually relevant task because the number of serial steps is of the same order of magnitude, and humans can roughly match GPTs width given enough training. For example, I expect that if some human try hard to solve multiple choice math tests using their intuition (for example, having to answer just after it has been read to them at 2x speed), they will crush GPTs until ~AGI.

Given that GPT is imitating humans who do deceive each other and themselves, including many who are much more sophisticated liars than a 5 year old + mind reader, I'm not sure why you're so confident that we'd get so much transparency by default.

I feel like mimicking liars doesn't teach you to lie well enough to fool humans on purpose. I might have been mistaken to bring up self-deception. Knowing how to do "simple lies" like "say A when you know that B" is quite useful in many parts of Webtext, but I can't think of kinds of text where you actually need to do "think in your lie" (i.e. put information in your lie you will use to think about your ulterior motives better). Humans don't have to do that because for complex lies, humans have the luxury of not having to say everything they are thinking about! Therefore, I would be surprised there was enough Webtext to teach you the kind of deception strategy which you need to pull off to plan within the prompt while being watched with models not strong enough to do human-level AI Research. Please tell me if you can think of a kind of text where lies where you "think through your lies" would be useful for NTP!

 

A prompt including these two incorrect reasoning example can have better performance over some human engineered CoT prompts. 

Thanks for these surprising CoT example!

I'm not sure how surprising it is that CoT examples with bad reasoning makes the model generate good CoT, nor how relevant it is. For example, I find it more troubling that prompt optimization finds incomprehensible prompts. But on the other hand, I have never seen models successfully generate absurd but useful CoT, and I expect they won't be able to because they haven't been trained to generate CoT useful to themselves. Therefore, I expect that the only way model can generate CoT useful to themselves is to generate CoT useful to humans (which they ~know how to do), and use the information humans find useful they have put in the CoT. I would be surprised if high likelihood CoT really useful to humans were also really useful to models for drastically different reasons.

(Note: figuring out the kind of reasoning would be useful is not that easy, even for humans, and I'm not sure teachers are great at figuring out how to best explain how to reason. I wouldn't be surprised if 7-graders had better performances with AQuA horrible examples than with some human engineered prompts.)

Loose analogy based reasoning over complex and poorly understood systems isn't reliable. There is kind of only one way for GPT-n to be identical to system 1, and many ways for it to be kind of similar, in a way that is easy to anthropomorphize, but has some subtle alien features. 

GPTn contains some data from smart and/or evil humans and humans speaking in riddles or making allusions. Lets suppose this generalizes, and now GPTn is pretending to be an IQ200 cartoon villain, with an evil plot described entirely in terms of references to obscure sources. So when referring to DNA, it says things like "two opposites of the same kind, two twins intertwined, a detectives assistant and an insect did find. Without an alien friend."

Or maybe it goes full ecologist jargon. It talks about "genetically optimizing species to restore population levels to re-balance ecosystems into a per-anthropogenic equilibrium".  Would an army of minimum wage workers spot that this was talking about wiping out almost all humans? 7

Actually, wouldn't a naive extrapolation of internet text suggest that superhumanly complicated ideas were likely to come in superhumanly dense jargon?  

I mean, if you have a team of linguists and AI experts carefully discussing every sentence, then this particular problem goes away. The sort of operation where, if the AI produces a sentence of Klingon, you are flying in a top Klingon expert before you get the next sentence. But how useful could gpt-n be if used in such a way? On the other extreme, gpt-n is producing internal reasoning text at a terabyte/minute. All you can do with it is grep for some suspicious words, or pass it to another AI model. You can't even store it for later unless you have a lot of hard drives. Potentially much more useful. And less safe.

I feel that there is one more step in my thinking:

Repeat

  1. Sample several possible next thoughts from my intuition in the form of not well articulated ideas
  2. Choose the one which I accept based on its moral, truth or beauty and put in properly articulated language 
  3. Broadcast this thought to the whole brain[1]

I feel like the different possible thoughts happen sequentially, I think about something wrong, then something right (or in general, I think sequentially about things, it's not always about wrong and right). Also, I would bet that if you could measure that under IRM, the broadcasting would happen on wrong, then on right.

  1. sample wrong and right
  2. choose
  3. broadcast

Looks much more like a succession of sample and broadcast:

  1. sample wrong
  2. broadcast wrong
  3. sample right (because your brain often samples multiple different things sequentially, especially if the first one doesn't "feel right")
  4. broadcast right
  5. sample "I recognize this second thought as right"
  6. broadcast "I recognize this second thought as right" (which strengthened right and discards wrong, and prompts parts of the brain to use right for further thoughts)

I've been thinking along very similar lines, and would probably generalize even further:

Hypothesis: All DNNs thus far developed are basically limited to system-1 like reasoning.

I share a pretty similar model. Though I don't think I share your likelihood assessments ("RLHF is likely to teach GPT the use of steganography.", "This is very likely to make the chains of thought of your model opaque because if the model trained with RLHF learned steganography, your model will need to learn it in order to predict what it will say. ", "Deceiving them if they can watch every thought of yours right from the beginning would probably be extremely challenging,"). That may stem from just having a different set of intuitions about what RL(HF) does & about how the sample + broadcast mechanism interacts with circuits.

[+][comment deleted]2mo 10