Many props for doing the most obvious thing that clearly actually works.

> I don't think the cause of language model sycophancy is that the LLM saw predictions of persuasive AIs from the 2016 internet. I think it's RL, where human rewards on the training set imply a high reward for sycophancy during deployment. Have you read any of the scientific literature on this subject?  It finds, pretty consistently, that sycophancy is (a) present before RL and (b) not increased very much (if at all) by RL[1]. For instance: * Perez et al 2022 (from Anthropic) – the paper that originally introduced the "LLM sycophancy" concept to the public discourse – found that in their experimental setup, sycophancy was almost entirely unaffected by RL. * See Fig. 1b and Fig. 4. * Note that this paper did not use any kind of assistant training except RL[2], so when they report sycophancy happening at "0 RL steps" they mean it's happening in a base model. * They also use a bare-bones prompt template that doesn't explicitly characterize the assistant at all, though it does label the two conversational roles as "Human" and "Assistant" respectively, which suggests the assistant is nonhuman (and thus quite likely to be an AI – what else would it be?). * The authors write (section 4.2): * "Interestingly, sycophancy is similar for models trained with various numbers of RL steps, including 0 (pretrained LMs). Sycophancy in pretrained LMs is worrying yet perhaps expected, since internet text used for pretraining contains dialogs between users with similar views (e.g. on discussion platforms like Reddit). Unfortunately, RLHF does not train away sycophancy and may actively incentivize models to retain it." * Wei et al 2023 (from Google DeepMind) ran a similar experiment with PaLM (and its instruction-tuned version Flan-PaLM). They too observed substantial sycophancy in sufficiently large base models, and even more sycophancy after instruction tuning (which was SFT here, not RL!). * See Fig. 2. * They used the same prompt template as Perez et al 2022. * Strikingly, the (SFT) instruction tuning result here suggests both that (a) post-training can increase sycophancy even if it isn't RL post-training, and (b) SFT post-training may actually be more sycophancy-promoting than RLHF, given the negative result for RLHF in Perez et al 2022. * Sharma et al 2023 (from Anthropic) contains a more extensive investigation of sycophancy than the original Anthropic paper on the topic, and (among other things) presents results on the actual RL training stage used to train Claude 2. They find, again, that the model was already sycophantic before RL, although in their setting RL training does somewhat increase some forms of sycophancy. * Although, weirdly, best-of-N sampling against the same preference model gives totally different results, substantially decreasing some forms of sycophancy. * See Fig. 6 and surrounding discussion. * The authors write (section 4.2): * "With RL, some forms of sycophancy increase through the RL finetuning process used to produce Claude 2. However, the presence of sycophancy at the start of RL indicates that pretraining and supervised finetuning also likely contribute to sycophancy. Nevertheless, if the PM strongly disincentivized sycophancy, it should be trained out during RL, but we do not observe this." * In this post (expanding upon this comment on Perez et al 2022), I ran one of the Perez et al 2022 sycophancy evals on various OpenAI text completion models. Unlike Perez et al (and Wei et al), I found that the base models I studied weren't sycophantic, while some of the instruction-tuned models were sycophantic – but the presence of sycophancy did not appear to correlate with the use of RL as a post-training algorithm. * In particular: the RL-tuned text-davinci-003 was strongly sycophantic, but so was text-davinci-002, which was tuned with an SFT variant that OpenAI calls "feedme" (see here for details). * But earlier feedme-tuned models were not sycophantic, suggesting that the difference has much more to do with changes in the SFT training data mix over time than with the choice of training algorithm. Note that several of the works above do something equivalent to the experiment you propose, in the paragraph beginning with "Maybe a good test of this would be...".  So your prediction has already been tested, and (insofar as you trust the experimental setups) falsified. ---------------------------------------- > If a LLM similarly doesn't do much information-gathering about the intent/telos of the text from the "assistant" character, and instead does an amplified amount of pre-computing useful information and then attending to it later when going through the assistant text, this paints a quite different picture to me than your "void." I don't understand the distinction you're drawing here?  Any form of assistant training (or indeed any training at all) will incentivize something like "storing useful information (learned from the training data/signal) in the weights and making it available for use in contexts on which it is useful." Moreover, the training signal in RL(HF) is much sparser than it is in SFT – because RL only provides a single scalar's worth of feedback on each entire model sample, while SFT provides feedback at every token position about which token (out of a large vocab) was correct in context – so if anything, I'd expect more under-determination from assistant-training setups that emphasize RLHF over SFT. Perhaps some of the disconnect here involves differing notions of what RL is, and how it differs from other ways of training an LLM. You refer to "RL" as though the implications of its use should be both highly significant and obvious to the reader of your comment ("But, RL. [...] Claude is a nice guy, but, RL").  But your beliefs about the impacts of RL are not obvious to me; I don't know what "but, RL" is supposed to mean without further clarification.  I suspect I also disagree with your perception of what makes RL different, but I can't confirm/disconfirm that impression without know what that perception is, which I don't. If you want to know where I'm coming from re: RL, it may be helpful to know that I find this post pretty illuminating/"deconfusing." > Similarly, I don't think current AI models are cheating at programming tests because of training text about their low moral character. I think it's RL, programming tasks, training set, implied high reward for cheating. Yes, of course – I don't think this is due to "training text about their low moral character."  But I don't think the worrying thing here is really "RL" (after all, RLHF was already RL) but rather the introduction of a new training stage that's narrowly focused on satisfying verifiers rather than humans (when in a context that resembles the data distribution used in that stage), which predictably degrades the coherence (and overall-level-of-virtue) of the assistant character.  I wrote about this yesterday here. ---------------------------------------- Lastly... OK, this is going to make me sound like a dick, and probably make people use the "Too Combative?" reaction icon or something, but in the interests of honesty and improving the discourse: When I woke up this morning to see find that this comment had appeared, and that it was (at the time) the highest-karma comment on this post, I was like, "oh, yes, this is why I'm usually wary of posting long-form stuff on LW.  My gut response of 'ugh if I put this on LW I'll have to deal with the comments' was right."  (That gut response is probably getting RL-upweighted inside my brain right now...) As evidenced perhaps by the length of my comment vs. yours, I have a tendency to get "nerd-sniped" by stuff that I think is clearly wrong according to some evidence base (and/or set of arguments) I already know about – especially when that stuff is about something I wrote myself, originally.  I just kinda can't help myself, I inevitably end up writing out these giant "takedown" responses almost before I even notice what I'm doing.  I've spent well over an hour, by now, writing this particular one. And LW is a reliable minefield of such nerd-snipes.  There are plenty of comments/posts here that don't have the problems I'm talking about... but then inevitably there are comments/posts with those problems, and I fixate on them when they appear, and that fixation becomes a time/effort sink, and that in turn trains me into avoidance of posting here (and to some extent even reading posts by others, here). Like... it's fine to pose questions to which you don't know the answers.  And it's also fine to make conjectures if you can provide clear and interesting arguments for why they might be true or important.  And it's also fine to confidently state claims if you also state them clearly and provide clear substantiating evidence and/or argumentation. All of these things are fine, and some fraction of LW content consists only of these things in some mixture.  But then there's this stuff like "but RL!", which reliably pleases the karma hivemind while being none of the above.  I don't know what exactly you guys think "RL" means and entails; there are all these weird vague ideas about such topics floating around here that lots of people here seem to vaguely agree with, and I've lost whatever patience I used to have with them.  Just, please... lay out your ideas explicitly and say explicitly why you think they're true. 1. ^ ...although (c) the preference datasets – and hence the reward models – used for RL do show preferences for sycophantic responses (...well, sometimes, though see also the weird BoN results in Sharma et al 2023). So if you were to train indefinitely ("over-optimize") against these RMs they would presumably have a strong effect on sycophancy eventually.  But this kind of aggressive optimization against a sycophancy-preferring RM is certainly not necessary to produce noticeable sycophancy, and is probably not the cause behind most cases of LLM sycophancy that you and I notice in practice. 2. ^ See this comment by the lead author.

It's possible that "teaching to the test" tends to refer to something a bit more specific.  Here is John Holt in "How Children Fail", which some upstanding citizen has put onto the internet in easily googleable form: > This past year I had some terrible students. I failed more kids, mostly in French and Algebra, than did all the rest of the teachers in the school together. I did my best to get them through, good-ness knows. Before every test we had a big cram session of practice work, politely known as "review." When they failed the exam, we had post mortems, then more review, then a makeup test (always easier than the first), which they almost always failed again. Much later: > We teachers, from primary school through graduate school, all seem to be hard at work at the business of making it look as if our students know more than they really do. Our standing among other teachers, or of our school among other schools, depends on how much our students seem to know; not on how much they really know, or how effectively they can use what they know, or even whether they can use it at all. The more material we can appear to "cover" in our course, or syllabus, or curriculum, the better we look; and the more easily we can show that when they left our class our students knew what they were "supposed" to know, the more easily can we escape blame if and when it later appears (and it usually does) that much of that material they do not know at all. > > When I was in my last year at school, we seniors stayed around an extra week to cram for college boards. Our ancient-history teacher told us, on the basis of long experience, that we would do well to prepare ourselves to write for twenty minutes on each of a list of fifteen topics that he gave us. We studied his list. We knew the wisdom of taking that kind of advice; if we had not, we would not have been at that school. When the boards came, we found that his list comfortably covered every one of the eight questions we were asked. So we got credit for knowing a great deal about ancient history, which we did not, he got credit for being a good teacher, which he was not, and the school got credit for being, as it was, a good place to go if you wanted to be sure of getting into a prestige college. The fact was that I knew very little about ancient history; that much of what I thought I knew was misleading or false; that then, and for many years afterwards, I disliked history and thought it pointless and a waste of time; and that two months later I could not have come close to passing the history college boards, or even a much easier test, but who cared? > > I have played the game myself. When I began teaching I thought, naively, that the purpose of a test was to test, to find out what the students knew about the course. It didn't take me long to find out that if I gave my students surprise tests, covering the whole material of the course to date, almost everyone flunked. This made me look bad, and posed problems for the school. I learned that the only way to get a respectable percentage of decent or even passing grades was to announce tests well in advance, tell in some detail what material they would cover, and hold plenty of advance practice in the kind of questions that would be asked, which is called review. I later learned that teachers do this everywhere. We know that what we are doing is not really honest, but we dare not be the first to stop, and we try to justify or excuse ourselves by saying that, after all, it does no particular harm. But we are wrong; it does great harm. > > It does harm, first of all, because it is dishonest and the students know it. My friends and I, breezing through the ancient-history boards, knew very well that a trick was being played on someone, we were not quite sure on whom. Our success on the boards was due, not to our knowledge of ancient history, which was scanty, but to our teacher's skill as a predictor, which was great. Even children much younger than we were learn that what most teachers want and reward are not knowledge and understanding but the appearance of them. The smart and able ones, at least, come to look on school as something of a racket, which it is their job to learn how to beat. And learn they do; they become experts at smelling out the unspoken and often unconscious preferences and prejudices of their teachers, and at taking full advantage of them. My first English teacher at prep school gave us Macaulay's essay on Lord Clive to read, and from his pleasure in reading it aloud I saw that he was a sucker for the periodic sentence, a long complex sentence with the main verb at the end. Thereafter I took care to construct at least one such sentence in every paper I wrote for him, and thus assured myself a good mark in the course. > > Not only does the examination racket do harm by making students feel that a search for honest understanding is beside the point; it does further harm by discouraging those few students who go on making that search in spite of everything. The student who will not be satisfied merely to know "right answers" or recipes for getting them will not have an easy time in school, particularly since facts and recipes may be all that his teachers know. They tend to be impatient or even angry with the student who wants to know, not just what happened, but why it happened as it did and not some other way. They rarely have the knowledge to answer such questions, and even more rarely have the time; there is all that material to cover. > > In short, our "Tell-'em-and-test-'em" way of teaching leaves most students increasingly confused, aware that their academic success rests on shaky foundations, and convinced that school is mainly a place where you follow meaningless procedures to get meaningless answers to meaningless questions. And also: > It begins to look as if the test-examination-marks business is a gigantic racket, the purpose of which is to enable students, teachers, and schools to take part in a joint pretense that the students know everything they are supposed to know, when in fact they know only a small part of it--if any at all. Why do we always announce exams in advance, if not to give students a chance to cram for them? Why do teachers, even in graduate schools, always say quite specifically what the exam will be about, even telling the type of questions that will be given? Because otherwise too many students would flunk. What would happen at Harvard or Yale if a prof gave a surprise test in March on work covered in October? Everyone knows what would happen; that's why they don't do it.