TurnTrout's shortform feed

TurnTrout

This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

The "shoggoth" meme is, in part, unfounded propaganda. Here's one popular incarnation of the shoggoth meme:

Shoggoth with Smiley Face (Artificial Intelligence) | Know Your Meme

This meme accurately portrays the (IMO correct) idea that finetuning and RLHF don't change the base model too much. Furthermore, it's probably true that these LLMs think in an "alien" way.

However, this image is obviously optimized to be scary and disgusting. It looks dangerous, with long rows of sharp teeth. It is an eldritch horror. It's at this point that I'd like to point out the simple, obvious fact that "we don't actually know how these models work, and we definitely don't know that they're creepy and dangerous on the inside."

In my opinion, the prevalence of the shoggoth meme is just another (small) reflection of how community epistemics have been compromised by groupthink and fear. If it's your job to try to accurately understand how models work—if you aspire to wield them and grow them for friendly purposes—then you shouldn't pollute your head with propaganda which isn't based on any substantial evidence.

I'm confident that if there were a "pro-AI" meme with a friendly-looking base model, LW / the shoggoth enjoyers would have nitpicked the friendly meme-creature to hell. They would (correctly) point out "hey, we don't actually know how these things work; we don't know them to be friendly, or what they even 'want' (if anything); we don't actually know what each stage of training does..."

Oh, hm, let's try that! I'll make a meme asserting that the final model is a friendly combination of its three stages of training, each stage adding different colors of knowledge (pre-training), helpfulness (supervised instruction finetuning), and deep caring (RLHF):

Enhance the shoggoth creature to be even more cheerful and delightful. Add smiling faces to each part of the creature, symbolizing 'base model', 'supervised fine-tuning', and 'RLHF', to convey a sense of happiness and friendliness. Incorporate elements of rainbows in the design, with vibrant and colorful arcs that blend harmoniously with the creature's fluffy and soft appearance. These rainbows can be integrated into the creature's body or as a background element, adding a playful and magical atmosphere. The overall look should exude positivity, making the creature appear even more approachable, whimsical, and enchanting.

I'm sure that nothing bad will happen to me if I slap this on my laptop, right? I'll be able to think perfectly neutrally about whether AI will be friendly.

You're making a valid point here, and appreciate the pushback on local groupthink.

However....

However, this image is obviously optimized to be scary and disgusting. It looks dangerous, with long rows of sharp teeth. It is an eldritch horror. It’s at this point that I’d like to point out the simple, obvious fact that “we don’t actually know how these models work, and we definitely don’t know that they’re creepy and dangerous on the inside.”
In my opinion, the prevalence of the shoggoth meme is just another (small) reflection of how community epistemics have b

... (read more)

2Noosphere892y

What I mean by this is that if you want to predict what an AI will do, for example will it do better in having a new capability than other models, or what it's values are like, especially if you want to predict OOD behavior accurately, you would be far better off if you knew what it's data sources are, as well as the quality of it's data, than if you only knew it's prior/architecture. Re my justification for it, my basic justification comes from this tweet thread, which points out that a lot of o1's success could come from high-quality data, and while I don't like the argument that search/fancy bits aren't happening at all (I do think o1 is doing a very small run-time search), I agree with the conclusion that the data quality was probably most of the reason o1 is so good in coding. https://x.com/aidanogara_/status/1838779311999918448 Somewhat more generally, I'm pretty influenced by this post, and while I don't go as far as claiming that all of what an AI is the dataset, I do think a weaker version of the claim is pretty likely to be true. https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ But one prediction we could have made in advance, if we knew that data was a major factor in how AIs learn values, is that value misspecification was likely to be far less severe of a problem than 2000-2010s thinking on LW had, and value learning had a tractable direction of progress, as training on human books and language would load into it mostly-correct values, and another prediction we could have made is that human values wouldn't all be that complicated, and could instead be represented by say several hundred megabyte/1 gigabyte codes quite well, and we could plausibly simplify that further. To be clear, I don't think you could have had too high of a probability writ it's prediction before LLMs, but you'd at least have the hypothesis in serious consideration. Cf here: [...] From here on Matthew Barnett's post about the historical value misspecificatio

2Noosphere892y

Note that in order for my argument to go through, we also need the brain to be similar enough to DL systems that we can validly transfer insights from DL to the brain, and while I don't think you could place too high of a probability 10-20 years ago on that, I do think that at the very least this should have been considered as a serious possibility, which LW mostly didn't do. However, we now have that evidence, and I'll post links below: https://www.lesswrong.com/posts/rjghymycfrMY2aRk5/llm-cognition-is-probably-not-human-like#KBpfGY3uX8rDJgoSj https://x.com/BogdanIonutCir2/status/1837653632138772760 https://x.com/SharmakeFarah14/status/1837528997556568523

TurnTrout's shortform feed

28

Ω 10

TurnTrout's shortform feed

28

Ω 10