We do this by performing standard linear regression from the residual stream activations (64 dimensional vectors) to the belief distributions (3 dimensional vectors) which associated with them in the MSP.
I don't understand how we go from this to the fractal. The linear probe gives us a single 2D point for every forward pass of the transformer, correct? How do we get the picture with many points in it? Is it by sampling from the transformer while reading the probe after every token and then putting all the points from that on one graph?
Is this result equivalent to saying "a transformer trained on an HMM's output learns a linear representation of the probability distribution over the HMM's states"?
Very comprehensive, thank you!
Epistemic status: I'm not familiar with the technical details of how LMs work, so this is more word association.
You can glide along almost thinking "a human wrote this," but soon enough, you'll hit a point where the model gives away the whole game. Not just something weird (humans can be weird) but something alien, inherently unfitted to the context, something no one ever would write, even to be weird on purpose.
What if the missing ingredient is a better sampling method, as in this paper? To my eye, the completions they show don't seem hugely better. But I do buy their point that sampling for high probability means you get low information completions.
How many of the decision makers in the companies mentioned care about or even understand the control problem? My impression was: not many.
Coordination is hard even when you share the same goals, but we don't have that luxury here.
An OpenAI team is getting ready to train a new model, but they're worried about it's self improvement capabilities getting out of hand. Luckily, they can consult MIRI's 2025 Reflexivity Standards when reviewing their codebase, and get 3rd-party auditing done by The Actually Pretty Good Auditing Group (founded 2023).
Current OpenAI wants to build AGI.[1] Current MIRI could confidently tell them that this is a very bad idea. Sure they could be advised that step 25... (read more)
TL;DR: Thought this post was grossly misleading. Then I saw that the GPT3 playground/API changed quite a lot recently in notable and perhaps worrying ways. This post is closer to the truth than I thought but I still consider it misleading.
Initially strongly downvoted since the LW post implies (to me) that humans provide some of the GPT3 completions in order to fool users into thinking it's smarter than it is. Was that interpretation of your post more in the eye of the beholder?
Nested three layers deep is one of two pieces of actual evidence:
... (read 544 more words →)InstructGPT is then further fine-tuned on a dataset labeled by human labelers. The labelers comprise a team of about
The Kefauver-Harris Drug Amendments of 1962 coincide with a drop in the rate of life-span increase.
I believe that, but I couldn't find a source. Do you remember where you got it from?
I wonder if, in that case, your brain picks the stopping time, stopping point or "flick" strength using the same RNG source that is used when people just do it by feeling.
What if you tried a 50-50 slider on Aaronson's oracle, if it's not too exhausting to do it many times in a row? Or write down a sequence here and we can do randomness tests on it. Though I did see some tiny studies indicating that people can improve at generating random sequences.
Hm, could we tell apart yours and Zack's theories by asking a fixed group of people for a sequence of random numbers over a long period of time, with enough delay between each query for them to forget?
I seriously doubt the majority of the participants in these casual polls are doing anything like that.
It was my impression that humans are bad at random (number) generation. Tell a person to arrange stars on a black canvas randomly, and they'll space them out more or less equidistantly and uniformly. If asked to generate a sequence of coin flips, we would avoid longer runs of the same coin face, which do happen with real coins.
The scientific consensus agrees. Here's a study where participants are asked to generate a sequence of 300 digits from 1 to 9. The researches successfully predict the next digit with a 27% chance (a truly random sequence would indicate an 11% chance). Scott Aaronson designed a simple program that predicts which of two keys... (read 168 more words →)
Yep, that's what I was trying to describe as well. Thanks!