Irving's team's terminology has been "behavioural alignment" for the green box - https://arxiv.org/pdf/2103.14659.pdf
The byte-pair encoding is probably hurting it somewhat here; forcing it to unpack it will likely help. Try using this as a one-shot prompt:
How many Xs are there in "KJXKKLJKLJKXXKLJXKJL"?
Numbering the letters in the string, we have: 1 K, 2 J, 3 X, 4 K, 5 K, 6 L, 7 J, 8 K, 9 L, 10 J, 11 K, 12 X, 13 X, 14 K, 15 L, 16 J, 17 X, 18 K, 19 J, 20 L. There are Xs at positions 3, 12, 13, and 17. So there are 4 Xs in total.
How many [character of interest]s are there in "[string of interest goes here]"?
If it's still getting confused, add more shots - I suspect it can figure out how to do it most of the time with a sufficient number of examples.
It seems like you're claiming something along the lines of "absolute power corrupts absolutely" ... that every set of values that could reasonably be described as "human values" to which an AI could be aligned -- your current values, your CEV, [insert especially empathetic, kind, etc. person here]'s current values, their CEV, etc. -- would endorse subjecting huge numbers of beings to astronomical levels of suffering, if the person with that value system had the power to do so.
I guess I really don't find that claim plausible. For example, here is my reaction to the following two questions in the post:
"How many ordinary, regular people throughout history have become the worst kind of sadist under the slightest excuse or social pressure to do so to their hated outgroup?"
... a very, very small percentage of them? (minor point: with CEV, you're specifically thinking about what one's values would be in the absence of social pressure, etc...)
"What society hasn’t had some underclass it wanted to put down in the dirt just to lord power over them?"
It sounds like you think "hatred of the outgroup" is the fundamental reason this happens, but in the real world it seems like "hatred of the outgroup" is driven by "fear of the outgroup". A godlike AI that is so powerful that it has no reason to fear the outgroup also has no reason to hate it. It has no reason to behave like the classic tyrant whose paranoia of being offed leads him to extreme cruelty in order to terrify anyone who might pose a threat, because no one poses a threat.
This reminded me of some findings associated with "latent semantic analysis", an old-school information retrieval technique. You build a big matrix where each unique term in a corpus (excluding a stoplist of extremely frequent terms) is assigned to a row, each document is assigned to a column, and each cell holds the number of times that term appeared in document , and with some kind of weighting scheme that downweights frequent terms), and you take the SVD. This also gives you interpretable dimensions, at least if you use varimax rotation. See for example pgs. 9-11 & pgs. 18-20 of this paper. Also, I seem to recall that the positive and negative singular values after doing latent semantic analysis are often both semantically interpretable, sometimes with antipodal pairs, although I can't find the paper where I saw this.
I'm not sure whether the right way to think about this is "you should be very circumspect about saying that 'semantic processing' is going on just because the SVD has interpretable dimensions, because you get that merely by taking the SVD of a slightly preprocessed word-by-document matrix", or rather "a lot of what we call 'semantic processing' in humans is probably just down to pretty simple statistical associations, which the later layers seem to be picking up on", but it seemed worth mentioning in any case!
edit: seems likely that the "association clusters" seen in the earlier layers might map onto what latent semantic analysis is picking up on, whereas the later layers might be picking up on semantic relationships that aren't as directly reflected in the surface-level statistical associations. could be tested!
Why do you expect Bitcoin to be excepted from being labelled a security along with the rest?
(Apologies if the answer is obvious to those who know more about the subject than me, am just genuinely curious)
Had a similar medical bill story from when I was a poor student: Medical center told me that insurance would cover an operation. They failed to mention that they were only talking about the surgeon's fee; the hospital at which they arranged the operation was out-of-network and I was stuck with 50% of the facility's costs. I explained my story to the facility. They said I still had to pay but that a payment plan would be possible, and that I could start by paying a small amount each month. I took that literally and just started paying a (very) small amount monthly. At some point they called back to tell me to formally arrange a payment plan through their online portal, which gave me options with such high interest rates that there was no way my future earnings would increase at a fast enough rate to make a payment plan make any sense whatsoever. I called back and explained this, and said that if those were the only options I guess I would just have to try to scrape the money together now, and that I was prepared to try to do this. The administrator, bless her heart, asked me to hold for awhile, and eventually came back to say "I've spoken with my colleagues, and your current balance owed to us is now zero dollars".
This (along with a few other experiences in my life) has underscored how sometimes an apparently immovable constraint can evaporate if you can manage to talk to the right person. That said, I felt very lucky to have been taken pity on in this way -- I feel like having one's balance explicitly zeroed out in this way is rare! But it's interesting to hear that Zvi knows of cases where someone just didn't pay, with no consequences. I would have assumed that they'd normally report nonpayers to credit agencies and crater their credit scores after long enough, as it costs them nothing or almost nothing to do so. Would be interested either to hear other people's anecdotes of what happened after nonpayment of a large hospital bill (positive or negative), or to see data on this if anyone knows of any.
I was using medical questions as just one example of the kind of task that's relevant to sandwiching. More generally, what's particularly useful for this research programme are
Prime examples are task types that require some kind of niche expertise to do and evaluate. Cotra's examples involve "[fine-tuning] a model to answer long-form questions in a domain (e.g. economics or physics) using demonstrations and feedback collected from experts in the domain", "[fine-tuning] a coding model to write short functions solving simple puzzles using demonstrations and feedback collected from expert software engineers", "[fine-tuning] a model to translate between English and French using demonstrations and feedback collected from people who are fluent in both languages". I was just making the point that Surge can help with this kind of thing in some domains (coding), but not in others.
It's worth knowing that there are some categories of data that Surge is not well positioned to provide. For example, while they have a substantial pool of participants with programming expertise, my understanding from speaking with a Surge rep is that they don't really have access to a pool of participants with (say) medical expertise -- although for small projects it sounds like they are willing to try to see who they might already have with relevant experience in their existing pool of 'Surgers'. This kind of more niche expertise does seem likely to become increasingly relevant for sandwiching experiments. I'd be interested in learning more about companies or resources that can help collect RLHF data from people with uncommon (but not super-rare) kinds of expertise for exactly this reason.
I did Print to PDF in Word after formatting my Word document to look like a standard LaTeX-exported document, it had no problem going through! But might depend on the particular moderator.
Nice, thanks for this!
Anecdotally, I feel like I've heard a number of instances of folks with what pretty clearly seemed to be long Covid coming on despite not having required hospitalization? And in this UK survey of "Estimated number of people (in thousands) living in private households with self-reported long COVID of any duration", it looks like only 4% of such people were hospitalized (March 2023 dataset table 1)