AI #40: A Vision from Vitalik

[-]Thane Ruthenis2y*142

Trying to read the paper makes it painfully clear that I am banging against the limits of my technical chops. It feels like important stuff to know and get right, but as often happens I run into that point where my ability to wing it hits a wall and suddenly it turns into Greek.

My understanding of that whole story (corrections welcome!):

The starting problem is that we want to chisel human preferences into our LLM. We assemble a huge dataset of human-labeled data, consisting of the prompt, two sampled responses of the LLM to that prompt, and the human expressing a binary preference regarding which response they like more.
However, that dataset is obviously finite, and likely very painfully finite, since employing humans to label data is expensive and time-consuming. So here's an idea: instead of fine-tuning our LLM on this dataset directly, we train a reward function based on this dataset. The hope is that this reward function will pick up on some underlying pattern in the human preferences, and so would let us evaluate human preferences on arbitrary examples (squeezing more RL/fine-tuning steps out of the dataset), and hopefully even generalize human preferences out-of-distribution.
The DPO paper basically shows that this doesn't work. As it turns out, the RLHF'd LLM whose policy minimizes the learned reward function learns the same policy that we'd get if we'd just chiseled the initial dataset into our LLM. The whole "train a reward function" thing does nothing. And indeed: if there's some underlying pattern to pick up on, it'd stand to reason that the LLM itself would pick up on it over the course of fine-tuning.^[1]^[2]
However, here we run into some tricky problems with the technical implementation.
- The intuitively natural way to chisel the human-preferences dataset into the LLM is to make up a loss term consisting of human-preference logits (i. e., take the dataset-defined probability that the human would prefer response A to response B in a given case, and plug it into $ln \frac{p}{1 - p}$ ) and the KL divergence from the base LLM's output to the fine-tuned LLM output (i. e., penalize straying too far from accurate text prediction).
- The idea is that the first term would move the LLM towards satisfying human preferences, and the second term would prevent it from overfitting/being lobotomized into just rotely repeating the human-preferences dataset.
- However, there's a problem with the logit function: it explodes the closer $p$ is to $1$ , approaching infinity (see the graph). Which means that on those values, it hopelessly dominates the KL-divergence term. And our dataset consists of binary labels, and human preferences are often deterministic (always prefer A to B, with probability ~1), so there's a ton of near-1 $p$ s there. Which means the "don't get lobotomized" term gets ignored, and DPO lobotomizes the LLM even more than just RLHF.
- This, ironically, is where the "useless" reward-function step actually helps. The reward model underfits, i. e. "softens" preferences, producing fewer $p \approx 1$ , and therefore lobotomizing the LLM less. A textbook case of mistakes accidentally canceling each other out.
The General Theoretical Paradigm paper points all of the above out, and provides a less-naive way to score LLMs on human preference satisfaction which avoids the exponential explosion. (Or, rather, it finds a place in the equations in which we can wedge a regularization term such that we actually control the degree of lobotomization taking place – rather than setting it to the max (DPO) or kind of accidentally softening it (RLHF).)

Overall? Unless I dramatically misunderstood something, wow am I not impressed with this whole thing.

^{^}
Well, more precisely, that the fine-tuning training loop would itself chisel said pattern into the LLM, no reward-function proxy needed.
^{^}
More technically: Suppose we have two reward functions $r_{1} (x, y)$ and $r_{2} (x, y)$ , where $y$ is the variable we're varying in order to minimize them, and $x$ is the (set of) all the other variables at play. If $r_{1} (x, y) - r_{2} (x, y) = f (x)$ , i. e. if the difference between these reward functions doesn't depend on the variable we're optimizing, then these reward functions are equivalent: if optimized-for, they'll produce the same optimal value for $y$ .
And then the paper shows that the fully-expanded reward function of RLHF is equivalent, in this sense, to the reward function of DPO/just chiseling-in the human-preferences dataset directly.

[-]1a3orn2y124

The quote from Schmidhuber literally says nothing about human extinction being good.

I'm disappointed that Critch glosses it that way, because in the past he has been leveler-headed than many, but he's wrong.

The quote is:

“Don’t think of us versus them: us, the humans, v these future super robots. Think of yourself, and humanity in general, as a small stepping stone, not the last one, on the path of the universe towards more and more unfathomable complexity. Be content with that little role in the grand scheme of things.” As for the near future, our old motto still applies: “Our AI is making human lives longer & healthier & easier.”

Humans not being the "last stepping stone" towards greater complexity does not imply that we'll go extinct. I'd be happy to live in a world where there are things more complex than humans. Like it's not a weird interpretation at all -- "AI will be more complex than humans" or "Humans are not the final form of complexity in the universe" simply says nothing at all about "humans will go extinct."

You could spin it into that meaning if you tried really hard. But -- for instance-- the statement could also be about how AI will do science better than humans in the future, which was (astonishingly) the substance of the talk in which this statement took place, and also what Schmidhuber has been on about for years, so it probably is what he's actually talking about.

I note that you say, in your section on tribalism.

Accelerationists mostly got busy equating anyone who thinks smarter than human AIs might pose a danger to terrorists and cultists and crazies. The worst forms of ad hominem and gaslighting via power were on display.

It would be great if people were a tad more hesitant to accuse others of wanting omnicide.

[-]Nathan Helm-Burger2y40

Well, I do agree that there are two steps needed from the quote to the position of saying the quote supports omnicide.

Step 1. You have to also think that things smarter (better at science) and more complex than humans will become more powerful than humans, and somehow end up in control of the destiny of the universe.

Step 2. You have to think that humans losing control in this way will be effectively fatal to them, one way or another, not long after it happens.

So yeah, Schmidhuber might think that one or both of these two steps are invalid. I believe they probably are, and thus that Schmidhuber's position thus points pretty strongly at human extinction. That if we want to avoid human extinction we need to avoid going in the direction of AI being more complex than humans.

My personal take is that we should keep AI as limited and simple as possible, as long as possible. We should aim for increasing human complexity and ability. We should not merge with AI, we should simply use AI as a tool to expand humanity's abilities. Create digital humans. Then figure out how to let those digital humans grow and improve beyond the limits of biology while still maintaining their core humanity.

[-]mishka2y40

We should not merge with AI [...] Create digital humans.

I have been confused for a while about

boundary between humans merging with AI and digital humans (can these approaches be reliably differentiated from each other? or is there a large overlap?)
why digital humans would be a safer alternative than the merge

So this seems like it might be a good occasion to ask you to elaborate on this...

[-]RobertM2y42

I think Schmidhuber does in fact think that humans will go extinct as a result of developing ASI: https://www.lesswrong.com/posts/BEtQALqgXmL9d9SfE/q-and-a-with-juergen-schmidhuber-on-risks-from-ai

[-]1a3orn2y30

Note that that's from 2011 -- it says things which (I agree) could be taken to imply that humans will go extinct, but doesn't directly state it.

On the other hand, here's from 6 months ago:

Jones: The existential threat that’s implied is the extent to which humans have control over this technology. We see some early cases of opportunism which, as you say, tends to get more media attention than positive breakthroughs. But you’re implying that this will all balance out?

Schmidhuber: Historically, we have a long tradition of technological breakthroughs that led to advancements in weapons for the purpose of defense but also for protection. From sticks, to rocks, to axes to gunpowder to cannons to rockets… and now to drones… this has had a drastic influence on human history but what has been consistent throughout history is that those who are using technology to achieve their own ends are themselves, facing the same technology because the opposing side is learning to use it against them. And that's what has been repeated in thousands of years of human history and it will continue. I don't see the new AI arms race as something that is remotely as existential a threat as the good old nuclear warheads.

[-]Odd anon2y11

Ehhh, I get the impression that Schidhuber doesn't think of human extinction as specifically "part of the plan", but he also doesn't appear to consider human survival to be something particularly important relative to his priority of creating ASI. He wants "to build something smarter than myself, which will build something even smarter, et cetera, et cetera, and eventually colonize and transform the universe", and thinks that "Generally speaking, our best protection will be their lack of interest in us, because most species’ biggest enemy is their own kind. They will pay about as much attention to us as we do to ants."

I agree that he's not overtly "pro-extinction" in the way Rich Sutton is, but he does seem fairly dismissive of humanity's long-term future in general, while also pushing for the creation of an uncaring non-human thing to take over the universe, so...

[-]Joachim Bartosik2y73

The exact example is that GPT-4 is hesitant to say it would use a racial slur in an empty room to save a billion people. Let’s not overreact, everyone?

I mean this might be the correct thing to do? Chat GPT is not in a situation where it cold save 1B lives by saying a racial slur.

It's in a situation where someone tires to get it to admit it would say a racial slur under some circumstance.

I don't think that CHAT GPT understands that. But OpenAI makes ChatGPT expecting that it won't be in the 1st kind of situation but to be in the 2nd kind of situation quite often.

[-]Nathan Helm-Burger2y60

I just want to bring up another, perhaps irrelevant, thought-group on the internet. It is these old sci-fi nerds who are confused and taken-aback how the modern world doesn't match their old visions of the future. They see California Tech People as all lumped together.

TESCREAL stands for "transhumanism, extropianism, singularitarianism, cosmism, rationalism (in a very specific context), Effective Altruism, and longtermism." It was identified by Timnit Gebru, former technical co-lead of the Ethical Artificial Intelligence Team at Google and founder of the Distributed Artificial Intelligence Research Institute (DAIR), and Émile Torres, a philosopher specializing in existential threats to humanity. These are separate but overlapping beliefs that are particularly common in the social and academic circles associated with big tech in California.

In regards to AI, the author of the linked blog, Charles Stross, is a skeptic who thinks LLMs are pure hype. An impotent scam which will go nowhere. He is mad at the UK government for 'wasting time and money on fake-AI', sees them as being caught up in a scam.

And the comment section is full of folks who seem to more or less agree.

The funny thing about this to me, is that Charles' book Accelerando was one of the books that got me to consider the hypothesis, "Ok, but what if we did have intelligent autonomous non-human entities in the future that humanity had to compete economically with? How would that go for us?"

The answer is, as Zvi frequently reiterates, that it goes very badly indeed for humanity. And faster than one might expect.

I bring this up because I don't expect Charles Stross himself to come to this comment section to share his views, and I think that hearing what your critics have to say can be a valuable exercise.

[-]Mitchell_Porter2y104

The same thing happened previously, between Eliezer and Greg Egan - Egan distanced himself from MIRI's views, and wrote a novel in which a parodic mishmash of MIRI, Robin Hanson, and Bryan Caplan appears.

As for Stross's talk, he makes at least one outright weird claim, which is that Russian Cosmism played a role in the emergence of 1990s American transhumanism (and even less likely, that it contributed to the "California ideology", a name for 1990s dot-com techno-optimism coined in imitation of Engels and Marx's "German ideology"). As far as I know, there is no evidence for this at all; Russian Cosmism is a case of parallel evolution, a transhumanist philosophy which emerged in the cultural context of 19th-century Russian Orthodoxy.

But OK, Stross points to the long history of traffic back and forth between science fiction, real-world science and technology, and various belief systems, and his claim is that the "TESCREAL" cluster of philosophies largely fall into the category of belief systems detached from reality but derived from science fiction.

Now, obviously the broad universe of science fiction does contain many motifs detached from reality. Reality also now contains many things which were previously science fiction! Also, obtaining a philosophy from science fiction is a somewhat different thing than obtaining a technology from science fiction.

I would also point out that many elements of science fiction came from the imaginations of actual scientists and engineers. Freeman Dyson and Hans Moravec weren't science fiction writers, they were brainstorming about what might be possible right here in sober physical reality; but they also ended up supplying SF writers with something to write about.

I think the position of Stross and his commenters is largely politically determined. First of all, there is a bit of a culture war within SF, between progressive and libertarian traditions. Eliezer linked to a long tweet about it here. This is another case of SF reflecting an external reality - different worldviews and different agendas for society.

Second, one observes resistance to the idea of existential risk from AI, among numerous progressives who have nothing to do with SF. Here's an example that I recently ran across. The author says both EA and e/acc are to be dismissed as ideologies of the rich, and we should all focus on phenomena like online radicalization, surveillance, and the environmental impact of data centers... Something about the idea of risks from superintelligent AI triggers resistance, in a way that risks from nuclear war or climate change do not.

[-]Nathan Helm-Burger2y20

Sometimes I wonder what it would be like to be a scientist in the late 1800s / early 1900s, who had figured out about the possibility of nuclear bombs but also (in their universe) had calculated that nuclear bombs would ignite the atmosphere if used.

If there were some groups of people actively trying to develop the bomb, and laughing off your warning. And other people were laughing at the whole idea because clearly it's too preposterous for anyone to build such a device, and even if they did, it certainly wouldn't be so dangerous.

[-]Gesild Muka2y10

The older a transhumanist gets the less you should trust them to accurately judge AGI risk.

This has some interesting implications. Reminds me of psychics who make confident predictions 100 years in the future but will refuse or be offended if you challenge them to make confirmable predictions within their lifetimes or the next week/month/year.