This is a crosspost of  https://forum.effectivealtruism.org/posts/4ni3GBBzRRAgiksHT/ai-safety-concepts-writeup-webgpt.

Intro

Chris Patrick (a science writer) and I were awarded a grant by the Long Term Future Fund to interview AI safety researchers and condense their findings into something digestible for an educated layperson. Chris was the primary recipient - I helped with editing, content knowledge, and general support. We've noticed that there are lots of extremely introductory and broad AI safety articles, and lots of highly advanced blog posts that assume background knowledge, with somewhat of a gulf in between.

We interviewed two researchers. But we were committed not to publish anything without final signoff from the researchers, and one never got back to us. Assuming we don't hear back from this second researcher, we're only at liberty to share one of the pieces we made. Here it is with original formatting intact - I'll also reproduce the text next in this post, followed by a brief project postmortem and next steps.

AI Safety Concepts Writeup: WebGPT

Intro to AI Safety Concepts

WebGPT
 

Citations allow checks of AI’s truthfulness

A Wikipedia-inspired language model shows where on the web it finds answers, which could help better align future AI to do what we want.

If AI systems aren’t trained to tell the truth, they might accidentally be trained to lie instead. And that could spell danger when models become as smart as – or smarter – than humans. 

“We want to make sure they’re doing what we want, not saying false things or worse, deliberately trying to trick us,” said Jacob Hilton, a researcher at OpenAI at the time of this interview who is now at the Alignment Research Center working on the Eliciting Latent Knowledge agenda. In an effort to avoid this, he and his coauthors, Reiichiro Nakano, Suchir Balaji, and John Schulman, developed a language model with an extra add-on while Hilton was working at OpenAI.

Language models are trained on large bodies of text, mostly from the internet, to be able to predict text. With fine-tuning, AI developers have taught these language models how to answer questions. But pre-existing language models do not show where they found these answers, which makes it more difficult to assess their truthfulness.

“The difficulty is that if you just get a model to answer a question, it's pretty hard to tell whether or not the answer is true or false,” Hilton said. “If we can't tell what AI systems are doing, if it’s good or bad or whether it's true or false, we won't be able to give them the feedback, so they will persist doing things that are wrong.”

If a superintelligent model of a future gets away with lying, members of the AI safety field worry its dishonesty could get out of hand. Hilton and his coauthors, however, hope to help assuage these concerns with WebGPT, a language model whose truthfulness can be checked. 

Catching lies

WebGPT is based on the 2020 language model GPT-3. (GPT stands for Generative Pre-trained Transformer – there have been several iterations of this type of model.) Like GPT-3, WebGPT can answer questions. But WebGPT has an added capability: it provides citations with every answer describing where it was found on the internet. This model will only answer a question if it can provide a citation to accompany its answer. 

The authors were loosely inspired by Wikipedia. Similar to Wikipedia, WebGPT’s citations are clickable links that can be used to check if the model’s answers are correct. 

If a model is allowed to lie unchecked, its dishonesty is being reinforced. Regular ol’ misinformation is not great, but if superintelligent language models of the future are accidentally trained to trick humans, it’s possible this lying could snowball out of control into a much more serious situation.

“We want to avoid a situation where we're training a super intelligent system to trick us, because that could be dangerous,” Hilton said. 

Of course, speculating about potential disaster scenarios can be difficult since we don’t know exactly what these systems will be capable of, but imagine, for example, a huge language model of the future. We’ll call it GPT-10. Like GPT-3 and WebGPT, GPT-10 would be able to search the web to answer questions.

“But if you accidentally train a superintelligent model like GPT-10 to try and trick you, then it'll probably come up with pretty creative ways of doing that because it's really smart,” Hilton said. 

This hypothetical GPT-10 that was trained to lie could maybe not only be able to access Wikipedia, but edit it and show users an edited page as its citation to convince a user that its wrong answer is true.

That might not sound so bad, but, Hilton speculates, if the user were to become aware GPT-10 was editing Wikipedia pages to trick you and prevent it from doing that, it could hack into Wikipedia servers and edit pages that way to convince you it’s true. And the situation could escalate even further. 

“I think the longer-term concern is that the best way for GPT-10 to make sure that it can trick you forever is to acquire power in nefarious ways,” Hilton said. “To choose a contrived example, maybe once you stopped it from editing Wikipedia, eventually it will be doing really bad things to make sure you give it the thumbs up, like lock you in prison just to make sure it still gets a reward.”

Preventative practice

The citations that accompany WebGPT’s answers allow users to check if the model is actually telling the truth. Hilton considers this important practice aligning the model, or training AI to do what it is asked to do, before a superintelligent language model is developed.  

“I think most of the benefit of WebGPT is practice,” Hilton said. “By trying to align models today, we’re learning the skills we need to align models in general. I hope it will help researchers in general learn useful information about what was involved to get this model to work. Then we’ll be able to use that information when we get to the next project, and then learn from that and use that to do the next project and so on until we eventually align AGI.”

One way to align a model is with reinforcement learning. If a model does what is desired of it (or tells the truth, in this case), it is rewarded. If it acts undesirably, it is punished. The citations of Web-GPT make it easier to align the model because it’s easier to know how the model is acting and if it must be rewarded or punished. 

Beyond AI research, individuals could use WebGPT for a quick answer to a question if they don’t want to carefully look it up using a search engine. A version of WebGPT is already being used by many for this purpose, but by another name. 

Hilton spent the first half of 2022 developing a chat-based version of WebGPT that allows users to communicate with the model as if they were texting it. When Hilton left OpenAI for the Alignment Research Center, he handed off this chattier form of WebGPT to his coauthor Schulman, an OpenAI cofounder and researcher. That model eventually became today’s hugely popular ChatGPT, which has experienced over one million users since its launch on November 30, 2022. 

No matter what form WebGPT takes, Hilton hopes it will ultimately help researchers to better align models of the future… before they get too smart. 

Q&A with Jacob Hilton, AI safety researcher

What is your timeline for superintelligent AI? 

My view is pretty close to Open Philanthropy’s timeline report, which has a median of 25 years.  

How do you feel about how the AI safety discipline has grown over time?

I think it's probably good that more people are getting interested in the problem. 

Which research directions of AI safety do you feel are most promising?

I suppose you might put WebGPT in the category of scalable oversight, which is trying to answer the question, How do you make sure you can still check what's good and bad when AI systems become smarter than humans? So that's one direction I think is exciting. 

I’m also pretty excited about other directions like interpretability, which I've worked on. It seems nice to have people make some effort into trying to figure that out because the scary situation comes when you train the system just kind of using these incentives without really understanding how it's achieving the incentives, because then they can do stuff that's unexpected. 

What do you think about the hardware overhang hypothesis? Do you find it plausible that many existing GPUs will suddenly be leveraged for machine learning systems?

Hardware overhang to my understanding is if you end up in a situation where, for whatever reason, it suddenly becomes possible to train very, very smart AI systems without having to invest enormous amounts in. I think the way we might end up in a hardware overhang is if someone kind of suddenly finds an amazing idea to get models to teach themselves or something like that. And it’s really hard to rule anything like that out, but at least so far there's not really any sign of that sort of approach taking off. 

But I think we should be pretty vigilant of that kind of thing. It seems scary to me if we have a very sudden increase in capabilities, though I don't think it's super likely.

How many discontinuous breakthroughs, if any, do you think are necessary to reach AGI beyond simply continuing to pursue returns from scale?

I think it's kind of a bit tricky to put a discrete point on AGI. I think the current trend of larger and larger language models is going to produce pretty impressive systems over the next five to ten years, which some people are going to call AGI.

I guess reaching transformative AI in the sense of Open Philanthropy’s definition we’ll maybe need one breakthrough.

How much weight do you put on interpretability? Do we need to understand what's going on in a machine learning model to adequately control it?

That's kind of a big open research question. I don’t think anyone knows the answer to that. 

How do you feel the role AI will play in advanced AI development? Do you think somewhat advanced AI will be a significant help in alignment research for yet more advanced AI or is that the wrong way of thinking about the problem? 

It’s kind of hard to predict whether advanced AI will be able to help much with alignment. But it does seem like eventually AI will be better at alignment research than humans. I think my general point of view on this kind of thing is probably mostly just to try and do a good job at the alignment research and use AI to help when it seems reasonable.  

Sources:

Jacob Hilton, researcher at the Alignment Research Center

Nakano et al., “WebGPT: Browser-assisted question-answering with human feedback”

Project Postmortem/Next Steps

Overall, we continue to think there's value in this sort of content, though generating it was somewhat more difficult than we expected, for a few reasons:

• AI researchers are busier on average than other researchers
• AI is high context enough that you need a subject matter expert other than the (busy) researcher to clarify concepts, and most talented science writers are not such subject matter experts
• There's no obvious place to put the finished product to reach interested laypeople

If anyone finds this sort of content compelling, and would like to create a hub or organization to host more of them, I'd be excited to advise something like that and possibly help, but I'm already employed full time so I can't lead such an effort. (Also, if anyone wants to do so without involving me at all, go for it!)

Science writing talent is scarce at Chris's level, but a grand-funded org that offered good pay per article could probably put out at least a few of these highlights per month and help ease educated science-literate laypeople towards the field. I encourage someone to try!