Announcing: OpenAI's Alignment Research Blog
36habryka
19boazbarak
11Caleb Biddulph
7Rohin Shah
6Naomi Bashkansky
2Richard_Kennaway
New Comment
I generally think blogging is a good way to communicate intellectual progress, so see this as a good development!
Some thoughts on your first blogpost:
At OpenAI, we research how we can develop and deploy increasingly capable AI, and in particular AI capable of recursive self-improvement (RSI)
My reaction: Wait, what, why? I guess it's nice to be as direct, but it feels sad that this is written as the bottom line.
To be clear, I agree that given this being OpenAI's stance it's good to say it plainly! But I was hoping that at least the safety team would have the position that "we will try to determine whether there is any way to build RSI safely, and will strongly advocate for not doing so if we think it cannot be done safely".
Like, a thing that feels particularly sad here is that I was assuming that figuring out whether this can be done safely, or studying that question, is one of the key responsibilities of the safety team. This is an update that it isn't, which is sad (and IMO creates some responsibility for members of the safety team to express concern about that publicly, but IDK, OpenAI seems like in a messy state with regards to that kind of stuff).
Thank you for pointing this out! While OpenAI have been public about our plans to build an AI scientists, it is of course crucial that we do this safely, and if it is not possible to do it safely, we should not do it at all.
We have written about this before:
OpenAI is deeply committed to safety, which we think of as the practice of enabling AI’s positive impacts by mitigating the negative ones. Although the potential upsides are enormous, we treat the risks of superintelligent systems as potentially catastrophic and believe that empirically studying safety and alignment can help global decisions, like whether the whole field should slow development to more carefully study these systems as we get closer to systems capable of recursive self-improvement. Obviously, no one should deploy superintelligent systems without being able to robustly align and control them, and this requires more technical work.
but we should have mentioned this in the hello world post too. We now updated it with a link to this paragraph.
1It looks like OpenAI is following Anthropic's lead, which is great!
Google DeepMind's alignment team also has a blog, but it's much more targeted towards laypeople, mostly shares papers as they come out rather than sharing informal research, and IMO isn't as nice compared to a dedicated website or even a Substack. It might be worth considering doing something like this at GDM, subject to tradeoffs on researchers' time and Google's internal publication restrictions.
We mostly just post more informal stuff directly to LessWrong / Alignment Forum (see e.g. our interp updates).
Having a separate website doesn't seem that useful to readers. I generally see the value proposition of a separate website as attaching the company's branding to the post, which helps the company build a better reputation. It can also help boost the reach of an individual piece of research, but this is a symmetric weapon, and so applying it to informal research seems like a cost to me, not a benefit. Is there some other value you see?
(Incidentally, I would not say that our blog is targeted towards laypeople. I would say that it's targeted towards researchers in the safety community who have a small amount of time to spend and so aren't going to read a full paper. E.g. this post spends a single sentence explaining what scheming is and then goes on to discuss research about it; that would absolutely not work in a piece targeting laypeople.)
Curated and popular this week