I think we should not be hesitant to talk about this in public. I used to be of the opposite opinion, believing-as-if there was a benevolent conspiracy that figured out which conversations could/couldn’t nudge AI politics in useful ways, whose upsides were more important than the upsides of LWers/etc. knowing what’s up. I now both believe less in such a conspiracy, and believe more that we need public fora in which to reason because we do not have functional private fora with memory (in the way that a LW comment thread has memory) that span across organizations.
It’s possible I’m still missing something, but if so it would be nice to have it spelled out publicly what exactly I am missing.
I agree with Lincoln Quirk’s comment that things could turn into a kind of culture war, and that this would be harmful. It seems to me it’s worth responding to this by trying unusually hard (on this or other easily politicizable topics) to avoid treating arguments like soldiers. But it doesn’t seem worthwhile to me to refrain from honest attempts to think in public.
Ineffective, because the people arguing on the forum are lacking knowledge about the situation. They don't understand OpenAI's incentive structure, plan, etc. Thus any plans they put forward will be in all likelihood useless to OpenAI.
Risky, because (some combination of):
a benevolent conspiracy that figured out which conversations could/couldn’t nudge AI politics in useful ways
functional private fora with memory (in the way that a LW comment thread has memory) that span across organizations
What's standing in the way of these being created?
How open do we think OpenAI would be to additional research showing the dangers of AGI? If OpenAI is pursuing a perilous course, perhaps this community should prioritize doing the kind of research that would persuade them to slow down. Sam Altman came across to me at the two SSC talks he gave as being highly rational as this community would define rational.
If this is the correct path, we would benefit from people who have worked at OpenAI explaining what kind of evidence would be needed to influence them towards Eliezer's view of AGI.
I don't know what 'we' think, but as a person somewhat familiar with OpenAI employees and research output, they are definitely willing to pursue safety and transparency research that's relevant to existential risk, and I don't really know how one could do that without opening oneself up to producing research that provides evidence of AI danger.
I just saw this, but this feels like a better-late-than-never situation. I think hard conversations about the possibilities of increasing existential risk should happen.
I work at OpenAI. I have worked at OpenAI for over five years.
I think we definitely should be willing and able to have these sorts of conversations in public, mostly for the reasons other people have listed. I think AnnaSalamon is the answer I agree with most.
I want to also add this has made me deeply uncomfortable and anxious countless times over the past few years. It can be a difficult thing to navigate well or navigate efficiently. I feel like I've gotten better at it, and better at knowing/managing myself. I see newer colleagues also suffering from this. I try to help them when I can.
I'm not sure this is the right answer for all context, but I am optimistic for this one. I've found the rationality community and the lesswrong community to be much better than average at dealing with bad faith arguments, and for cutting towards the truth. I think there are communities where it would go poorly enough that it could be net-negative to have the conversation.
Side note: I really don't have a lot of context about the Elon Musk connection, and the guy has not really been involved for years. I think the "what things (including things OpenAI is doing) might increase existential risk" is an important conversation to have when analyzing forecasts, predictions, and research plans. I am less optimistic about "what tech executives think about other tech executives" going well.
I'd like to put in my vote for "this should not be discussed in public forums". Whatever is happening, the public forum debate will have no impact on it; but it does create the circumstances for a culture war that seems quite bad.
Whatever is happening, the public forum debate will have no impact on it;
I think this is wrong. I think a lot of people who care about AI Alignment read LessWrong and might change their relationship to Open AI depending on what is said here.
I disagree with Lincoln's comment, but I'm confused that when I read it just now it was at -2; it seems like a substantive comment/opinion that deserves to be heard and part of the conversation.
If comments expressing some folks' actual point of view are downvoted below the visibility threshold, it'll be hard to have good substantive conversation.
I'm skeptical of OpenAI's net impact on the spirit of cooperation because I'm skeptical about the counterfactual prospects of cooperation in the last 6 years had OpenAI not been founded.
The 2000s and early 2010s centralized and intermediated a lot of stuff online, where we trusted centralized parties to be neutral arbiters. We are now experiencing the after effects of that naivete, where Reddit, Twitter and Facebook are censoring certain parties on social media, and otherwise neutral infrastructure like AWS or Cloudflare kick off disfavored parties. I am at the point where I am scared of centralized infrastructure and try to minimize my reliance on it because it allows third parties to apply pressure on me.
These general trends are much larger than Elon Musk and would have happened without him. I'm uncertain to what extent Elon was just reacting to this trend and was ahead of the curve.
Darkly humorous, but OpenAI's recent destruction of otherwise entirely benign users of GPT-3 is teaching people to not rely on centralized AI. Now that OpenAI is large, the Blue Egregore can apply media pressure on the tech company to extract submission, in this case censorship demands. The downstream effects of this demand to think of the non-existent children was the destruction of trust in AI Dungeon leading people to mostly switch to various GPT-J alternatives (including local setups) and the shutdown of the Samantha, which Mr. Rohrer is taking hard.
Releasing GPT-3 non-trivially increased the odds of doomsday. So yeah they are not good actors.
Can you elaborate on that? It seems non-obvious; I feel like I could tell a different story
... like, OpenAI has much fewer resources than Deepmind, so if Deepmind released GPT-3 first, and if it is the case that scaling up a language model far enough gives you AGI, there would have been less time in between knowing what AGI will look like and having AGI.
Do you think there are reasons to avoid publicly debating OpenAI's impact that wouldn't apply to, say, Microsoft, or to the EU?
If you're asking me, I could imagine that there is a higher chance to insult individual people who matter, but really I don't know. I have no qualms if the answer to my first question is 'basically no reason to be hesitant'.
What kind of cooperation is OpenAI supposed to have "trashed"? Did they make AI R&D more competitive? Did they prevent AI safety researchers from talking to each other?
There is a "AI-arms-race-o'meter" and as a result of OpenAI the meter has gone up. This is real bad, because when we start getting close to AGI/APS-AI/etc. it's gonna be real important that we slow down and go carefully, but we won't do that if we are racing.
May I suggest that for an organization which really could create superhuman general AI, more important than a vague vow to go slow and be careful, would be a concrete vow to try to solve the problem of friendliness/safety/alignment. (And I'll promote June Ku's https://metaethical.ai as the best model we have of what that would look like. Not yet practical, but a step in the right direction.)
Perhaps it is a result of my being located on the margins, but I have never seen slowing the overall AI race as a tactic worth thinking about. That it is already an out-of-control race in which no one even knows all the competitors, has been my model for a long time. I also think one should not proceed on the assumption that there are years (let alone decades) to spare. For all we know, a critical corner could be turned at any moment.
Perhaps it feels different if you're DeepMind or MIRI; you may feel that you know most of the competitors, and have some chance of influencing them in a collegial way. But was OpenAI's announcement of GPT-3 in 2020, a different kind of escalation than DeepMind's announcement of AlphaGo in 2016? OpenAI has been concerned with AI safety from the beginning; they had Paul Christiano on their alignment team for years.
I am not exactly here to say that DeepMind is that much better! :) One thing I dislike about the OP is that it makes it seem like the problem is specifically with OpenAI compared to other companies. If OpenAI came first and then Elon went and founded DeepMind that would approximately just as bad, or even slightly worse.
I agree that maybe an arms race was inevitable, in which case founding OpenAI maybe wasn't a bad thing after all. Maybe. But maybe not.
It's true that OpenAI had some great safety researchers. Now most of them have quit. (There are still some that remain). But they probably could have got jobs at DeepMind, so this isn't relevant to evaluating Elon's decision.
Also, there's the whole openness ideal/norm. Terrible idea, for reasons various people (e.g. Scott Alexander) said at the time. (I can try to remember what the post was called if you like... it made the same point as Yudkowsky here, if we haven't solved alignment yet and we give AI to everyone then we are killing ourselves. If we have solved alignment, great, but that's the difficult part and we haven't done that yet. That point and a few others.)
For the sake of discussion, let's suppose that the next big escalation in AI power is the final one, and that it's less than five years away. Any thoughts on what the best course of action is?
Warm take: Our main hope lies in the EA/longtermist/AI-safety community. I say this because my first-order answer to your question is "no idea," so instead I go meta and say "Rally the community. Get organized. Create common knowledge. Get the wisest people in a room together to discuss the problem and make a plan. Everybody stick to the plan." (The plan will of course be a living document that updates over time in response to new info. The point is that for a crisis, you need organization; you need a chain of command. It seems to be the main way that humans are able to get large numbers of people working effectively together on short notice. The main problem with organizations is that they tend to become corrupt and decay over time, hence the importance of competition/markets/independence. However this is less of a problem on short timescales and anyway what choice do we have?)
I don't know what you mean by skipped. Here's some more concreteness though:
--Thanks to OpenAI, there is more of an "AI research should be made available to everyone" ethos, more of a "Boo anyone who does AI research and doesn't tell the world what they did or how they did it or even decides not to share the weights!" Insofar as this ethos persists during the crucial period, whichever labs are building AGI will be under more internal and external pressure to publish/share. This makes it harder for them to go slow and be cautious when the stakes are high.
--Thanks to OpenAI, there were two world-leading AGI labs, not one. Obviously it's a lot harder to coordinate two than one. This is not as bad as it sounds because plausibly before the crucial period more AGI labs would have appeared anyway. But still.
--Thanks to OpenAI, scaling laws and GPT tech are public knowledge now. This is a pretty big deal because it's motivating lots of other players to start building AGI or AGI-like things, and because it seems to be opening up lots of new profit opportunities for the AI industry which will encourage further investment, shortening timelines and increasing the number of actors that need to coordinate. Again, presumably this would have happened eventually anyway. But OpenAI made it happen faster.
Ironically, I am a believer in FOSS AI models, and I find OpenAI’s influence anything but encouraging in this regard. The only thing they are publicly releasing is marketing nowadays.
Yep! :) But the damage is done; thanks to OpenAI there is now a large(r) group of people who believe in FOSS AI than there otherwise would be, and there are various new actors who have entered the race who wouldn't have if not for OpenAI publications.
To be clear, I'm not confident FOSS AI is bad. I just think it probably is, for reasons mentioned. I won't be too surprised if I turn out to be wrong and actually (e.g. for reasons Dirichlet-to-Neumann mentioned, or because FOSS AI is harder to make a profit on and therefore will get less investment and therefore will be developed later, buying more time for safety research) the FOSS AI ethos was net-positive.
I'd be interested to hear your perspective on FOSS AI.
Epistemic status: I am not an expert on this debate, I have not thought very deeply about it, etc.
Thanks! In case you are interested in my opinion: I think I agree with 1 (but I expect us to fail miserably) and 2 and 6. I disagree with 3 (AGI, unlike nukes, can be used in ways that aren't threatening and don't hurt anyone and just make loads of money and save loads of lives. So people will find it hard to resist using it. So the more people have access to it, the more likely it is it'll be used before it is safe.) My timelines are about 50% by 10 years; I gather from point 4 that yours are longer. I think 5 might be true but might not be; history is full of examples of different groups fighting each other yet still managing to conquer and crush some third group. For example, the Spanish conquistadors were fighting each other even as they conquered Mexico and Peru. Maybe humans will be clever enough to play the AIs off against each other in a way that lets us maintain control until we solve alignment... but I wouldn't bet on it.
It seems to me that being open about what you are working on, and having a proven record of publishing/sharing critical informations, including weights, is a very good way to fight the arm race.
If you don't know where your concurrent are, it is much more difficult to stop to think about alignment than to rush toward capacity first. If you know where your concurrent are, and if you know that you will be at worst a couple weeks or months late because they always publish and you will thus be able to catch up, you have much more slack to pursue alignment (or speculative research in general).
For the strategic arms reduction treaties signed between Russia and the USA, verification tools were a crucial part of the process, because you need to know what the other is doing to disarm:
https://en.wikipedia.org/wiki/START_I#Verification_tools
https://en.wikipedia.org/wiki/New_START#Monitoring_and_verification
Yes, when we are getting really close to AGI it will be good for the leading contenders to share info with each other. Even then it won't be a good idea for the leading contenders to publish publicly, because then there'll be way more contenders! And now, when we are not really close to AGI, public publication accelerates research in general and thus shortens timelines, while also bringing more actors into the race.
Trust between partners do not happen overnight. You don't suddenly begin sharing information with concurrents when the prize is in sight. We need a history of shared information to build upon, and now - when, as you said, AGI is not really close - is the good time to build it.
Because if you don't trust someone with GPT-3, you are certainly not going to trust them with an AGI.
Because if you don't trust someone with GPT-3, you are certainly not going to trust them with an AGI.
Choosing to not release GPT-3's weights to the whole world doesn't imply that you don't trust DeepMind or Anthropic or whoever. It just implies that there exists at least one person in the world you don't trust.
I agree that releasing everything publicly would make it easier/more likely to release crucial things to key competitors when the time comes. Alas, the harms are big enough to outweigh this benefit, I think.
Nobody involved with OpenAI's launch can reasonably have been said to have done anything else of relative importance in their lives. The net impact of their lives is their contribution to the huge negative impact of OpenAI's launch, plus a rounding error.
Using this statement to describe Elon Musk and Sam Altman seems to imply that founding a single AI company is much more important than privatizing spaceflight, inventing practical self-driving cars and leading Y-Combinator.
From the little I know about how Elon Musk and Sam Altman they see the world, both of them would agree AI as the most important issue near-term for humanity and they started OpenAI in order to do something about it. The question isn't whether OpenAI is important. It's whether OpenAI has had a positive or negative effect.
OpenAI is the only organization I know of which is explicitly dedicated to AI safety and is pushing the technical field of AI forward. This seems like a good thing to me. Pushing the technical field of AI forward is how you provide an empirical test of whether you know what you're talking about. If you do the AI safety without the technical advancement you can get lost in an ivory tower.
I'm curious about this whole "nascent spirit of cooperation" thing. We're a species that can kinda sorta cooperate on nuclear Armageddon, carbon emissions and vaccines. Cooperation on AI seems like a much harder problem because the capital expenditures are so low, the strategic advantage is so high and the technology advances so fast.
Using this statement to describe Elon Musk and Sam Altman seems to imply that founding a single AI company is much more important than privatizing spaceflight, inventing practical self-driving cars and leading Y-Combinator.
From Eliezer's perspective that's the case. Privatized spaceflight or self-driving cars won't change the likelihood that humanity survives signficiantly and Eliezer sees the amount of value that can be created if humanity survives and there's FAI as big enough that those other things are relatively unimportant.
Agree with we should reach out to him & the community is connected enough to do so. If he's concerned about AI risk but either being misguided or doing harm (see e.g. here/here and here), then someone should just... talk to him about it? The richest man in the world can do a lot either way. (Especially someone as addicted to launching things as him, who knows what detrimental thing he might do next if we're not more proactive.)
I get the impression the folks at FLI are closest to him so maybe they are the best ones to do that.
I'm aware. I'm just saying a new effort is still needed because his thoughts on alignment/AI risk are still clearly very misguided listening to all his recent public comments on the topic and what he's trying to do with Neuralink etc. so someone really needs to reach out and set him straight.
This incident makes me feel sympathy for people whose words have more than personal consequence. And for twitter users. Seems like a tough place.
I assume Eliezer is at least 50% BSing, but now the words are going to get taken out of context and interpreted as some kind of official statement. Which, to be fair, should have been completely obvious.
I don't know why you think Eliezer is BSing at all. I agree he is an incredible troll in-person and is very funny, but this reads to me as straightforwardly describing how he sees the situation.
Yeah, I was assuming he was completely serious , otherwise I would have been hesitant to pick out the quote.
Going off of Musk's public persona, I think there's some not-insignificant chance that calling him out like Yudkowsky did will be counter-productive. If this is accurate and if Yudkowsky is correct about the negative value of OpenAI...Yudkowsky just put himself in the same shoes as everyone he's describing as having contributed to OpenAI's success by causing Musk to dig his heels in on the subject.
Note that Musk parted ways with OpenAI back in 2018, in part because of a conflict of interest (between Tesla and OpenAI).
You know, I actually knew this, but kind of forgot about it momentarily! I think it was because of the framing of the post that almost makes it sound like an ongoing concern of Musk's.
That being said, I wouldn't be surprised to hear if the general point I'm making holds for at least some subset of current OpenAI people.
Anyone want to make a Downfall meme video titled "MIRI reacts to the announcement of OpenAI?" I for one could sure use a bit of humor to be less depressed, and Yudkowsky's tweet provides plenty of juicy source material.
I'm unsure why this is getting downvoted. Is there something wrong with downfall memes in particular, or is it just the general idea of making memes about this situation that people don't like?
I downvoted it because it's unimportant, and I don't want it distracting from the conversation. If I'd found it funny or clever, I'd have upvoted it or abstained.
On the meta level: I tentatively think LW is too sparing with downvotes, and I think asking people to go into why they didn't find your joke funny (or didn't find it useful, or just hate laughter and sunshine), in response to very small amounts of net downvoting (e.g., -5 rather than -30), makes it a bit harder to shift that norm. I guess I feel like I can guess the general character of others' downvotes, and I don't understand the decision-relevance of investigating the details.
Now please downvote this comment to the extent it was bad / you want to see fewer things like it in the future / you want it to be less prominent on this page. :P
Eliezer Yudkowsky writes on twitter:
Nothing else Elon Musk has done can possibly make up for how hard the "OpenAI" launch trashed humanity's chances of survival; previously there was a nascent spirit of cooperation, which Elon completely blew up to try to make it all be about who, which monkey, got the poison banana, and by spreading and advocating the frame that everybody needed their own "demon" (Musk's old term) in their house, and anybody who talked about reducing proliferation of demons must be a bad anti-openness person who wanted to keep all the demons for themselves.
Nobody involved with OpenAI's launch can reasonably have been said to have done anything else of relative importance in their lives. The net impact of their lives is their contribution to the huge negative impact of OpenAI's launch, plus a rounding error.
Previously all the AGI people were at the same conference talking about how humanity was going to handle this together. Elon Musk didn't like Demis Hassabis, so he blew that up. That's the impact of his life. The end.
I've found myself repeatedly uncertain about what to make of OpenAI and their impact. The most recent LessWrong discussion that I'm aware of has happened on the post Will OpenAI-s work unintentionally increase existential risk, but the arguments for negative impact are different from the thing Eliezer named.
I'm also not entirely sure whether publicly debating sensitive questions like whether a person or organization accidentally increased existential risk is a good idea in the first place. However, spontaneously bringing up the issue on a twitter thread is unlikely to be optimal. At the very least, it should be beneficial to discuss the meta question, i.e., how we should or shouldn't talk about this. With that in mind, here are three things I would like to understand better:
And -- conditional on discussing them being a good idea:
What is the more detailed story of how the "nascent spirit of cooperation" has degraded or changed since the inception of OpenAI?
What interventions are possible here, if any? (Is it really that difficult to organize some kind of outreach to Elon to try and reverse some of the effects? Naively speaking, my impression has been that our community is sufficiently well connected to do this, and that Elon is amenable to arguments.)
I'm less interested in estimating the total impact of any specific person.