Dialogue on the Claim: "OpenAI's Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI"

Ruby

There's apparently been a lot of EA-hate on twitter as a result. I personally expect this to matter very little, if at all, in the long run, but I'd expect it to be extremely disproportionately salient to rationalists/EAs/alignment folk.

I think this matters insofar as thousands of tech people just got radicalized into the "accelerationist" tribe, and have a narrative of "EAs destroy value in their arrogance and delusion." Whole swaths of Silicon Valley are now primed to roll their eyes at and reject any technical, governance, or policy proposals justified by "AI safety."

See this Balaji thread for instance, which notably is close to literally correct (like, yeah, I would endorse "anything that reduces P(doom) is good"), but slips in the presumption that "doomers" are mistaken about which things reduce P(doom) / delusional about there being doom at all. Plus a good dose of the attack-via-memetic-association (comparing "doomers" to the unibomber). This is basically just tribalism.

I don't know if this was inevitable in the long run. It seems like if the firing of Sam Altman was handled better, if they had navigated the politics so that Sam "left to spend more time with his family" or whatever is typical when you oust a CEA, we could have avoided this impact.

6ChristianKl2y

I'm not sure. This episode might give some people the idea that Sam Altman position in regards to regulation is a middle of two extremes when they previously opposed the regulations that Sam sought as being too much.

5JenniferRM2y

That's part of the real situation though. Sam would never quit to "spend more time with his family". Link in sauce.

5Eli Tyre2y

He can leave "to avoid a conflict of interest with AI development efforts at Tesla", then. It doesn't have to be "relaxation"-coded. Let him leave with dignity. Insofar as Sam would never cooperate with the board firing him at all, even more gracefully, then yeah, I guess the board never really had any governance power at all, and it's good that the fig leaf has been removed. And basically, we'll never know if they bungled a coup that they could have pulled off if they were more savvy, or if this was a foregone conclusion.

5JenniferRM2y

With apologies for the long response... I suspect the board DID have governance power, but simply not decisive power. Also it was probably declining, and this might have been a net positive way to spend what remained of it... or not? It is hard to say, and I don't personally have the data I'd need to be very confident. "Being able to maintain a standard of morality for yourself even when you don't have all the data and can't properly even access all the data" is basically the core REASON for deontic morality, after all <3 Naive consequentialism has a huge GIGO data problem that Kant's followers do not have. (The other side of it (the "cost of tolerated ignorance" so to speak) is that Kantian's usually are leaving "expected value" (even altruistic expected value FOR OTHERS) on the table by refraining from actions that SEEM positive EV but which have large error bars based on missing data, where some facts could exist that they don't know about that would later cause them to have appeared to lied or stolen or used a slave or run for high office in a venal empire or whatever.) I personally estimate that it would have been reasonable and prudent for Sam to cultivate other bases of power, preparing for a breach of amity in advance, and I suspect he did. (This is consistent with suspecting the board's real power was declining.) Conflict in general is sad, and often bad, and it usually arises at the boundaries where two proactive agentic processes show up with each of them "feeling like Atlas" and feeling that that role morally authorizes them to regulate others in a top-down way... to grant rewards, or to judge conflicts, or to sanction wrong-doers... ...if two such entities recognize each other as peers, then it can reduce the sadness of their "lonely Atlas feelings"! But also they might have true utility functions, and not just be running on reflexes! Or their real-agency-echoing reflexive tropisms might be incompatible. Or mixtures thereof? Something I think I

[-]faul_sname2y*5231

There are what get called "stand by the Levers of Power" strategies and I don't know if they're good, but things like getting into positions within companies and governments that let you push for better AI outcomes, and I do think SBF might have made that a lot harder.

I think this is an important point: one idea that is very easy to take away from the FTX and OpenAI situations is something like

People associated with EA are likely to decide at some point that the normal rules for the organization do not apply to them, if they expect that they can generate a large enough positive impact in the world by disregarding those rules. Any agreement you make with an EA-associated person should be assumed to have an "unless I think the world would be better if I broke this agreement" rider (in addition to the usual "unless I stand to personally gain a lot by breaking this agreement" rider that people already expect and have developed mitigations for).

Basically, I expect that the strategy of "attempt to get near the levers of power in order to be able to execute weird plans where, if the people in charge of the decision about whether to let you near the levers of power knew about your pl... (read more)

[-]Ruby2y2628

People associated with EA are likely to decide at some point that the normal rules for the organization do not apply to them, if they expect that they can generate a large enough positive impact in the world by disregarding those rules.

I am myself consequentialist at my core, but invoking consequentialism to justify breaking commitments, non-cooperation, theft, or whatever else is just a stupid bad policy (the notion of people doing this generates some strong emotions for me) that as a policy/algorithm, won't result in accomplishing one's consequentialist goals.

I fear what you say is not wholly inaccurate and is true of at least some in EA, I hope though it's not that true of many.

Where it does get tricky is potentially unilateral pivotal acts about which I think go in this direction but also feel different from what you describe.

[-]mtaran2y446

A few others have commented about how MSFT doesn't necessarily stifle innovation, and a relevant point here is that MSFT is generally pretty good at letting its subsidiaries do their own thing and have their own culture. In particular GitHub (where I work), still uses Google Workspace for docs/email, slack+zoom for communication, etc. GH is very much remote-first whereas that's more of an exception at MSFT, and GH has a lot less suffocating bureaucracy, and so on. Over the years since the acquisition this has shifted to some extent, and my team (Copilot) is more exposed to MSFT than most, but we still get to do our own thing and at worst have to jump through some hoops for compute resources. I suspect if OAI folks come under the MSFT umbrella it'll be as this sort of subsidiary with almost complete ability to retain whatever aspects of its previous culture that it wants.

Standard disclaimer: my opinions are my own, not my employer's, etc.

[-]Vladimir_Nesov2y272

For me the crux is influence of these events on Sutskever ending up sufficiently in charge of a leading AGI project. It appeared borderline true before; it would've become even more true than that if Altman's firing stuck without disrupting OpenAI overall; and right now with the strike/ultimatum letter it seems less likely than ever (whether he stays in an Altman org or goes elsewhere).

(It's ambiguous if Anthropic is at all behind, and then there's DeepMind that's already in the belly of Big Tech, so I don't see how timelines noticeably change.)

9mishka2y

Exactly. And then one's estimate of the actual impact depends on whether one believes Sutskever is one of the best people to lead an AI existential safety effort. If one believes that, and if the outcome is that he ends up less likely to do so in the context of the leading AGI/ASI project, then the impact on safety might be very negative. If one does not believe that he is one of the best people to lead this kind of effort, then one might think that the impact is not negative. ---------------------------------------- (I personally believe Ilya's approach is one of the better ones, and it seems to me that he has been in the process of fixing the defects in the original OpenAI superalignment plan, and basically trying to gradually create a better plan, but people's views on that might differ.)

[-][anonymous]2y235

The main reason I think a split OpenAI means shortened timelines is that the main bottleneck to capabilities right now is insight/technical-knowledge. Quibbles aside, basically any company with enough cash can get sufficient compute. Even with other big players and thousands/millions of open source devs trying to do better, to my knowledge GPT4 is still the best, implying some moderate to significant insight lead. I worry by fracturing OpenAI, more people will have access to those insights, which 1) significantly increases the surface area of people working on the frontiers of insight/capabilities, 2) we burn the lead time OpenAI had, which might otherwise have been used to pay off some alignment tax, and 3) the insights might end up at a less scrupulous (wrt alignment) company.

A potential counter to (1): OpenAI's success could be dependent on having all (or some key subset) of their people centralized and collaborating.

Counter-counter: OpenAI staff, especially the core engineering talent but it seems the entire company at this point, clearly wants to mostly stick together, whether at the official OpenAI, Microsoft, or with any other independent solution. So them moving... (read more)

[-]Stephen McAleese2y177

GPT-4 is the model that has been trained with the most training compute which suggests that compute is the most important factor for capabilities. If that wasn't true, we would see some other company training models with more compute but worse performance which doesn't seem to be happening.

[-]Vladimir_Nesov2y*100

Falcon-180b illustrates how throwing compute at an LLM can result in unusually poor capabilities. Epoch's estimate puts it close to Claude 2 in compute, yet it's nowhere near as good. Then there's the even more expensive PaLM 2, though since weights are not published, it's possible that unlike with Falcon the issue is that only smaller, overly quantized, or incompetently tuned models are being served.

[-]Ben Pace2y221

FTR I am not spending much time calculating the positive or negative direct effect of this firing. I am currently pretty concerned about whether it was done honorably and ethically or not. It looks not to me, and so I oppose it regardless of the sign of the effect.

7Eli Tyre2y

What does the ethical / honorable version of the firing look like?

[-]Ben Pace2y18-8

It entirely depends on the reasoning.

Quick possible examples:

"Altman, we think you've been deceiving us and tricking us about what you're doing. Here are 5 documented instances where we were left with a clear impression about what you'd do that is counter to what eventually occurred. I am pretty actively concerned that in telling you this, you will cover up your tracks and just deceive us better. So we've made the decision to fire you 3 months from today. In that time, you can help us choose your successor, and we will let you announce your departure. Also if anyone else in the company should ask, we will also show them this list of 5 events."
"Altman, we think you've chosen to speed ahead with selling products to users at the expense of having control over these home-grown alien intelligences you're building. I am telling you that there needs to be fewer than 2 New York Times pieces about us in the next 12 months, and that we must overall slow the growth rate of the company and not 2x in the next year. If either of these are not met, we will fire you, is that clear?"

Generally not telling the staff why was extremely disrespectful, and generally not highlighting it to him ahead of time, is also uncooperative.

[-]Lukas_Gloor2y*130

Maybe, yeah. Definitely strongly agree with not telling the staff a more complete story seems to be bad for both intrinsic and instrumental reasons.

I'm a bit unsure how wise it would be to tip Altman off in advance given what we've seen he can mobilize in support of himself.

And I think it's a thing that only EAs would think up that it's valuable to be cooperative towards people who you're convinced are deceptive/lack integrity. [Edit: You totally misunderstood what I meant here; I was criticizing them for doing this too naively. I was not praising the norms of my in-group. Your reply actually confused me so much that I thought you were being snarky in some really strange way.] Of course, they have to consider all the instrumental reasons for it, such as how it'll reflect on them if others don't share their assessment of the CEO lacking integrity.

-1Ben Pace2y

Absolutely not. When I make an agreement to work closely with you on a crucial project, if I think you're deceiving me, I will let you know. I will not surprise backstab you and get on with my day. I will tell you outright and I will say it loudly. I may move quickly to disable you if it's an especially extreme circumstance but I will acknowledge that this is a cost to our general cooperative norms where people are given space to respond even if I assign a decent chance to them behaving poorly. Furthermore I will provide evidence and argument in response to criticism of my decision by other stakeholders who are shocked and concerned by it. Shame on you for suggesting only your tribe knows or cares about honoring partnerships with people after you've lost trust in them. Other people know what's decent too.

9Thane Ruthenis2y

That does not seem sensible to me. We're not demons for the semantics of our contracts to be binding. If you've entered an agreement with someone, and later learned that they intend (and perhaps have always intended) to exploit your acting in accordance with it to screw you over, it seems both common-sensically and game-theoretically sound to consider the contract null and void, since it was agreed-to based on false premises. If the other person merely turned out to be generically a bad/unpleasant person, with no indication they're planning to specifically exploit their contract with you? Then yes, backstabbing them is dishonorable. But if you've realized, with high confidence, that they're not actually cooperating with you the way you understood them to be, and they know they're not cooperating with you the way you expect them to be, and they deliberately maintain your misunderstanding in order to exploit you? Then, I feel, giving them advance warning is just pointless self-harm in the name of role-playing being cooperative; not a move to altruistically preserve social norms.

4Ben Pace2y

If you make a trade agreement, and the other side does not actually pay up, then I do not think you are bound to provide the good anyway. It was a trade. If you make a commitment, and then later come to realize that in requesting that commitment the other party was actually taking advantage of you, I think there are a host of different strategies one could pick. I think my current ideal solution is "nonetheless follow-through on your commitment, but make them pay for it in some other way", but I acknowledge that there are times when it's correct pick other strategies like "just don't do it and when anyone asks you why give them a straight answer" and more. Your strategy in a given domain will also depend on all sorts of factors like how costly the commitment is, how much they're taking advantage of you for, what recourse you have outside of the commitment (e.g. if they've broken the law they can be prosecuted, but in other cases it is harder to punish them). The thing I currently believe and want to say here is that it is not good to renege on commitments even if you have reason to, and it is better to not renege on them while setting the incentives right. It can be the right choice to do so in order to set the incentives right, but even when it's the right call I want to acknowledge that this is a cost to our ability to trust in people's commitments.

3Thane Ruthenis2y

Sure, I agree with all of the local points you've stated here ("local" as in "not taking account the particulars of the recent OpenAI drama"). For clarity, my previous disagreement was with the following local claim: In my view, "knowingly executing your part of the agreement in a way misaligned with my understanding of how that part is to be executed" counts as "not paying up in a trade agreement", and is therefore grounds for ceasing to act in accordance with the agreement on my end too. From this latest comment, it sounds like you'd agree with that? Reading the other branch of this thread, you seem to disagree that that was the situation in which the OpenAI board had been. Sure, I'm hardly certain of this myself. However, if it were, and if they were highly certain of being in that position, I think their actions are fairly justified. My understanding is that OpenAI's foundational conceit was prioritizing safety over profit/power-pursuit, and that their non-standard governance structure was explicitly designed to allow the board to take draconian measures if they concluded the company went astray. Indeed, going off these sort of disclaimers or even the recent actions, it seems they were hardly subtle or apologetic about such matters. "Don't make us think that you'd diverge from our foundational conceit if given the power to, or else" was part of the deal Sam Altman effectively signed by taking the CEO role. And if the board had concluded that this term was violated, then taking drastic and discourteous measures to remove him from power seems entirely fine to me. Paraphrasing: while in the general case of deal-making, a mere "l lost trust in you" is not reasonable grounds for terminating the deal, my understanding is that "we have continued trust in you" was part of this specific deal, meaning losing trust was reasonable grounds for terminating this specific deal. I acknowledge, though, that it's possible I'm extrapolating from the "high-risk investment" di

6Lukas_Gloor2y

I agree that there are versions of "agreeing to work closely together on the crucial project" where I see this as "speak up now or otherwise allow this person into your circle of trust." Once someone is in that circle, you cannot kick them out without notice just because you think you observed stuff that made you change your mind – if you could do that, it wouldn't work as a circle of trust. So, there are circumstances where I'd agree with you. Whether the relationship between a board member and a CEO should be like that could be our crux here. I'd say yes in the ideal, but was it like that for the members of the board and Altman? I'd say it depends on the specific history. And my guess is that, no, there was no point where the board could have said "actually we're not yet sure we want to let Altman into our circle of trust, let's hold off on that." And there's no yes without the possibility of no. I agree that one needs to do this if one lost faith in people who once made it into one's circle of trust. However, let's assume they were never there to begin with. Then, it's highly unwise if you're dealing with someone without morals who feels zero obligations towards you in response. Don't give them an advance warning out of respect or a sense of moral obligation. If your mental model of the person is "this person will internally laugh at you for being stupid enough to give them advance warning and will gladly use the info you gave against you," then it would be foolish to tell them. Batman shouldn't tell the joker that he's coming for him. What I meant to say in my initial comment is the same thing as you're saying here. "Acknowledging the cost" is also an important thing in how I think about it, (edit) but I see that cost as not being towards the Joker (respect towards him), but towards the broader cooperative fabric. [Edit: deleted a passage here because it was long-winded.] "If I assign a decent chance to them behaving poorly" – note that in my description,

6Ben Pace2y

1. I find the situation a little hard to talk about concretely because whatever concrete description I give will not be correct (because nobody involved is telling us what happened). Nonetheless, let us consider the most uncharitable narrative regarding Altman here, where members of the board come to believe he is a lizard, a person who is purely selfish and who has no honor. (To be clear I do not think this is accurate, I am using it for communication purposes.) Here are some rules. * Just because someone is a lizard, does not make it okay to lie to them * Just because someone is a lizard, does not make it okay to go back on agreements with them * While the lizard had the mandate to make agreements and commitments on behalf of your company, it is not now okay to disregard those agreements and commitments The situation must not be "I'll treat you honorably if I think you're a good person, but the moment I decide you're a lizard then I'll act with no honor myself." The situation must be "I will treat you honorably because it is right to be honorable." Otherwise the honor will seep out of the system as probabilities we assign to others' honor wavers. I think it is damaging to the trust people place in board members, to see them act with so little respect or honor. It reduces everyone's faith in one another to see people in powerful positions behave badly. II. I respect that in response to my disapproval of your statement, you took the time to explain in detail the reasoning behind your comment and communicate some more of your perspective on the relevant game theory. I think that generally helps, when folks are having conflicts, to examine openly the reasons why decisions were made and investigate those. And it also gives us more surface area to locate key parts of the disagreement. I still disagree with you. I think it was an easy-and-wrong thing to suggest that only people in the EA tribe would care about this important ethical principle I care about. But

6Ruby2y

I see interesting points on both sides here. Something about how this comment(s) is expressed makes me feel uncomfortable, like this isn't the right tone for exploring disagreements about correct moral/cooperative behavior, it at least it makes it a lot harder for me to participate. I think it's something like it feels like performing moral outrage/indignation in a way that feels more persuadey than explainy, and more in the direction of social pressure, norms-enforcery. The phrase "shame on you" is a particularly clear thing I'll point at that makes me perceive this.

4Lukas_Gloor2y

a) A lot of your points are specifically about Altman and the board, whereas many of my points started that way but then went into the abstract/hypothetical/philosophical. At least, that's how I meant it – I should have made this more clear. I was assuming, for the sake of the argument, that we're speaking of a situation where the person in the board's position found out that someone else is deceptive to their very core, with no redeeming principles they adhere to. So, basically what you're describing in your point "I" with the lizardpeople. I focused on that type of discussion because I felt like you were attacking my principles, and I care about defending my specific framework of integrity. (I've commented elsewhere on things that I think the board should or shouldn't have done, so I also care about that, but I probably already spent too many comments on speculations about the board's actions.) Specifically about the actual situation with Altman, you say: "I'm saying that you should honor the agreement you've made to wield your power well and not cruelly or destructively. It seems to me that it has likely been wielded very aggressively and in a way where I cannot tell that it was done justly." I very much agree with that, fwiw. I think it's very possible that the board did not act with integrity here. I'm just saying that I can totally see circumstances where they did act with integrity. The crux for me is "what did they believe about Altman and how confident were they in their take, and did they make an effort to factor in moral injunctions against using their power in a self-serving way, etc?" b) You make it seem like I'm saying that it's okay to move against people (and e.g. oust them) without justifying yourself later or giving them the chance to reply at some point later when they're in a less threatening position. I think we're on the same page about this: I don't believe that it would be okay to do these things. I wasn't saying that you don't have to st

5kave2y

I'm uncomfortable reading this comment. I believe you identified as an EA for much of your adult life, and the handle "EA" gives me lots of the bits that distinguish you from the general population. But you take for granted that Lukas' "EA" is meant to exclude you. By contrast, I would not have felt uncomfortable if you had been claiming adherence to the same standards by your childhood friends, startup culture, or some other clearly third party group.

4Ben Pace2y

Oh, but I don't mean to say that Lukas was excluding me. I mean he was excluding all other people who exist who would also care about honoring partnerships after losing faith in the counter-party, of which there are more than just me, and more than just EAs.

1kave2y

OK, I guess the confusion was that it seemed like your counter-evidence to his claim was about how you would behave.

2Ben Pace2y

Yep, I can see how that could be confusing in context.

4Linch2y

I think it was clear from context that Lukas' "EAs" was intentionally meant to include Ben, and is also meant as a gentle rebuke re: naivete, not a serious claim re: honesty.

4Linch2y

I feel like you misread Lukas, and his words weren't particularly unclear.

5Eli Tyre2y

This is a serious question, playfully asked. Ben, in private conversation recently, you said that you were against sacredness, because it amounted to deciding that there's something about which you'll ignore tradeoffs (feel free to correct my compression of you here, if you don't endorse it). So here's a tradeoff: what percentage probability of successfully removing Sam Altman would you trade for doing so honorably. How much does behaving honorably have to cause the success probability to fall before it's not worth it? Or is this a place where you would say, "no, I defy the tradeoff. I'll take my honor over any differences in success probability." (In which case, I would put forth, you regard honor as sacred.) : P

8Ben Pace2y

This question seems like a difficult interaction between utilitarianism and virtue ethics... I think not being honorable is in large part a question of strategy. If you don't honor implicit agreements on the few occasions when you really need to win, it's a pretty different strategy from whether you honor implicit agreements all of the time. So it's not just a local question to a single decision, it's a broader strategic question. I am sympathetic to consequentialist evaluations of strategies. I am broadly like "If you honor implicit agreements then people will be much more willing to trade with you and give you major responsibilities, and so going back on them in one occasion generally strikes down a lot of ways you might be able to effect the world." It's not just about this decision, but about an overall comparison of the costs and benefits of different kinds of strategies. There are many strategies one can play. I could potentially make up some fake numbers to give a sense of how different decisions change which strategies to run (e.g. people who play more to the letter than the spirit of agreements, people who will always act selfishly if the payoff is at least going to 2x their wealth, people who care about their counter-parties ending up okay, people who don't give a damn about their counter-parties ending up okay, etc). I roughly think much more honest and open, straightforward, pro-social, and simple strategies are widely trusted, better for keeping you and your allies sane, are more effective on the particular issues you care about, but less effective at getting generic un-scoped power. I don't much care about the latter relative to the first three so it seems to me way better at achieving my goals. I think it's extremely costly for trust to entirely change strategies during a single high-stakes decision, so I don't think it makes sense to re-evaluate it during the middle of the decision based on a simple threshold. (There could be observations that wo

2Eli Tyre2y

Right. I strongly agree. I think this is the at least part of the core bit of law that underlies sacredness. Having some things that are sacred to you is a way to implement this kind of reliable-across-all-worlds kind of policy, even when the local incentives might tempt you to violate that policy for local benefit.

3Ben Pace2y

Eh, I prefer to understand why the rules exist rather than blindly commit to them. Similarly, the Naskapi hunters used divination as a method of ensuring they'd randomize their hunting spots, and I think it's better to understand why you're doing it, rather than doing it because you falsely believe divination actually works.

[-]gwern2y*123

Minor point: the Naskapi hunters didn't actually do that. That was speculation which was never verified, runs counter to a lot of facts, and in fact, may not have been about aboriginal hunters at all but actually inspired by the author's then-highly-classified experiences in submarine warfare in WWII in the Battle of the Atlantic. (If you ever thought to yourself, 'wow, that Eskimo story sounds like an amazingly clear example of mixed-strategies from game theory'...) See some anthropologist criticism & my commentary on the WWII part at https://gwern.net/doc/sociology/index#vollweiler-sanchez-1983-section

2Eli Tyre2y

I certainly don't disagree with understanding the structure of good strategies!

8johnswentworth2y

(One could reasonably say that the upside of removing Sam Altman from OpenAI is not high enough to be worth dishonor over the matter, in which case percent chance of success doesn't matter that much. Indeed, success probabilities in practice only range over 1-2 orders of magnitude before they stop being worth tracking at all, so probably the value one assigns to removing Sam Altman at all dominates the whole question, and success probabilities just aren't that relevant.)

2Eli Tyre2y

Ok. Then you can least-convenient-possible-world the question. What's the version of the situation where removing the guy is important enough that the success probabilities start mattering in the calculus? Also to be clear, I think my answer to this is "It's just fine for some things to be sacred. Especially for features like honor or honesty, in which their strength comes from them being reliable under all circumstances (or at least that they have strength in proportion to the circumstances in which they hold).

2Ruby2y

My intuition (not rigorous) is there a multiple levels in the consequentialist/deontoligical/consequentialist dealio. I believe that unconditional friendship is approximately something one can enter into, but one enters into it for contingent reasons (perhaps in a Newcomb-like way – I'll unconditionally be your friend because I'm betting that you'll unconditionally be my friend). Your ability to credibly enter such relationships (at least in my conception of them) is dependent on you not starting to be more "conditional" because you doubt that the other person is also being there. This I think is related to not being a "fair weather" friend. I continue to be your friend even when it's not fun (you're sick, need taking care of whatever) even if I wouldn't have become your friend to do that. And vice versa. Kind of a mutual insurance policy. Same thing could be with contracts, agreements, and other collaborations. In a Newcomb-like way, I commit to being honest, being cooperative, etc to a very high degree even in the face of doubts about you. (Maybe you stop by the time someone is threatening your family, not sure what Ben, et al, think about that.) But the fact I entered into this commitment was based on the probabilities I assigned to your behavior at the start.

[-]peterbarnett2y2013

I think this misses one of the main outcomes I'm worried about, which is if Sam comes back as CEO and the board is replaced by less safety-motivated people. This currently seems likely (Manifold at 75% Sam returning, at time of posting).

You could see this as evidence that the board never had much power, and so them leaving doesn't actually change anything. But it seems like they (probably) made a bunch of errors, and if they hadn't then they would have retained influence to use to steer the org in a good direction.

(It is also still super unclear wtf is going on, maybe the board acted in a reasonable way, and can't say for legal (??) reasons.)

[-]johnswentworth2y10-1

You could see this as evidence that the board never had much power, and so them leaving doesn't actually change anything.

In the world where Sam Altman comes back as CEO and the board is replaced by less safety-motivated people (which I do not currently expect on an inside view), that would indeed be my interpretation of events.

[-]junk heap homotopy2y1513

Time to update then 🥲

https://x.com/OpenAI/status/1727206187077370115?s=20

4tremmor192y

Yea, im with you on it being a false dichotomy: some people are saying that because the board lost when they tried to use their legal power, it means they never had any power to begin with, so it doesnt matter. It seems plausible they had some level of power, but it was fragile and subject to failure if used carelessly in the wrong circumstances. Like, well, most real world political power.

3Thane Ruthenis2y

I'd model it as them having had the amount of power equal to their on-paper power times Sam's probability that they can successfully wield it. Being perceived as having social power is what having social power means, after all. I doubt he'd been certain they'd lose in a conflict like this, so he would've been at least a bit wary of starting it, i. e. would've shied away from actions that the board would dislike. Now it's established knowledge that they have no real power, and so they truly don't have it anymore, and so Sam is free to do whatever he wants and he at last knows it.

[-]Bird Concept2y207

In the poll most people (31) disagreed with the claim John is defending here, but I'm tagging the additional few (3) who agreed with it @Charlie Steiner @Oliver Sourbut @Thane Ruthenis

Interested to hear your guys' reasons, in addition to John's above!

[-]Oliver Sourbut2y*3727

Quick dump.

Impressions

Having met Sam (only once) it's clear he's a slick operator and is willing to (at least) distort facts to serve a narrative
I tentatively think Sam is one of the 'immortality or die trying' crowd (which is maybe acceptable for yourself but not when gambling with everything else too)
Story from OpenAI leadership re racing has always struck me as suspicious rationalisation (esp re China)
- 'You aren't stuck in traffic/race; you are the traffic/race'
A few interactions with OpenAI folks weakly suggests even the safety-conscious ones aren't thinking that clearly about safety
I've been through org shakeups and it tends to take time to get things ticking over properly again, maybe months (big spread)

Assumptions

I admit I wasn't expecting Sam to come back. If it sticks, this basically reverses my assessment!
I've been assuming that the apparent slavish loyalty of the mass of employees is mostly a fog of war illusion/artefact
- the most suckered might follow, but I'm tentatively modelling those as the least competent
- crux: there aren't large capabilities insights that most employees know
- looks like we won't get a chance to find out (and nor will they!)

Other

... (read more)

[-]Thane Ruthenis2y155

Hard to say what the effect is on Sam's personal brand; lots to still cash out I expect. It could enhance his charisma, or he might have spent something which is hard to get back.

I think my model of cycles of power expansion and consolidation is applicable here:

When you try to get something unusual done, you "stake" some amount of your political capital on this. If you win, you "expand" the horizon of the socially acceptable actions available to you. You start being viewed as someone who can get away with doing things like that, you get an in with more powerful people, people are more tolerant of you engaging in more disruptive action.
But if you try to immediately go for the next, even bigger move, you'll probably fail. You need buy-in from other powerful actors, some of which have probably only now became willing to listen to you and entertain your more extreme ideas. You engage in politicking with them, arguing with them, feeding them ideas, establishing your increased influence and stacking the deck in your favor. You consolidate your power.

I'd model what happened as Sam successfully expanding his power. He's staked some amount of his political capital on the counter-revolution,... (read more)

3Oliver Sourbut2y

Yep, this is a nicely explained and more detailed version of 'it could enhance his charisma' and is my model there too. But there's also a lot of latent (to us) stuff that might have been traded away, and furthermore a bunch of public 'potentially reputation-tarnishing' fallout which might swing it the other way.

7Charlie Steiner2y

I guess the big questions for me were "relative to what?", "did the board have good-faith reasons?", and "will the board win a pyrrhic victory or get totally routed?" At the time of answering I thought the last two answers were: the board probably had plausible reasons, and they would probably win a pyrrhic victory. Both of these are getting less likely, the second faster than the first. So I think it's shaping up that I was wrong and this will end up net-negative. Relative to what? I think I mean "relative to the board completely checking out at their jobs," not "relative to the board doing their job masterfully," which I think would be nice to hope for but is bad to compare to.

3Oliver Sourbut2y

Thane has a good take in this comment. I'm pointing at the difference between 'evidential' and 'causal' (?) updates in my answer but Thane does it more clearly.

[-]Thane Ruthenis2y1211

And on top of that, my not-very-informed-impression-from-a-distance is that [Sam]'s more a smile-and-rub-elbows guy than an actual technical manager

I agree, but I'm not sure that's insufficient to carve out a productive niche at Microsoft. He appears to be a good negotiator, so if he goes all-in spending his political capital to ensure his subsidiary isn't crippled by bureaucracy, he has a good chance of achieving it.

The questions are (1) whether he'd realize he needs to do that, and (2) whether he'd care to do that, versus just negotiating for more personal power and trying to climb to Microsoft CEO or whatever.

(1) depends on whether he's actually generally competent (as in, "he's able to quickly generalize his competence to domains he's never navigated before"), as opposed to competent-at-making-himself-appear-competent.
- I've never paid much attention to him before, so no idea on him specifically. On priors, though, people with his profile are usually the latter type, not the former.
(2) depends on how much he's actually an AGI believer vs. standard power-maximizer who'd, up to now, just been in a position where appearing to be an AGI believer was aligned with maximizing power.
- The

... (read more)

[-]orthonormal2y103

I mean, I don't really care how much e.g. Facebook AI thinks they're racing right now. They're not in the game at this point.

The race dynamics are not just about who's leading. FB is 1-2 years behind (looking at LLM metrics), and it doesn't seem like they're getting further behind OpenAI/Anthropic with each generation, so I expect that the lag at the end will be at most a few years.

That means that if Facebook is unconstrained, the leading labs have only that much time to slow down for safety (or prepare a pivotal act) as they approach AGI before Facebook gets there with total recklessness.

If Microsoft!OpenAI lags the new leaders by less than FB (and I think that's likely to be the case), that shortens the safety window further.

I suspect my actual crux with you is your belief (correct me if I'm misinterpreting you) that your research program will solve alignment and that it will not take much of a safety window for the leading lab to incorporate the solution, and therefore the only thing that matters is finishing the solution and getting the leading lab on board. It would be very nice if you were right, but I put a low probability on it.

[-]Ruby2y95

I was going to write stuff about integrity, and there's stuff to that, but the thing that is striking me most right now is that the whole effort seemed very incompetent and naive. And that's upsetting.

I am now feeling uncertain about the incompetence and naivety of it. Whether this was the best move possible that failed to work out, or best move possible that actually did get a good outcome, or a total blunder is determined by info I don't have.

I have some feeling of they were playing against a higher-level political player which both makes it hard but als... (read more)

[-]Thane Ruthenis2y120

The initial naive blunder was putting Sam Altman in CEO position to begin with. It seems like it was predictable-in-advance (from e. g. Paul Graham's comments from years and years ago) that he's not the sort of person to accept being fired, rather than mounting a realpolitik-based counteroffensive, and that he would be really good at the counteroffensive. Deciding to hire him essentially predestined everything that just happened; it was inviting the fox into the henhouse. OpenAI governance controls might have worked if the person subjected to them was not specifically the sort of person Sam is.

How was the decision to hire him made, and under what circumstances?

What needs to happen for this sort of mistake not to be repeated?

[-]interstice2y81

Missing from this discussion is the possibility that Sam might be reinstated as CEO, which seems like a live option at this point. If that happens I think it's likely that the decision to fire him was a mistake.

[-][anonymous]2y83

I think this discussion is too narrow and focused on just Sama and Microsoft.

The global market "wants" AGI, ASI, human obsolescence*.

The consequences of this event accelerate that:

Case 1: Microsoft bureaucracy drags Sama's teams productivity down to zero. In this case, OpenAI doesn't develop a GPT-5, and Microsoft doesn't release a better model either. This opens up the market niche for the next competitor at a productive startup to develop the model, obviously assisted by former openAI employees who bring all the IP with them, and all the money and b

... (read more)

5Thane Ruthenis2y

While you have a point, I think this model might have too much faith in the Efficient Market Hypothesis. It's true that the market "wants" human obsolescence, in the sense that companies that sell it would earn a ton of money. But if DeepMind and Anthropic went bankrupt tomorrow, it's not obvious that anyone would actually step up to fill the niche left by them. Market failures are common, and the major AI Labs arguably sprung up because of ideological motives; not profit motives. Because of people buying into EA-style AI risk and deciding to go about solving it in a galaxy-brained manner. They're something like high-risk high-reward startups. And if a speculative startup fails, it's not at all obvious that a replacement startup will spring up to fill its niche the very next day, even if the idea were solid. (Or, another analogy, they're something like institutions concerned with fundamental theoretical research, and those famously need to be funded by governments, not the market.) On the contrary, the reference point for how AGI-research-based-on-profit-motives goes is FAIR, and FAIR is lagging behind. The market, left to its own devices, will push for finding LLM applications or scaling them in the dumbest manner possible or such; not for racing for AGI. I don't have so little faith in the market as to completely discount your model, but I wouldn't assign it 100% either.

2[anonymous]2y

Were the ai startups high risk and dependent on investor largess to exist, then absolutely I would agree with your model. But : https://sacra.com/c/openai/#:~:text=OpenAI has surpassed %241.3B,at the end of 2022. Assume openAI was collecting a 10 percent profit margin (the other 90 percent paid for compute). Allegedly they burned about 540 million in 2022 when gpt-4 was developed. Call it 1 billion total cost for a gpt-4 (compute + staff compensation). Then 130 million annual net revenue on a 1 billion investment is 13 percent ROI. In terms of "monetary free energy" that's net output. Large businesses exist that run on less margin. I am not specced in economics or finance but that looks like a sustainable business, and it's obviously self amplifying. Assuming the nascent general ai* industry has easy short term potential growth (as in, companies can rent access to a good model and collect many more billions) then it's possibly self sustaining. Even without outside investment some of those profits would get invested into the next model, and so on. You can also think of a second form of "revenue" as investor hype. Fresh hype is created with each major improvement in the model that investors can perceive, and they bring more money each round. While yes the EMH is imperfect,* investors clearly see enormous profit in the short term from general ai. This is the main thing that will drive the general ai industry to whatever is technically achievable : tens of billions in outside money. And yes, blowing up openAI slows this down...but the investors who were willing to give openAI tens of billions in a few weeks still have their money. Where's it going to go next? * By general ai I mean a model that is general purpose with many customers without needing expensive modifications. Different from AGI, which means also human level capabilities. General ai appears, from the data above, to be financially self sustaining even if it is not human level. *As other lesswrong po

7Thane Ruthenis2y

"Make a giant LLM and deploy it" is a self-sustaining business, yes, and if all major AI Labs died tomorrow a plethora of companies filling the niche of "make a giant LLM and deploy it" would spring up, yes. "Re-invest revenue into making an even larger LLM" is a sensible company policy, as well. But is "have a roadmap to AGI and invest into research that brings your models closer to it, even if that doesn't immediately translate into revenue" a self-sustaining business model? I'm much less confident on that. It's possible that the ideas of "AGI = money" have already propagated enough that profit-oriented non-imaginative business people would decide to spend their revenue on that. But that's not obvious to me. I expect the non-ideologically-motivated replacements for the current major AI labs to just have no idea what "racing to AGI" even means, in terms of "what research directions to pursue" as opposed to "what utterances to emit". The current AI industry as a whole is pretty bad at it as-is, but the major AI labs explicitly have some vision of what it physically means. I don't expect the replacements for them that the market would generate on its own to have even that much of a sense of direction. Again, it's possible that it's no longer true, that the ideas propagated enough that some of the "native" replacement companies would competently race as well. But it's not an open-and-shut case, I think.

4[anonymous]2y

So your model is that people can make big llms, and the innovation from openAI and from open source will eventually all be in one large model. Aka "gpt 4.1". But that each llm shop, while free of encumbrances and free to seek maximum profit, would not have the necessary concentration of money and talent in one place to develop AGI. Instead they would simply keep making smaller delta's to their product, something a less talented and GPU poorer crew could do, and capabilities would be stuck in a local minimum. So you believe that either this would push back AGI several years (eventually the staff at these smaller shops would skill up from experience and as compute gets cheaper they would eventually have what 100B of compute will buy in 2024) or possibly longer if there is no smooth path of small incremental steps from gpt-4.1 to AGI. I will add one comment to this : it's not actually a threshold of "gpt4.1 to AGI". Assuming you believe RSI will work, you need "a good enough seed model plus sufficient compute to train and benchmark thousands of automatically generated AGI candidates". Gpt4.1 plus a reinforcement learning element might be enough for the "seed AI".

4Thane Ruthenis2y

That summary sounds right, yep! Except that. It might, but I don't think that's particularly likely.

2Vladimir_Nesov2y

Giant LLMs are as useful as they are agentic (with ability to remain aware of a specific large body of data and keep usefully chipping away at a task), which doesn't seem particularly different from AGI as a direction (at least while it hasn't yet been walked far enough to tell the difference). The distinction is in AGI being a particular crucial threshold of capability that local pursuit of better agentic LLMs will ignore until it's crossed.

4johnswentworth2y

Wait, is OpenAI net positive profit on marginal users? I had assumed probably not, although it's not particularly central to any of my models right at the moment.

4[anonymous]2y

As Zvi mentioned in one of the roundups, the conventional wisdom for entering a new monopolistic tech niche is to grow as fast as possible. So it's likely that OpenAI loses money per user. GitHub copilot allegedly costs $40 in compute per $20 a month subscriber. So yes, you are right, but no, it doesn't matter. This is because there's other variables. The cost of compute is driven up by outside investment. If somehow dynamiting openAI causes all the outside investors to go invest somewhere else - sort of like the hype cycles for nft or crypto - the cost of compute would drop. For example Nvidia is estimated to pay $3000 to build each H100. If Nvidia charges $5000 a card, and stops charging a 20 percent software license fee, that essentially cuts the compute cost by more than half*, making current AI models at current prices more than profitable. Nvidia would do this in the hypothetical world of "investors get bored and another ai winter begins". This neglects Nvidia reducing their costs and developing a cheaper to build card per unit of LLM performance, which they obviously are doing. *Quick and dirty sanity check: assuming 50 percent utilization (GPU is bounded by memory I/o then it would use $33,000 in electricity over 5 years and currently costs $50,000 at current prices, 25k is list price, 25k is license fee. Were Nvidia to simply charge a more modest margin the all in cost would drop from 83k to 38k. Data center electricity is probably cheaper than 13 cents but there are costs for backup power and other systems) Conclusion: what's different now is general ai is bringing in enough revenue to be a self sustaining business. It's not an industry that can fold and go dormant like failed tech startups that folded and the product or service they developed ceased to be available anywhere. The time to blow up openAI was prior to the release of chatGPT.

[-]Simon Fischer2y61

Microsoft is the sort of corporate bureaucracy where dynamic orgs/founders/researchers go to die. My median expectation is that whatever former OpenAI group ends up there will be far less productive than they were at OpenAI.

I'm a bit sceptical of that. You gave some reasonable arguments, but all of this should be known to Sam Altman, and he still chose to accept Microsoft's offer instead of founding his own org (I'm assuming he would easily able to raise a lot of money). So, given that "how productive are the former OpenAI folks at Microsoft?" is the crux of the argument, it seems that recent events are good news iff Sam Altman made a big mistake with that decision.

6Thane Ruthenis2y

Or if Sam Altman isn't actually primarily motivated by the desire to build an AGI, as opposed to standard power-/profit-maximization motives. Accelerationists are now touting him as their messiah, and he'd obviously always been happy to generate hype about OpenAI's business vision. But it's not necessarily the case that it translates into him actually believing, at the gut level, that the best way to maximize prosperity/power is to build an AGI. He may realize that an exodus into Microsoft would cripple OpenAI talent's ability to be productive, and do it anyway, because it offers him personally better political opportunities for growth. It doesn't even have to be a dichotomy of "total AGI believer" vs "total simulacrum-level-4 power-maximizer". As long as myopic political motives have a significant-enough stake in his thinking, they may lead one astray. "Doomers vs. Accelerationists" is one frame on this conflict, but it may not be the dominant one. "Short-sighted self-advancement vs. Long-term vision" is another, and a more fundamental one. Moloch favours capabilities over alignment, so it usually hands the victory to the accelerationists. But that only goes inasmuch as accelerationists' motives coincide with short-sighted power-maximization. The moment there's an even shorter-sighted way for things to go, an even lower energy-state to fall into, Moloch would cast capability-pursuit aside. The current events may (or may not!) be an instance of that.

1O O2y

He was involved in the rationalist circle for a long time iirc. His said social status would still matter in a post AGI world so I suspect his true goal is either being known forever as the person who came across AGI (status) or immortality related.

[-]the gears to ascension2y60

microsoft has put out some pretty impressive papers lately. not sure how that bodes for their overall productivity, of course.

[-]Feel_Love2y50

Thanks for the good discussion.

I could equally see these events leading to AI capability development speeding or slowing. Too little is known about the operational status quo that has been interrupted for me to imagine counterfactuals at the company level.

But that very lack of information gives me hope that the overall PR impact of this may (counterintuitively) incline the Overton window toward more caution.

"The board should have given the press more dirt to justify this action!" makes sense as an initial response. When this all sinks in, what will people ... (read more)

[-]Bird Concept2y50

One of my takeaways of how the negotiations went is that it seems sama is extremely concerned with securing access to lots of compute, and that the person who ultimately got their way was the person who sat on the compute.

The "sama running Microsoft" idea seems a bit magical to me. Surely the realpolitik update here should be: power lies in the hands of those with legal voting power, and those controlling the compute. Sama has neither of those things at Microsoft. If he can be fired by a board most people have never heard of, then for sure he can get fired... (read more)

[-]Vladimir_Nesov2y127

People seem to think he is somehow a linchpin of building AGI. Remind me... how many of OpenAI's key papers did he coauthor?

Altman's relevant superpowers are expertise at scaling of orgs and AI-related personal fame and connections making him an AI talent Schelling point. So wherever he ends up, he can get a world class team and then competently scale its operations. The personality cult is not specious, it's self-fulfilling in practical application.

2Eli Tyre2y

This seems right in principle, but I think he's way less likely to be fired by anyone at microsoft, because they can play a positive-sum political game together, which was (apparently) less true of Sam and the OpenAI board.

2Ruby2y

If he can lead an exodus from OpenAI to Microsoft, he can lead one from Microsoft to somewhere else.

1ErickBall2y

Here's a market, not sure how to define linchpin but we can at least predict whether he'll be part of it. https://manifold.markets/ErickBall/will-the-first-agi-be-built-by-sam?r=RXJpY2tCYWxs

[-]Noosphere892y*40

I tend to view the events of OpenAI's firing of Sam Altman much more ambiguously than others, and IMO, it probably balances out to nothing in the end, so I don't care as much as some other people here.

To respond more substantially:

From johnswentworth:

Here's the high-gloss version of my take. The main outcomes are:

The leadership who were relatively most focused on racing to AGI and least focused on safety are moving from OpenAI to Microsoft. Lots of employees who are relatively more interested in racing to AGI than in safety will probably follow. Micro

... (read more)

[-]FireStormOOO2y4-1

I wouldn't count on Microsoft being ineffective, but there's good reason to think they'll push for applications for the current state of the art over further blue sky capabilities stuff. The commitment to push copilot into every Microsoft product is already happening, the copilot tab is live in dozens of places in their software and in most it works as expected. It's already good enough to replace 80%+ of the armies of temps and offshore warm bodies that push spreadsheets and forms around today without any further big capabilities gains, and th... (read more)

[-]Jonas V2y30

Market on the primary claim discussed here:

[-]Hide2y30

It seems intuitively bad:

Capabilities and accelerationist-focused researchers have gone from diluted and restrained to concentrated and encouraged
Microsoft now has unbounded control, rather than a 49% stake
Microsoft cannot be expected to have any meaningful focus on alignment/safety
They are not starting from scratch: a huge chunk of their most capable staff and leadership will be involved
The "superalignment" project will be at best dramatically slowed, and possibly abandoned if OpenAI implodes
Other major labs smell blood in the water, possibly exacerbating race dynamics, not to mention a superficial increase (by 1) in the number of serious players.

[-]michael_mjd2y*3-2

One fear I have is that the open source community will come out ahead, and push for greater weight sharing of very powerful models.

Edit: To make more specific, I mean that the open source community will become more attractive, because they will say, you cannot rely on individual companies whose models may or may not be available. You must build on top of open source. Related tweet:

https://twitter.com/ylecun/status/1726578588449669218

Whether their plan works or not, dunno.

[-]trevor2y2-5

I read the whole thing, glad I did. It really makes me think that many of AI safety's best minds are doing technical work like technical alignment 8 hours a day, when it would be better for them to do 2 hours a day to keep their skills honed, and spend 6 hours a day acting as generalists to think through the most important problems of the moment.

They should have shared their reasons/excuses for the firing. (For some reason, in politics/corporate politics, people try to be secretive all the time and this seems-to-me to be very stupid in like 80+% of cases,

... (read more)

9Ruby2y

These recent events have me thinking the opposite: policy and cooperation approaches to making AI go well are doomed – while many people are starting to take AI risk seriously, not enough are, and those who are worried will fail to restrain those who aren't (where not being risked in a consequence of humans often being quite insane when incentives are at play). The hope lies in somehow developing enough useful AI theory that leading labs adopt and resultantly build an aligned AI even though they never believed they were going to cause AGI ruin. And so maybe let's just get everyone to focus on the technical stuff. Actually more doable than wrangling other people to not build unsafe stuff.

4trevor2y

That largely depends on where AI safety's talent has been going, and could go. I'm thinking that most of the smarter quant thinkers have been doing AI alignment 8 hours a day and probably won't succeed, especially without access to AI architectures that haven't been invented yet, and most of the people research policy and cooperation weren't our best. If our best quant thinkers are doing alignment research for 8 hours a day with systems that probably aren't good enough to extrapolate to the crunch time systems, and our best thinkers haven't been researching policy and coordination (e.g. historically unprecedented coordination takeoffs), then the expected hope from policy and coordination is much higher, and our best quant thinkers should be doing policy and coordination during this time period; even if we're 4 years away, they can mostly do human research for freshman and sophomore year and go back to alignment research for junior and senior year. Same if we're two years away.

[+]bluefalcon2y-180

LESSWRONG
Petrov Day
LW

LESSWRONG
Petrov Day
LW

73

Dialogue on the Claim: "OpenAI's Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI"

73

73