I agree with a small update in this direction.

The board has backed down after Altman rallied staff into a mass exodus.

How would that be bad if you were trying to shut it down? [On edit: how would the exodus be bad, not how would backing down be bad]

Especially because the people most likely to quit would be the ones driving the risky behavior?

The big problem would seem to be that they might (probably would/will) go off and recreate the danger elsewhere, but that's probably not avoidable anyway. If you don't act, they'll continue to do it under your roof. If you force them to go set up elsewhere, then at least you've slowed them down a bit.

And you might even be able to use the optics of the whole mess to improve the "you can do whatever you want as long as you're big enough" regulatory framework that seems to have been being put into place, partly under OpenAI's own influence. Probably not, but at least you can cause policymakers to perceive chaos and dissent, and perhaps think twice about whether it's a good idea to give the chaotic organizations a lot of rope.

Granted this all rests on unsubstantiated rumors and hypotheticals, but in a scenario in which the board said "shut it down this is too risky", doesn't the response suggest we're doomed either way? Either

a) Investors have more say than the board and want money, so board resigns and SA is reinstated to pursue premiere AGI status

b) Board holds firm in decision to oust SA, but all his employees follow him to a new venture and investors follow suit and they're up and running with no more meaningful checks on their pursuit of godlike AI

After some recent (surprising) updates in favor of "oh maybe people are taking this more seriously than I expected and maybe there's hope", this ordeal leads me to update in the opposite direction of "we're in full speed ahead arms race to AGI and the only thing to stop it will be strong global government interventionist policy that is extremely unlikely". Not that the latter wasn't heavily weighted already, but this feels like the nail in the coffin.

Alternative framing: The board went after Altman with no public evidence of any wrongdoing. This appears to have backfired. If they had proof of significant malfeasance, and presented it to their employees, the story may have gone a lot differently. 

Applying this to the AGI analogy would be be a statement that you can't shut down an AGI without proof that it is faulty or malevolent in some way. I don't fully agree though: I think if a similar AGI design had previously done a mass murder, people would be more willing to hit the off switch early. 

In my view, this is an example of low-resolution thinking that occurs prior to us having direct experience in the actual mechanisms that would lead to actual danger. 

It's easy to imagine bad outcomes and thus activate associated emotions. Then we wish to (urgently) control those emotions through problem-solving.

However, our knowledge of the mechanism leading to the outcome is necessarily low resolution prior to the mechanism actually existing. So, we can only reason in terms of "switching it off", and other highly abstract concepts.

To address the argument: I don't think this is a case of the "AI winning" or an example of the "switch-off-ability" of AI. I think it just feels like it is.

As AI develops, I doubt this will ever be the useful question. I think there will be more boring questions that are hard to predict. In other words, we'll keep asking this question until an actual mechanism of danger exists, and we'll fairly easily know the right question at that point, and will solve it. 

I notice in myself the instinct to "play it safe", but I think this is downstream of a "fear of the unknown" emotional reflex.

With regard to the bigger question of ai safety, I think my argument here pushes me towards a lower risk assessment than I might have had before elucidating it. This is tangential though.

As a side note: perhaps this is why Sam has commented he thinks the most sensible approach to ai safety is to keep releasing stuff incrementally. This builds collective knowledge of actual mechanisms and thus progressively improves the debate into more and more useful territory. 

There's a whole part of the argument which is missing which is the framing of this as being about AI risk.
I've seen various propositions for why this happened, and the board being worried about AI risk is one of them but not the most plausible afaict.
 

In addition this is phrased similarly to technical problems like the corrigibility, which it is very much not about.
People who say "why can't you just turn it off" typically refer to literally turning off the AI if it appears to be dangerous, which this is not about. This is about turning off the AI company, not the AI.