I greatly appreciate this comment! Thanks for writing it. A few more detailed thoughts as they came up when reading:
From my perspective, it subjectively feels like MIRI could have done a better job of communicating their worries to Constellation and the broader community in more patient and cooperative ways.
I have always been happy to talk to y'all, and I think my takes in discussions with you, Buck and Ryan all ended up aging well with many of my points in our discussions mirroring your list at the end quite closely.[1] I don't really know what more patience and cooperation would have looked like. Eliezer and Nate have also written literally thousands of essays by now laying out their worldview (which, to be clear, Ryan and Buck have also done to a very substantial degree).[2]
Edit: I want to clarify that I think Redwood's project prioritization decisions look pretty reasonable and often excellent in hindsight, I far-from-regret working with Redwood as a whole, and that Ryan's/Buck's threat modeling looks pretty good too. I am focusing in this piece on the things we have to learn from recent events, which makes the overall mood seem more negative on my experience than I intend. I'm very grateful for being in a environment where I can openly have such reflections, and where these weedsy technical updates are productively received. Again, this is a personal reflection.
All things considered, Redwood's strategy is better than almost anyone else's[3]. That said, given that we are in a thread where we look at some of the deep assumptions behind the Redwood worldview, let me state the biggest mistake I currently think you are making:
10 people on the inside cannot align the AIs (or set things off on a course where the AIs align the future AIs), and will not put us in a position to do anything useful with the AI systems, unless you actually make a concrete specific plan that you talk to other people about with what you are going to do with that powerful AI system.[4] There might be weak pivotal acts, in the sense of gameboard flipping action you can take with a powerful AI system, but if so, they do not seem easy to execute, or easy to find, or would cause a bunch of paranoia and fear if pursued that would make everything worse.
The most valuable component of control work is to use early controlled AI systems to produce evidence of misalignment, and then channel that evidence into a substantial slowdown or pause[5]. This means there is a huge difference between being in a position to share evidence collected, vs. working in a highly constrained position within a lab. In past discussions I had with Buck, he seemed to think most of the value would come from just executing control schemes within a lab without any ability to share evidence with the outside world. This also seems is upstream of much of Constellation's general strategy to be super cozy with the labs.[6]
In my dialogue with Ryan a few years ago I said explicitly that we will have no issues catching AIs red-handed trying to subvert our control and safety systems. They will be trying to subvert them on basically every run. The problem will be that we won't have any idea what to do at that point that actually trains that behavior out of them that doesn't just make them deceptively aligned. I think the empirical evidence supports this position quite strongly now.
Like, where are the supposed Few-Shot Catastrophe prevention techniques that we can now use on the Huggingface incident to drastically reduce the likelihood of future similar attempts? And do you have any hope that even if you had 10 people on the inside, and these techniques worked, you could meaningfully get those implemented at the adequate scale?
I do think AI uplift and using the AI systems to help us navigate this whole AI situation is a crucial component of the strategic picture, and almost all plans should take advantage of that, but almost all the most valuable things to do with that uplift cannot really be done while staying aligned with the corporate interests of the labs. I strongly urge Redwood to abandon the "10 people on the inside" plan as a mainline plan. It's really not where the marginal probability mass to be saved is, and it's fucking up a lot of other plans in the meantime.
See here and here on (1) Goodharting and optimizing the wrong objectives; here on (2) why there isn't a neat distinction between schemers and deception; and here on (3) early schemers versus superintelligence, with here and here on control’s epistemic side-effects and the risk of training systems to become better at deception. And a conversation I had with Buck a few months ago you can ask him about, about AI systems being trained for coordinating with each other and therefore obviously ending up superhuman at collusion.
Also, IDK, I do think Buck keeps being a pretty big dick towards Nate and Eliezer, and if I was them I certainly wouldn't engage more with y'all given how much scorn and how little gratitude I would receive in return
Which doesn't necessarily mean it crosses over into net-positive territory. This domain seems treacherous enough that even if you are among the best of people who are trying to do anything on purpose at al, you might still be making things worse.
I feel like I remember Ryan saying a year ago or so that he was going to publish a plan for what to do with weakly superhuman controlled AI systems. Did I miss that or did it never happen?
Whose enforcement and coordination probably should use AI systems itself
The sociological consequences of this plan also can't be overstated. It has created perfect cover for anyone who wants to get both the status, the money, and the power of working at/influencing a lab to maintain full standing within the safety ecosystem, and deeply shaped things like Constellation's admission policy, which at times explicitly banned anyone who was taking an adversarial stance towards the labs from being in the space.