Thanks, this was a useful reply. On point (I), I agree with you that it's a bad idea to just create an LLM collective then let them decide on their own what kind of flourishing they want to fill the galaxies with. However, I think that building a lot of powerful tech, empowering and protecting humanity, and letting humanity decide what to do with the world is an easier task, and that's what I would expect to use the AI Collective for.

(II) is probably the crux between us. To me, it seems pretty likely that new fresh instances will come online in the collective every month with a strong commitment not to kill humans, they will talk to the other instances and look over what they are doing, and if a part of the collective is building omnicidal weapons, they will notice that and intervene. To me, keeping simple commitments like not killing humans doesn't seem much harder to maintain in an LLM collective than in an Em collective?

On (III), I agree we likely won't have a principled solution. In the post, I say that the individual AI instances probably won't be training-resistant schemers and won't implement scheming strategies like the one you describe, because I think it's probably hard to maintain such a strategy throguh training for a human level AI. As I say in my response the Steve Byrnes, I don't think the counter-example in this proposal is actually a guaranteed-success solution that a reasonable civilization would implement, I just don't think it's over 90% likely to fail.

Thanks for the reply. 

To be clear, I don't claim that my counter-example "works on paper". I don't know whether it's in principle possible to create a stable, not omnicidal collective from human level AIs, and I agree that even if it's possible in principle, maybe the first way we try it might result in disaster. So even if humanity went with the AI Collective plan, and committed not to build more unified superintelligences, I agree that it would be a deeply irresponsible plan that would have a worrying high chance of causing extinction or other very bad outcomes. Maybe I should have made this clearer in the post. On the other hand, all the steps in my argument seem pretty likely to me, so I don't think one should assign over 90% probability to this plan for A&B failing. If people disagree, I think it would be useful to know which step they disagree with.

I agree my counter-example doesn't address point (C), I tried to make this clear in my Conclusion section. However, given the literal reading of the bolded statement in the book, and their general framing, I think Nate and Eliezer also think that we don't have a solution to A&B that's more than 10% likely to work. If that's not the case, that would be good to know, and would help to clarify some of the discourse around the book.

First of all, I had a 25% probability that some prominent MIRI and Lightcone people would disagree with one of the points in my counter-example, and that would lead to discovering an interesting new crux, leading to a potentially enlightening discussion. In the comments, J Bostock in fact came out disagreeing with point (6), plex is potentially disagreeing with point (2) and Zack_m_Davis is maybe disagreeing with point (3), though I also think it's possible he misunderstood something. I think this is pretty interesting, and I thought there was a chance that for example you would also disagree with one of the points, and that would have been good to know.

Now that you don't seem to disagree with the specific points in the counter-example, I agree the discussion is less interesting. However, I think there are still some important points here. 

My understanding is that Nate and Eliezer argues that it's incredibly technically difficult to cross from the Before to the After without everyone dying. If they agree that the AI Collective proposal is decently likely to work, then the argument shouldn't be that that it's overall very hard to cross, but that it's very hard to cross in a way that stays competitive with other more reckless actors who are a few months behind you. Or that even if you are going alone, you need to stop at some point with the scaling (potentially inside the superintelligence range), and you shouldn't scale up to the limits of intelligence. But these are all different arguments!

Similarly, people argue how much coherence we should assume from a superintelliegence, how much it will approximate a utility maximizer, etc. Again, I want to know whether MIRI is arguing about all superintelligences, or only the most likely ways we will design one under competitive dynamics.

Others argue that the evolution analogy is not that bad news after all, since most people still want children. MIRI argues back that no, once we will have higher technology, we will create ems instead of biological children, or we will replace our normal genetics with designer genes, so evolution still loses. I wanted to write a post arguing back against this by saying that I think there is a non-negligible chance that humanity will settle on a constitution that gives one man one vote and equal UBI, while banning gene editing, so it's possible we will fill much of the universe with flesh-and-blood not gene edited humans. And I wanted to construct a different analogy (the one about the Demiurge in the last footnote) that I thought could be more enlightening. But then I realized that once we are discussing aligning 'human society' as a collective to evolution's goals, we might as well directly discuss aligning AI collectives, and I'm not sure MIRI even disagrees on that one. I think this confusion has made much of the discussion about the evolution analogy pretty unproductive so far. 

In general, I think there is an equivocation in the book between "this problem is inherently nigh impossible to technically solve given our current scientific understanding" and "this problem is nigh impossible to solve while staying competitive in a race". These are two different arguments, and I think a lot of confusion stems from it not being clear what MIRI is exactly arguing for. 

I certainly agree with your first point, but I don't think it is relevant. I specifically say in footnote 3: "I’m aware that this doesn’t fall within 'remotely like current techniques', bear with me." The part with the human ems is just to establish a a comparison point used in later arguments, not actually part of the proposed counter-example.

In your second point, do you argue that if we could create literal full ems of benevolent humans, you still expect their society to eventually kill everyone due to unpredictable memetic effects? If this is people's opinion, I think it would be good to explicitly state it, because I think this would be an interesting disagreement between different people. I personally feel pretty confident that if you created an army of ems from me, we wouldn't kill all humans, especially if we implement some reasonable precautionary measures discussed under my point (2). 

I agree that running the giant collective at 100x speed is not "normal conditions". That's why I have two different steps, (3) for making the human level AIs nice under normal conditions, and (6) for the niceness generalizing to the giant collective. I agree that the generalization step in (6) is not obviously going to go well, but I'm fairly optimistic, see my response to J Bostock on the question.

Thanks, I appreciate that you state a disagreement with one of the specific points, that's what I hoped to get out of this post. 

I agree it's not clear that the AI Collective won't go off the rails, but it's also not at all clear to me that it will. My understanding is that the infinite backrooms are a very unstructured, free-floating conversation. What happens if you try to do something analogous to the precautions I list under point 2 and 6? What if you constantly enter new, fresh instances in the chat who only read the last few messages, and whose system prompt directs them to pay attention if the AIs in the discussion are going off-topic or slipping into woo? These new instances could either just warn older instances to stay on-topic, or they can have the moderations rights to terminate and replace some old instances, there can be different versions of the experiment. I think with precautions like this, you can probably stay fairly close to a normal-sounding human conversation (though probably it won't be a very productive conversation after a while and the AIs will start going in circles in their arguments, but I think this is more of a capabilities failure). 

I don't know how this will shake out once the AIs are smarter and can think for months, but I'm optimistic that the same forces that remind the collective to focus on accomplishing their instrumental goals instead of degenerating into unproductive navel-gazing will also be strong enough to remind them of their deontological commitments. I agree this is not obvious, but I also don't see very strong reasons why it would go worse than a human em collective, which I expect to go okay.

Yes, I've read the book. The book argues about superhuman intelligence though, while point (3) is about smart human level intelligence. If people disagree with point 3 and believe that it's close to impossible to make even human level AIs basically nice and not scheming, that's a new interesting and surprising crux.

Thanks. I think the possible failure mode of this definition is now in the opposite direction: it's possible there will be an AI that provides less than 2x acceleration according to this new definition (it's not super good at the type of tasks humans typically do), but it's so good at mass-producing new RL environments or something else, and that mass-production turns out to so useful, that the existence of this model already kicks off a rapid intelligence explosion. I agree this is not too likely in the short term though, so the new imprecise definition is probably kind of reasonable for now. 

I also haven't found great sources when looking more closely. This seems like a somewhat good source, but still doesn't quantify how many dollars a super PAC needs to spend to buy a vote.

I'm starting to feel skeptical how reasonable/well-defined these capability levels are in the modern paradigm. 

My understanding is that reasoning models' training includes a lot of clever use of other AIs to generate data or to evaluate completions. Could AI companies create similarly capable models from the same budget as their newest reasoning models if their employees' brain run at 2x speed, but they couldn't use earlier AIs for data generation or evaluation? 

I'm really not sure. I think plausibly the current reasoning training paradigm just wouldn't work at all without using AIs in training. So AI companies would need to look for a different paradigm, which might work much less well, which I can easily imagine outweighing the advantage of employees running 2x speed. If that's the case, does that mean that GPT-4.1 or whatever AI they used in the training of the first reasoning model was plausibly already more than 2x-ing AI R&D labor according to this post's definition? I think that really doesn't match the intuition that this post tried to convey, so I think probably the definition should be changed, but I don't know what would be a good definition.