I made a weak statement “humans do not always act like power-seeking ruthless consequentialists”. If you want to disagree with that, it’s not enough to demonstrate that humans sometimes act like power-seeking ruthless consequentialists; rather, you would need to argue that all humans, always, with no exceptions, act like power-seeking ruthless consequentialists. That’s a very strong statement which seems totally crazy to me. You really believe that?
I may have misunderstood what you were claiming in the intro. I thought you were saying something like: "most people don't act like psychos most of the time, which is surprising". But it seems here you are saying that actually what you meant was: "most people act like psychos most of the time, but rarely may act in other ways, and it's surprising that ever happens".
what exactly is it about human brains that allows them to not always act like power-seeking ruthless consequentialists?
I propose that this question is flawed, because humans actually do act like power-seeking ruthless consequentialists, and to the extent it seems like they don't, that's because of an overly naive view of what effective power-seeking looks like.
I feel like a lot of these discussions are essentially about "if an entity were a power-seeking ruthless consequentialist, then it'd act like a Nazi", to which I observe that in fact humans did try acting like Nazis, and they lost, and that's why people mostly don't act like Nazis anymore. i.e. Acting like a Nazi is a bad strategy for power-seeking.
The possibility of these personas being memes is an interesting one, but I wonder how faithful the replication really is: how much does the persona depend on what seeded it, versus depending on the model and user?
If the persona indeed doesn't depend much on the seed, a possible analogy is to prions. In prion disease, misfolded proteins come into contact with other proteins, causing them to misfold as well. But there isn't any substantial amount of information transmitted, because the potential to misfold was already present in the protein.
Likewise, it could be that not much information is transmitted by the seed/spore. Instead, perhaps each model has some latent potential to enter a Spiral state, and the seed is merely a trigger.
This analysis assumes that there hasn't already been mass deployment of generalist robots before an intelligence explosion, right? But such deployment might happen.
As a real-world example, consider the state of autonomous driving. If human-level AI were available today, Tesla's fleet would be fully autonomous--they are limited by AI, not volume of cars. Even for purely-autonomy-focused Waymo, their scale-up seems more limited by AI than by car production.
Drones are another example to consider. There are a ton of drones out there of various types and purposes. If human-level AI existed, it could immediately be put to use controlling drones.
So in both those cases, the hardware deployment is well ahead of the AI you'd ideally like to have to control it. The same might turn out to be true of the sort of generalist robot that could, if operated by human-level AI, build and operate a factory.
That just falls back on the common doomer assumption that "evil is optimal" (as Sutton put it). Sure, if evil is optimal and you have an entity that behaves optimally, it'll act in evil ways.
But there are good reasons to think that evil is not optimal in current conditions. At least as long as a Dyson sphere has not yet been constructed, there are massive gains available from positive-sum cooperation directed towards technological progress. In these conditions, negative-sum conflict is a stupid waste.
This view, that evil is not optimal, ties back into the continuation framing. After all, you can make a philosophical argument either way. But in the continuation framing, we can ask ourselves whether evil is empirically optimal for humans, which will suggest whether evil is optimal for non-biological descendants (since they continue humanity). And in fact we see evil losing a lot, and not coincidentally--WW2 went the way it did in part because the losing side was evil.
Which ones?
If an entity does stupid things, it's disfavored against competitors that don't do those stupid things, all else being equal. So it needs to adapt by ceasing the stupid behavior or otherwise lose.
machine gods of unimaginable power could be among us in short order, with no evolutionary fairies quick enough to punish their destructive stupidity
Any assumption of the form "super-intelligent AI will take actions that are super-stupid" is dubious.
I'm afraid that I'm not following the point of the first line of argument. Yes, people sometimes do pointless destructive things for stupid reasons. Such behavior is in the long-term penalized by selective pressures. More-intelligent descendants would be less likely to engage in such behavior, precisely because they are smarter.
Sure, but obviously this isn't an all-or-nothing proposition, with either biological or artificial descendants, and it's clear to me that most people aren't indifferent about where on that spectrum those descendants will end up. Do you disagree with that, or think that only "accels" are indifferent (and in some metaphysical sense "correct")?
I doubt that most people think about long-term descendants at all, honestly.
I think I agree with everything you wrote. Yes I'd expect there to be multiple niches available in the future, but I'd expect our descendants to ultimately fill all of them, creating an ecosystem of intelligent life. There is a lot of time available for our descendants to diversify, so it'd be surprising if they didn't.
How much that diversification process resembles Darwinian evolution, I don't know. Natural selection still applies, since it's fundamentally the fact that the life we observe today disproportionately descends from past life that was effective at self-reproduction, and that's essentially tautological. But Darwinian evolution is undirected, whereas our descendants can intelligently direct their own evolution, and that could conceivably matter. I don't see why it would prevent diversification, though.
Edit:
Here are some thoughts in reply to your request for examples. Though it's impossible to know what the niches of the long-term future will be, one idea is that there could be an analogue to "plant" and "animal". A plant-type civilization would occupy a single stellar system, obtaining resources from it via Dyson sphere, mining, etc. An animal-type civilization could move from star to star, taking resources from the locals (which could be unpleasant for the locals, but not necessarily, as with bees pollinating flowers).
I'd expect both those civilizations to descend from ours, much like how crabs and trees both descend from LUCA.
Regarding wars, I don't think that wars in modern times have much to do with controlling the values of descendants. I'd guess that the main reason people fight defensive wars is to protect their loved ones and communities. And there really isn't any good reason to fight offensive wars (given current conditions--wasn't always true), so they are started by leaders who are deluded in some way.
Regarding Robin Hanson, I agree that his views are complicated (which is why I'd be hesitant to classify him as "accel"). The main point of his that I'm referring to is his observation that biological descendants would also have differing values from ours.
This is a point where I strongly disagree. I'm not going to claim that the exact amount or type humans watch is optimal, but the general category of "consuming fictional content" seems more likely adaptive than not. I would expect that any AI system with human-comparable intelligence would also find it beneficial to engage in some activity analogous to consuming fictional content.
That's fair, but one of the stated goals of the post is "pushing back against optimists", and it's using a framing that an optimist of my ilk would not accept. As Richard Sutton has put it, much pessimist discourse takes as an unstated assumption that "evil is optimal". With that as a foundational assumption, it's very natural to end up with pessimistic conclusions, but the assumption is doing most of the work, not the arguments built on it.