@one reason (not from Mitchell) for questioning the validity of many works on x-risk in AIs?

Ilio: Intelligence is not restricted to agents aiming at solving problems (https://www.wired.com/2010/01/slime-mold-grows-network-just-like-tokyo-rail-system/) and it’s not even clear that’s the correct conceptualisation for our own minds (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/).

Thanks for that. However, my definition of "intelligence" would be "the ability to find solutions for complex decision problems". It's unclear whether the ability of slime molds to find the shortest path through a maze or organize in seemingly "intelligent" ways has anything to do with intelligence, although the underlying principles may be similar.

We don’t want to argue over définition so I can change mine at least for this discussion. However, I want to point that, by your own definition, slime molds are very much intelligent, in that you can take any instance of an NP complete problem, reduce that to an instance of the problem of finding the most efficient roads for connecting corn flakes dots, and it will probably approximately solve it. Would you say your pocket calculator doesn’t compute because it has no idea it’s computing?

I haven't read the article you linked in full, but at first glance, it seems to refer to consciousness, not intelligence.

Depends on you. Say I ask a person with split-brain if they’d vote blue or red. Is that intelligence or isthat consciousness? In any case, the point is: sometime I can interrogate the left or right hemisphere specifically, and sometime they disagree. So, we don’t have a unique structure that can act as cartesian theater, which means our perception that our thoughts come from a single agent is, well, a perception. Does that invalidate any paper that starts from this single mind assumption? Not necessarily -models are always wrong, sometime useful despite wrong. But each time I read one that seems to break something important in the analysis.

Maybe that is a key to understanding the difference in thinking between me, Melanie Mitchell, and possibly you: If she assumes that for AI to present an x-risk, it has to be conscious in the way we humans are, that would explain Mitchell's low estimate for achieving this anytime soon. However, I don't believe that.

Maybe for Mitchell, I don’t know her personally. But no, that’s the opposite for me: I suspect building a conscious AI might be the easiest way to keep it as interpretable as a human mind, and I suspect most agents with random values would be constantly crashing, like LLMs you don’t keep in check. I fear civil war from minds polarization much more (for x-risks from AI).

To become uncontrollable and develop instrumental goals, an advanced AI would probably need what Joseph Carlsmith calls "strategic awareness" - a world model that includes the AI itself as a part of its plan to achieve its goals.

That’s one of the thing I tend to perceive as magic thinking, even while knowing the poster would likely say it’s a flawed perception. Let’s discuss that @magic.

@the more intelligent human civilization is becoming, the gentler we are

I wish that were so. We have invented some mechanisms to keep power-seeking and deception in check, so we can live together in large cities, but this carries only so far. What I currently see is a global deterioration of democratic values. In terms of the "gentleness" of the human species, I can't see much progress since the days of Buddha, Socrates, and Jesus. The number of violent conflicts may have decreased, but their scale and brutality have only grown worse. The way we treat animals in today's factory farms certainly doesn't speak for general human gentleness.

Here I think we might be in the kind of perception that contribute to self-definition. I have to admit Í’m both strongly optimistic (as in: it seems obvious we’re making progress) and totally incapable of proving that the measure you’re choosing are not better than the measure I’d be choosing. Anecdotally I would point that the west seem to get past colonisation and nationalist wars and killing first nations, whereas by Jesus time it would have been kind of expected of military chiefs to genocide a bit before enslaving the rest. I agree the road seems bumpy at time, but not with the stronger bits. How would you decide what set of priors was best for this question?

@convergence

Evolution is all about instrumental convergence IMO. The "goal" of evolution, or rather the driving force behind it, is reproduction. This leads to all kinds of instrumental goals, like developing methods for food acquisition, attack and defense, impressing the opposite sex, etc. "A chicken is an egg's way of making another egg", as Samuel Butler put it.

Yes, yes, and very bad wording from me! Yes me too crabs or mastication seem like attractors. Yes eggs are incredibly powerfull technology. What I meant was: when I see someone says « instrumental convergence », most of the time I (mis?)percieve: « Just because we can imagine a superintelligent agent acting like a monkey, we are utterly and fully convinced that it must act like a monkey, because acting like a monkey is a universal attractor that anything intelligent enough tend to, even though, for some reason, ants must have forgot it was the direction. And slim Molts. And plants. And dinosaure. And birds. And every human administration. Like they were more sensitive to local gradient than the promises of a long term computational power.

@magic:

of course nobody says « Hey my rational thoughts somewhat depend on confusing intelligence with magic ». However, I still perceive some tension in the semantic landscape that activates my perception of « confusing intelligence with magic » in a few key sentences, and this perception is strong enough that I can’t even guess what kind of exemple is best to evoke the same qualia in your mind. Let’s test that if you don’t mind: what would you say is magic thinking below?

+Superintelligence means it can beat you at tic-tac-toe. +Superintelligence means it can guess the password you got when you clicked on « generate a password ». +Superintelligence means hard takeoff, within years/month/weeks/days/hour/minute/second. +Superintelligence means hard takeoff + jumping to random values then kill us all. +Superintelligence just means a team of the best human experts plus speed. +Superintelligence means you behave like a chimps-like monkey hiding it’s intention to dominate every competitors around.

@anyhelper: damn it takes me time doing stupid things like formatting. If some young or old amanuensis with free time has enough pitty to help me have my thoughts out, preferably in gpt-enhanced mode, shoot a private message and we’ll set a zoom or something.