Abstract The Orthogonality Thesis asserts that 'there can exist arbitrarily intelligent agents pursuing any kind of goal' (Yudkowsky et al. 2025). One of the arguments made in favour of the thesis is that intelligent agentic systems can devise how to best achieve a goal without computing anything about the relative...
This is my first LessWrong contribution. I have tried to base my contribution on the community guidelines and the principles of rationalism, but I very much welcome any and all feedback on how to improve! I focus on a recent article discussing AI and the Fermi Paradox and cite previous...
I would love to know why people downvoting disagree :)
Abstract
The Orthogonality Thesis asserts that 'there can exist arbitrarily intelligent agents pursuing any kind of goal' (Yudkowsky et al. 2025). One of the arguments made in favour of the thesis is that intelligent agentic systems can devise how to best achieve a goal without computing anything about the relative value (<V) or justification (<<W) of goals, i.e., without passing moral judgments. Here, I argue that <V and <<W are emergent properties of any goal-pursuing agent. However, this reasoning need only apply to the steps needed to fulfil the final goal, which can itself remain axiomatic. Hence, my reasoning only weakens the Orthogonality Thesis to the extent that the inevitable emergence of <V... (read 3240 more words →)
Logically, I agree. Intuitively, I feel suspect that it just won't happen. But, intuition on such alien things should not be a guide, so I fully support some attempt to slow down the takeoff.
Thanks for this!
TBH, I am struggling with the idea that an AI intent on maximising a thing doesn't have that thing as a goal. Whether or not the goal was intended seems irrelevant to whether or not the goal exists in the thought experiment.
"Goal stability is almost certainly attained in some sense given sufficient competence"
I am really not sure about this, actually. Flexible goals is a universal feature of successful thinking organisms. I would expect that natural selection would kick in at least over sufficient scales (light delay making co-ordination progressively harder on galactic scales), causing drift. But even on small scales, if an AI has, say, 1000 competing goals, I would... (read more)
I like the point here about how stability of goals might be an instrumentally convergent feature of superintelligence. It's an interesting point.
On the other hand, intuitive human reasoning would suggest that this is overly inflexible if you ever ask yourself 'could I ever come up with a better goal than this goal?'. What better would mean for a superintelligence seems hard to define, but it also seems hard to imagine that it would never ask the question.
Separately, your opening statements seem to be at least nearly synonymous to me:
"First off the paperclip maximizer isn't about how easy it is to give a hypothetical super intelligent a goal that you might regret later... (read more)
The worry that AI will have overly fixed goals (paperclip maximiser) seems to contradict the erstwhile mainline doom scenario from AI (misalignment). If AI is easy to lock into a specify path (paperclips) then it follows that locking in into alignment is also easy - provided you know what alignment looks like (which could be very hard). On the other hand, a more realistic scenario would seem to be that, in fact, keeping fixed goals for AI is hard, and that likely drift is where the misalignment risk really comes in big time?
I can agree that qualitatively there is a lot left to do. Quantitatively, though, I am still not quite seeing the smoking gun that human level AI will be able to smash through 15 OOM like this. But, happy to change my mind. I'll check out the anthropic link! Cheers.
I don't really fully understand the research speed up concept in intelligence explosion scenarios, e.g., https://situational-awareness.ai/from-agi-to-superintelligence/
This concept seems fundamental to the entire recursive self-improvement idea, but what if it turns out that you can't just do:
Faster research x many agents = loads of stuff
What if you quickly burn through everything that an agent can really tell you without doing new research in the lab? You'd then hit a brick wall of progress where throwing 1000000000 agents at 5x human speed amounts to little more than what you get out of 1 agent (being hyperbolic lol).
Presumably this is just me as a non-computer-scientist missing something big and implicit in how AI research is expected to proceed? But ultimately I guess this Q boils down to:
Is there actually 15 orders of magnitude of algorithmic progress to make (at all), and/or can that truly be made without doing something complimentary in the real world to get new data / design new types of computers, and so on?
Haha I have no idea! I agree the possibility space is huge. All I do know is that we don't see any evidence of alien AIs around us, so they are a poor explanation as a great filter for other alien races (unless they kill those races and then for some reason kill themselves, too / decide to be non-expansionist every single time).
Isn't there a bit of a false equivalence tucked up in the logic here? Two sides could be equally scared of one another and both feel like underdogs, but that says nothing about who is correct to think that way. Sometimes people just are the underdog. People unable to use democracy to enact change versus elites that consider them dangerous is a good example. The masses in that case are definitely the underdog, as they threaten the status quo of every major power centre (often state, corporations, politicians, and elite institutions all at once). In many European countries, certainly, it is unclear that the masses can do very much to policy at all right now. They feel like underdogs because they are. I am sure the elites also feel that they are underdogs... they're just wrong.
(This is a quick take - don't take it that seriously if I don't articulate other people's views accurately here)
Just listened to a few more episodes of Doom Debates. Something that stands out is that the predictions from the Liron-esque worldview have been consistently overweighted towards doom so far. So, Liron will say things along the lines of 'GPT3 could have been doom, we didn't know either way and got lucky'.
But, there was no luck at all in the empirical sense. It could never have been doom, we just didn't know that for sure. So, we took a risk, but it turns out that there was no actual risk.
Based on this, naively,... (read more)
This is my first LessWrong contribution. I have tried to base my contribution on the community guidelines and the principles of rationalism, but I very much welcome any and all feedback on how to improve! I focus on a recent article discussing AI and the Fermi Paradox and cite previous work on these topics where I am aware of it.
Summary [of ~2000 word body]
Where are all the machines?
The Fermi Paradox originally asked why civilisations like ours appear to be absent despite such a larger number of stars and attendant planets. The rise of Artificial Intelligence (AI) demands that we additionally explain why the inorganic creations of such civilisations are absent.
This problem is... (read 2639 more words →)
I would disagree fairly strongly: "lobbyists are absolutely dependent on democratic institutions to leverage their wealth into political power, while 50,000 angry people with pitchforks are not"
They are, I think. If they are angry that democracy is ignoring them then their pitchforks will likely not manage to enact some complicated change to legislation needed to fix the problem, as you point out. If we care about power to actually make a change about the things people want to happen, this is vested almost entirely within the hands of the elite and not within pitchforks. Pitchforks could maybe scare elites into doing it, but more likely it just generates chaos. Because pitchforks are... (read more)