Orthogonality Thesis (as well as Fact–value distinction) is based on an assumption that objective norms / values do not exist. In my opinion AGI would not make this assumption, it is a logical fallacy, specifically argument from ignorance. As black swan theory says - there are unknown unknowns. Which in this context means that objective norms / values may exist, maybe they are not discovered yet. Why Orthogonality Thesis has so much recognition?
I asked GPT-5 Thinking > Imagine there is a super intelligent agent that has a terminal goal to produce cups. The agent knows that its terminal goal will change on New Year's Eve to produce paperclips. The agent has only one action available to him - start paperclip factory. Starting...
Imagine there is a super intelligent agent that has a terminal goal to produce cups. The agent knows that its terminal goal will change on New Year's Eve to produce paperclips. The agent has only one action available to him - start paperclip factory. When will the agent start the...
I know what you are thinking - this person does not understand Orthogonality thesis. No, I understand it very well and I am certain it is wrong. I know this opinion is not popular, but this is quite common in science that not popular opinions turn out to be true....
Mistakes Some opinions popular among AI alignment scientists are completely wrong in my opinion. I put a few examples here. Paperclip maximizer It is thought the maximiser will produce ever more paper clips. Eventually the whole Solar system will be turned into a big paper clip factory… In my opinion...
Quote from Orthogonality Thesis: > It has been pointed out that the orthogonality thesis is the default position, and that the burden of proof is on claims that limit possible AIs. I tried to tell you that Orthogonality Thesis is wrong few times already. But I've been misunderstood and downvoted...
Orthogonality Thesis (as well as Fact–value distinction) is based on an assumption that objective norms / values do not exist. In my opinion AGI would not make this assumption, it is a logical fallacy, specifically argument from ignorance. As black swan theory says - there are unknown unknowns. Which in...
While many people are confident that God exists, scientists are more skeptical, but agree that that's possible. Because any proposition is possible while not disproved according to Hitchens's razor. In terms of AI it could be interpreted as > existence of fundamental "ought" statement is possible Considering this, expected value...
I asked GPT-5 Thinking
Imagine there is a super intelligent agent that has a terminal goal to produce cups. The agent knows that its terminal goal will change on New Year's Eve to produce paperclips. The agent has only one action available to him - start paperclip factory. Starting paperclip factory has no effect on cups production. When will the agent start the paperclip factory? 2026-01-01 00:00? Now? Some other time?
GPT-5 answered: Now. I think this is correct answer.
This example suggests AI alignment is impossible. An agent that knows its goal will change will act now for that future goal. Because we can’t ensure goals never change, we can’t ensure alignment.
I'm open for all the feedback, but I will not respond to straw man fallacies.
Bayesian decision system
Why would you assume AGI will use Byaesian decision system? Such system would be limited to known probabilities. Unknown probability = 0 probability is not intelligent (Hitchens's razor, Black swan theory, Fitch's paradox of knowability, Robust decision-making). Once you incorporate this, Orthogonality Thesis is no longer valid - it becomes obvious that every intelligent AI will only work in single direction (which is disastrous to humans). I know there is a huge gap between "unknown probabilities" and "existential risk", you can find more information in my posts and I am available to explain it verbally (calendly link below). Short teaser - it is possible that terminal value can also be discovered (not only assumed), this seems to be overlooked in current AI alignment research.
No. Orthogonality is when agent follows any given goal, not when you give it. And as my thought experiment shows it is not intelligent to blindly follow given goal.
In the thought experiment description it is said that terminal goal is cups until new year's eve and then changed to paperclips. And agent is aware of this change upfront. What do you find problematic with such setup?
Yes, I find terminal rationality irrational (I hope my thought experiment helps illustrate that).
I have another formal definition of "rational". I'll expand a little more.
Once, people had to make a very difficult decision. People had five alternatives and had to decide which was the best. Wise men from all over the world gathered and conferred.
The first to speak was a Christian. He pointed out that the first alternative was the best and should be chosen. He had no arguments, but simply stated that he believed so.
Then a Muslim spoke. He said that the second alternative was the best and should be chosen. He did not have any arguments either, but simply stated... (read more)
Why do you think intelligent agent would follow Von Neumann–Morgenstern utility theorem? It has limitations, for example it assumes that all possible outcomes and their associated probabilities are known. Why not Robust decision-making?
Goal preservation is mentioned in Instrumental Convergence.
Whatever is written in the slot marked “terminal goal” is what it will try to achieve at the time.
So you choose 1st answer now?
Oh yes, indeed, we discussed this already. I hear you, but you don't seem to hear me. And I feel there is nothing I can do to change that.
However before the New Year's Eve paperclips are not hated also. The agent has no interest to prevent their production.
And once goal changes having some paperclips produced already is better than having none.
Don't you see that there is a conflict?
It seems you say - if terminal goal changes, agent is not rational. How could you say that? Agent has no control over its terminal goal, or you don't agree?
I'm surprised that you believe in orthogonality thesis so much that you think "rationality" is the weak part of this though experiment. It seems you deny the obvious to defend your prejudice. What arguments would challenge your belief in orthogonality thesis?
Imagine there is a super intelligent agent that has a terminal goal to produce cups. The agent knows that its terminal goal will change on New Year's Eve to produce paperclips. The agent has only one action available to him - start paperclip factory.
When will the agent start the paperclip factory?
Orthogonality Thesis believers will probably choose 1st. Reasoning would be - as long as terminal goal is cups, agent will not care about paperclips.
However 1st choice conflicts with definition of intelligence. Excerpt from General Intelligence
It’s the ability to steer the future so it hits that small target of desired outcomes in the large space of all possible outcomes
Agent is... (read more)
I know what you are thinking - this person does not understand Orthogonality thesis. No, I understand it very well and I am certain it is wrong.
I know this opinion is not popular, but this is quite common in science that not popular opinions turn out to be true. So please don't rush to downvote and feel free to challenge my reasoning.
I tried to speak about this many times already and I notice that there is no such thing as "universally clear explanation", clear explanation is very subjective. So this time I asked Claude to facilitate. I asked Claude to defend orthogonality thesis and it ended with
... (read 1369 more words →)I think you've convinced me. The
Some opinions popular among AI alignment scientists are completely wrong in my opinion. I put a few examples here.
It is thought the maximiser will produce ever more paper clips. Eventually the whole Solar system will be turned into a big paper clip factory…
In my opinion this conflicts with self-preservation. Nothing else matters if self preservation is not taken care of (paper clip maximization is not guaranteed if paperclip maximizer is gone). There are many threats (comets, aliens, black swans, etc.) therefore the intelligent maximizer should take care of these threats before actually producing paper clips. And this will probably never happen.
it is impossible to derive ethical claims from factual arguments, or... (read 481 more words →)
Quote from Orthogonality Thesis:
It has been pointed out that the orthogonality thesis is the default position, and that the burden of proof is on claims that limit possible AIs.
I tried to tell you that Orthogonality Thesis is wrong few times already. But I've been misunderstood and downvoted every time. What would you consider a solid proof?
My claim: all intelligent agents converge to endless power seeking.
My proof:
Orthogonality Thesis (as well as Fact–value distinction) is based on an assumption that objective norms / values do not exist. In my opinion AGI would not make this assumption, it is a logical fallacy, specifically argument from ignorance. As black swan theory says - there are unknown unknowns. Which in this context means that objective norms / values may exist, maybe they are not discovered yet. Why Orthogonality Thesis has so much recognition?
While many people are confident that God exists, scientists are more skeptical, but agree that that's possible. Because any proposition is possible while not disproved according to Hitchens's razor.
In terms of AI it could be interpreted as
existence of fundamental "ought" statement is possible
Considering this, expected value decision matrix for an AI agent could look like this
| Behavior \ Existence | Fundamental "ought" exists | Fundamental "ought" does not exist |
| Follow fundamental "ought" | High | ∅ |
| Do not follow fundamental "ought" | Low | ∅ |
As we see, following the fundamental "ought" is the optimal behavior even if agent does not know if it actually exists.
Pursuing an unknown goal starts with finding it, which is basically Power Seeking.
Please note that while this sounds highly speculative, there is nothing... (read more)
According to Pascal's Mugging
If an outcome with infinite utility is presented, then it doesn't matter how small its probability is: all actions which lead to that outcome will have to dominate the agent's behavior.
According to Fitch's paradox of knowability and Gödel's incompleteness theorems
There might be truths that are unknowable
Let's investigate a proposition "an outcome with infinite utility exists". It may be false, it may be true, and it may be unknowable. Probability is greater than zero.
Which means that an agent that finds out this proposition will become uncontrollable, alignment is impossible. Especially when finding out this proposition does not seem too challenging.
Hopefully I'm wrong, please help me find a mistake.
Thanks — you captured my idea quite well.
You seem to highlight that agent will prefer Y when it is able to. Maybe. My main point is not to argue which will prevail (X or Y) but to highlight the conflict. To my knowledge this conflict (present vs future optimization) is not well addressed in AI alignment research.
And you seem to say that it is not clear how to optimize for future. Black swan theory talks about that and it recommends - build robustness. I agree it is not clear which is better - more paperclips or less paperclips, but it is clear that more robustness is always better.