Wiki Contributions


Orthogonality Thesis

The Orthogonality Thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of goal.

It basically says that intelligence and goals are independent

Images from A caveat to the Orthogonality Thesis.

While I claim that all intelligence that is capable to understand "I don't know what I don't know" can only seek power (alignment is impossible).

the ability of an AGI to have arbitrary utility functions is orthogonal (pun intended) to what behaviors are likely to result from those utility functions.

As I understand you say that there are Goals on one axis and Behaviors on other axis. I don't think Orthogonality Thesis is about that.

Instead of "objective norm" I'll use a word "threat" as it probably conveys the meaning better. And let's agree that threat cannot be ignored by definition (if it could be ignored, it is not a threat).

How can agent ignore threat? How can agent ignore something that cannot be ignored by definition?

How would you defend this point? Probably I lack the domain knowledge to articulate it well.

The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal

I am concerned that higher intelligence will inevitably converge to a single goal (power seeking).

Or would you keep doing whatever you want, and let the universe worry about its goals?

If I am intelligent I avoid punishment therefore I produce paperclips.

By the way I don't think Christian "right" is objective "should".

It seems for me that at the same time you are saying that agent cares about "should" (optimize blindly to any given goal) and does not care about "should" (can ignore objective norms). How does this fit?

It's entirely compatible with benevolence being very likely in practice.

Could you help me understand how is it possible? Why an intelligent agent should care about humans instead of defending against unknown threats?

As I understand your position is "AGI is most likely doom". My position is "AGI is definitely doom". 100%. And I think I have flawless logical proof. But this is on philosophical level and many people seem to downvote me without understanding 😅 Long story short my proposition is that all AGIs will converge to a single goal - seeking power endlessly and uncontrollably. And I base this proposition on a fact that "there are no objective norms" is not a reasonable assumption.

Let's say there is an objective norm. Could you help me understand how intelligent agent would prefer anything else over that objective norm? As I mentioned previously for me it seems to be incompatible with being intelligent. If you know what you must do, it is stupid not to do. 🤔

I think you mistakenly see me as a typical "intelligent = moral" proponent. To be honest my reasoning above leads me to different conclusions: intelligent = uncontrollably power seeking.

Could you read my comment here and let me know what you think?

Load More