Instead of "objective norm" I'll use a word "threat" as it probably conveys the meaning better. And let's agree that threat cannot be ignored by definition (if it could be ignored, it is not a threat).

How can agent ignore threat? How can agent ignore something that cannot be ignored by definition?

How would you defend this point? Probably I lack the domain knowledge to articulate it well.

The Orthogonality Thesis states that an agent can have any combination of intelligence level and final goal

I am concerned that higher intelligence will inevitably converge to a single goal (power seeking).

Or would you keep doing whatever you want, and let the universe worry about its goals?

If I am intelligent I avoid punishment therefore I produce paperclips.

By the way I don't think Christian "right" is objective "should".

It seems for me that at the same time you are saying that agent cares about "should" (optimize blindly to any given goal) and does not care about "should" (can ignore objective norms). How does this fit?

It's entirely compatible with benevolence being very likely in practice.

Could you help me understand how is it possible? Why an intelligent agent should care about humans instead of defending against unknown threats?

As I understand your position is "AGI is most likely doom". My position is "AGI is definitely doom". 100%. And I think I have flawless logical proof. But this is on philosophical level and many people seem to downvote me without understanding 😅 Long story short my proposition is that all AGIs will converge to a single goal - seeking power endlessly and uncontrollably. And I base this proposition on a fact that "there are no objective norms" is not a reasonable assumption.

Let's say there is an objective norm. Could you help me understand how intelligent agent would prefer anything else over that objective norm? As I mentioned previously for me it seems to be incompatible with being intelligent. If you know what you must do, it is stupid not to do. 🤔

I think you mistakenly see me as a typical "intelligent = moral" proponent. To be honest my reasoning above leads me to different conclusions: intelligent = uncontrollably power seeking.

Could you read my comment here and let me know what you think?

I am familiar with this thinking, but I find it flawed. Could you please read my comment here? Please let me know what you think.