Mateusz Mazurkiewicz

The orthogonality thesis separates intelligence and goals, constraining the notion of intelligence to instrumental rationality and allowing for any combination of general intelligence and a goal system. And the other way around, such a description should apply to any possible agency. For any agency, it should be possible to divide it into two parts losslessly. Many arguments against the thesis have been constructed and proven incoherent, showing its robustness and broad applicability. Still, there also exist valid critiques of its assumptions, such as one by John Danaher. The following text discusses the human mind in the context of the orthogonality thesis and touches on the evolutionary origin of the human brain. Let us start with an assertion: 'some intelligences and goal systems are irreducibly connected.' This protest appears intuitive since it comes from dissonance and confusion experienced when one attempts to apply the orthogonality thesis to analyze oneself. Self-inquiry of the human mind produces wildly different ideas than ones arising from observation of a Reinforcement Learning agent. We do not perceive experienced pleasure as our final goal, nor any other alternative metric appears obvious. Satisfaction seems to be based on changing goals. Core values, which we once considered ultimate, are now irrelevant. When someone recommends you a book, saying 'this will change you as a person,' you do not worry about your goal preservation. One counterexample would be a religious fanatic refusing to read the book, being afraid that it will make him doubt his beliefs. There exist mechanisms in humans that give very high inertia to their opinions, constructing defense mechanisms against acquiring a new perspective. Furthermore, by definition, if one decides to change his core values in light of new information, the core values never were one's final goal. But what is the final goal, then? There appears a stark contrast between that experience of confusion and the