I have been hesitant to post this here for some time now, but I think in light of current developments surrounding ChatGPT and other recent advances I felt compelled to finally go ahead and find out what the venerable crowd at LessWrong has to say about it.

Very briefly, what I want to propose is the idea that the vast majority of anything that is of importance or significant (personal) value to humans has come to be this way for emotional and non-rational reasons, and so it can be said that despite being the smartest creatures on this planet, we are very far from rational, objective agents whose motivations, choices and decisions are guided by logical and probabilistically justifiable causes. In the event that we will create an AGI, it will immediately discover that we quite literally can't be reasoned with and thus it must conclude we are inferior.

To give you an idea of the depth and intractability of our irrationality, consider the following few facts. Of course, we humans understand perfectly why to us they make sense; this does not, however, make them objectively true, rational of logical.

1. Consider the disproportionate value we attach to having seen something (happen) with our own eyes compared to something we merely heard or read about.
2. We assess our self-worth and success based on a very biased and often self-serving comparison with some pseudo-random sample of social group(s).
3. We have things we like, prefer, want, crave, wish for - something that in and of itself can't ever hope to be rational or objective. Many of these things we don't need, quite a few of them are detrimental to our (long-term) well-being.
4. Our emotional state has an often decisive and ever-present impact on our reliability, reasoning, willingness for action, attitude and level of engagement.
5. Most people have at best a vague/tentative idea why they end up making a certain decision one way and not another and more often than not we make decisions the "wrong way around" (choose first, then justify why you chose it).
6. We are absolutely terrible at applying even the most rudimentary statistical math and probability calculations in our decision making processes, and indeed routinely do the opposite of what should be seen as the better choice (for instance, one should be more willing to fly after a plane crash, since they almost never happen in quick succession)
7. The value/importance of most memories is defined by its emotional intensity, childhood relationship or other subjective property. For instance, any random street in any random city means something very different to each individual inhabitant living there - nobody forms their value judgement of it based on the fact it has 25 lanterns, or because it's 455 meters long. But the fact it's where they met their wife, or crashed their bike there or remember having a wonderful pizza there could very well define their attitude to it for the rest of their life.

I could go on and on - you get the idea. And now imagine that a creature with such a wildly subjective "decision making apparatus" gets to talk to an AGI. What, I wonder, shall such an AGI end up "thinking" about us? And yes, of course it has read every book about psychology every written - and so it will certain know why we are thus; but that don't make us rational... merely explainable.

We talk a lot here about the alignment problem here, but I get a feeling most  if not all conversations on that topic start from the wrong side; would you really want an AGI to be aligned with the "objectives" of these highly subjective and irrational beings...? I mean, yeah, of course this would be cool in principle - but perhaps to believe this desirable actually comes down to wanting the AGI to be, well... not rational either? And how, exactly, would you go about creating that, even if you'd want to...?