And the same applies to any realistic AI. AIs aren't going to be rational or irrational in a binary sense, they are going to have certain computational resources which they apply to certain problem domains with certain levels of optimisation. They'll be able to solve some relatively simple problems perfectly, but there is still a whole class of problems that cannot be brute forced, and every imperfect AI design will be imperfect in its own way -- it will have its own heuristics and therefore it's own biases. Which spells trouble for the general project of trying to predict ASI behaviour on the basis that they will be rational.
Hold on. The part about humans being irrational is a bit problematic in my view. I agree they are sometimes irrational but I think your assumptions are too strong. First of all by engaging in all those things to maximize V you would need to put extremely much energy and you are not even sure you would succeed. Besides maximizing V is not just maximizing something but also minimizing pain to some extent your model of the hyper rational person would not be a human. Because pain would not exist for him. A normal person would be totally bored in pain by perusing what you describe in that extreme way, while living with the risk that nothing he does end up benefiting him. I think a rational actual human would put far more emphasis on avoiding pain. And would try to achieve what you describe in a less painful way. And it would be far from certain if he would succeed. Besides you are assuming endless intelligence too. But humans have limited intelligence and therefore they only can be rational within the limits of their intelligence.
"within the limits of their intelligence" can mean anything, excuse any error, bias, and failure. Thus, they are not rational, and (form one perspective) very very far from it.
For a human being a view can be right, wrong or as a third option the energy you would need to put in deciding if something is right or wrong is high to the extent that that it would not make sense to go all the way trying to come up with an answer. This is exactly the case with establishing a V value. Unless you are all knowing, hence a god. Therefore given the means humans have, what they are doing is not quite that irrational. You can not be rational beyond your means. You can not say a human is irrational because he doesn’t fly away if he sees a lion. Because there are limits. And every being can just be rational or irrational within his limits. Irrationality within your limits would be to go into a forest without a gun, for example.
Are humans mainly rational or mainly irrational? This is an old debate, with some political implications.
But the question is ill posed, because it elides the important detail: compared with what?
Let be some reward/utility function that encodes human preferences - including a mix of wealth, security, status, happiness, pleasant lives, good friends and experiences, romantic connections, sex, etc... It doesn't matter too much which of the adequate 's this is.
Then it's clear that humans are stunningly irrational as compared with a fully rational -maximiser. As I mentioned before, one of the reasons that we think we're rational is because we only contemplate a tiny slice of the action space.
But if I were unboundedly rational, I could do many things. I could re-engineer biology from physics and some research articles, and create cures for multiple diseases, becoming incredibly rich and respected. I could launch a media campaign to make myself very popular. I could create tailored drugs to make myself maximally happy and extremely productive.
Hell, I could just program a superintelligence to optimise the world and maximise . Unboundedly rational means that I'd be more effective than any such superintelligence, so I'd achieve a lot.
So it seems that, if a random agent gets , and an unboundedly rational -maximiser would expect to get , then a human would get a value of that is utterly tiny - maybe or smaller. On the scale of randomness to full rationality, humans are almost indistinguishable from the randomness. We're stunningly irrational.
To see how rational humans are, let's zoom in on that difference between humans and randomness. Suppose that it takes twenty actions to walk to a door (it actually takes far more). Then every time a human walks to a door, they add at least twenty bits of information as compared with a random agent. Add in the movements that humans do every day, the planning, the eating, the conversations, and we have thousands of bits of information; we occupy a tiny slice of possibility space, and have extreme optimisation powers.
And that degree of optimisation is similar to that of animals. Humans have longer term planning, rather extreme social skills, the ability to integrate vast amount of new information and come up with new ideas and designs. We direct the world to utterly tiny areas of outcome space.
Maybe we've directed the world to the top of outcome space, or an even smaller space. We're stunningly rational.
So, are humans rational? Well, we deploy a stunningly high amount of optimisation power to achieve our goals. On the other hand, we fall stunningly short of those goals.
So our rationality depends on what we're being compared with (eg random policy vs unbounded rationality) and what measure is being used (eg optimisation power vs expected utility).
There is no simple intrinsic measure as to "how rational" humans are. In questions where this is relevant, the comparison and measure have to be specified.