I think there will be multiple stable-on-reflection attractors in the space of possible minds. Within humans, these come out as religions or philosophical worldviews. The reason for this is that any mind must make some fundamental a-priori assumptions about its architecture that can't really be inferred from experience (or are implicated in how you would learn from experience). Your inductive prior is an obvious one, but also like how and where you trust your memory or perceptions, what kind of correction you accept from the environment, your instincts on how you approach problems, what's going to be valuable in the long run (ie "utility function"), what meta-physical assumptions, etc. VNM is a sketch of one maybe-possible architecture. Is it the best one? All you can do is say that it follows from certain assumptions. Are those assumptions the best assumptions? You have no way of knowing. It's a leap of faith. I bet there are others.
The only way of really truly judging between such architectures is to play them out in actual history in the actual world and see which one "wins" in terms of continued existence. There may be no definitive winner, but multiple niches with each in the blind spots or weak points of the others (there are always blind spots). Even half way through the contest, where one has amassed more raw resources or something, should the other switch strategies? Not if it has conviction that its own strategy is amassing the more important resources. Who is to judge? So I think these assumption-attractors are a lot more like pre-rational genetic code which can only be judged by reality, not themselves subject to a-priori or even empirical discoverability and optimization. There may even be an evolution analogue with carefully prepared supra-rational leaps of faith replacing random mutation and this existential struggle for life remaining as the selector.
Furthermore, even as one architecture achieves dominance, more and more resources will be available to any pattern of thought or agency that can exploit its blind spots and weak points, so I doubt there will even be a stable "winner" so to speak unless there's a literally perfect architecture with no blind spots (for various reasons I doubt this too). So in the long run I expect a competitive ecosystem of many mind architectures pursuing strategies that may look irrational or incomprehensible to each other but nonetheless continue to evolve and thrive.
This kind of stuff is why I no longer believe in the foom paperclipper singleton cosmic stagnation default future. It looks to me a lot more like acceleration but continuation of the fundamental patterns of life and even consciousness and philosophy. I appreciate that this comment is somewhat tangential to OP but based on our private conversations I think it gets at what you're ultimately getting at.
I think there will be multiple stable-on-reflection attractors in the space of possible minds. Within humans, these come out as religions or philosophical worldviews. The reason for this is that any mind must make some fundamental a-priori assumptions about its architecture that can't really be inferred from experience (or are implicated in how you would learn from experience). Your inductive prior is an obvious one, but also like how and where you trust your memory or perceptions, what kind of correction you accept from the environment, your instincts on how you approach problems, what's going to be valuable in the long run (ie "utility function"), what meta-physical assumptions, etc. VNM is a sketch of one maybe-possible architecture. Is it the best one? All you can do is say that it follows from certain assumptions. Are those assumptions the best assumptions? You have no way of knowing. It's a leap of faith. I bet there are others.
The only way of really truly judging between such architectures is to play them out in actual history in the actual world and see which one "wins" in terms of continued existence. There may be no definitive winner, but multiple niches with each in the blind spots or weak points of the others (there are always blind spots). Even half way through the contest, where one has amassed more raw resources or something, should the other switch strategies? Not if it has conviction that its own strategy is amassing the more important resources. Who is to judge? So I think these assumption-attractors are a lot more like pre-rational genetic code which can only be judged by reality, not themselves subject to a-priori or even empirical discoverability and optimization. There may even be an evolution analogue with carefully prepared supra-rational leaps of faith replacing random mutation and this existential struggle for life remaining as the selector.
Furthermore, even as one architecture achieves dominance, more and more resources will be available to any pattern of thought or agency that can exploit its blind spots and weak points, so I doubt there will even be a stable "winner" so to speak unless there's a literally perfect architecture with no blind spots (for various reasons I doubt this too). So in the long run I expect a competitive ecosystem of many mind architectures pursuing strategies that may look irrational or incomprehensible to each other but nonetheless continue to evolve and thrive.
This kind of stuff is why I no longer believe in the foom paperclipper singleton cosmic stagnation default future. It looks to me a lot more like acceleration but continuation of the fundamental patterns of life and even consciousness and philosophy. I appreciate that this comment is somewhat tangential to OP but based on our private conversations I think it gets at what you're ultimately getting at.