Prayer, by any other name …
"What can a twelfth-century peasant do to save themselves from annihilation? Nothing."
She did something. She passed on a religious meme whose descendents have inspired me, in turn, to pass on the idea that we should engineer a world that can somehow reach backward to save her from annihilation. That may not prove possible, but some possibilities depend on us for their realization.
A Jewish prophet once wrote something like this: "Behold, I will send you Elijah the prophet before the coming of the great and dreadful day of the Lord: And he shall turn the heart of the fathers to the children, and the heart of the children to their fathers, lest I come and smite the earth with a curse." The Elijah meme has often turned my heart toward my ancestors, and I wonder whether we can eventually do something for them.
Unless we are already an improbable civilization, our probable future will be the civilization we would like to become only to the extent that such civilizations are already probable. The problem of evil is for the absolutely omnipotent God -- not for the progressing God.
Eliezer, your characterization of religion is not generally accurate, as evidenced by the fact that not all religious persons posit an irreducibly complex God. As one example, Mormons posit a material God that became God through organizing existing matter according to existing laws.
On the other hand, I wonder, do you attribute irreducible complexity to quarks?
Here's my metamorality. Using these terms broadly, law is to a community as will is to an individual, and law is to an environment as desire is to an anatomy. Good is communal law congruent with individual will, anatomical desire and environmental law. Joy is individual will congruent with communal law, environmental law and anatomical desire. Pleasure is anatomical desire congruent with environmental law, communal law and individual will. Order is environmental law congruent with anatomical desire, individual will and communal law. Evil, misery, pain and chaos are incongruencies among communal laws, individual wills, anatomical desires and environmental laws.
"So at the end of the day, I embrace the principle: 'Question your brain, question your intuitions, question your principles of rationality, using the full current force of your mind, and doing the best you can do at every point.'"
. . . to the extent that doing so increases your power, as illustrated by the principle you embrace to a greater extent:
"The point is to win."
That's the faith position.
"Everything, without exception, needs justification."
. . . except that toward which justification is aimed: power.
"The important thing is to hold nothing back in your criticisms of how to criticize . . ."
Yet you do, as illustrated by your power, which is actually that toward which you are applying full force, playing to win. Rationalism is great to the extent it is empowering. To the extent it weakens us, we abandon it.
Eliezer, do you intend your use of "artificial intelligence" to be understood as always referencing something with human origins? What does it mean to you to place some artificial intelligences outside the scope of posthuman mindspace? Do you trust that human origins are capable of producing all possible artificial intelligences?
Indeed. Faith, by any other name ...