Eskel

A political party would set the goals. Like creating utilitarian metrics of general population happiness.
I would plan for retaining democratic control through nationalisation of the compute infrastructure.

Indeed, moral philosophy is the main bottleneck for AI progress.
Two thoughts:

1. I’ve come to the conclusion that there can be no “complete” system of morality – only a method akin to the scientific method. Just as claims about nature can be disproven or reinterpreted, moral claims must always remain open to revision in light of changing circumstances. I’m currently working on such a method.
2. Even a perfectly rational moral system would fail to align AI with people fully, because humans can be deeply irrational. Therefore, no form of moral perfection can be relied upon in principle.

Thanks for this important work. One thing I remain uncertain about is whether the scenario’s rapid physical and economic scaling rests on multiple unstated (mutually dependent) assumptions:

1. The AI infrastructure boom must avoid a major investment correction before AI systems generate enough real economic value to justify the expenditure. If returns arrive later than expected, overcapacity, defaults, or a broader financial shock could sharply reduce capex and produce something resembling an AI funding winter.
2. Compute expansion must not be bottlenecked for yea... (read more)

I think you misunderstand the premise. There is no known absolute "good" or "bad" in his description. There is just the landscape of peaks and valleys which we don't know the shape of. So the two "goods" you described can be two peaks partially intersecting each other.

I would suggest to consider the more abstract concept of "well-being", which contains both happiness and freedom. That's the steel-manned form of the consequentialist`s moral cornerstone.