Humanity as a super-organism, more colloquially known as civilization, is often viewed as having an indefinite lifespan. Therefore, when modelling the risk/reward of delaying or accelerating the development of superintelligence, risk caused by insufficient time to align ASI infinitely outweighs any possible reward. At least, this is the case for a simple model. Complicating the model by assigning a lifetime to civilization and using human lives as our utility function, there is now a point in time ludicrously far into the future where the risk/reward ratio of delaying ASI is below 1. Further complicating our model by incorporating other existential risks that ASI could resolve but that we potentially could not (e.g., asteroid strikes), the point in time at which the ratio falls below 1 goes from being ludicrously far into the future to just being very far into the future. It's even feasible to place that point within the next few decades by making certain overoptimistic assumptions about the likelihood of a threat like nuclear war, and very conservative assumptions about whether ASI could solve it. This could also be accomplished by additionally complicating the model.
Overall, though, modelling risk/reward in a way that regards humanity as a super-organism typically places the point at which delaying ASI development is less rewarding than it is risky quite far out into the future.
Now let's try modelling humanity as a collection of individuals that exist in the present. We make one core assumption with this model: that the probability of reversing aging or achieving any other form of immortality has a low probability of occurring within the average lifespan of the current population. Under this model, risk is ultimately capped by the fact that individual humans have limited lifespans. In fact, for the average individual, the point at which the risk/reward for delaying ASI that could reverse aging may have already gone below 1.
Put simply, it's likely that every individual currently alive today is going to die of aging, which massively caps the downsides of accelerating the development of existentially risky ASI. In fact, such a model would suggest that the upsides of accelerating ASI development are infinitely higher than the downsides.
I believe such a model is useful as a rational alternative explanation for the current accelerationism and shirking of regulations in the AI sector, as opposed to proverbial "launch fever." It's also a worthwhile reminder that even collectively, the subset of humans living during the development of AGI and ASI are heavily misaligned with humanity as an extremely long-living super-organism.
Humanity as a super-organism, more colloquially known as civilization, is often viewed as having an indefinite lifespan. Therefore, when modelling the risk/reward of delaying or accelerating the development of superintelligence, risk caused by insufficient time to align ASI infinitely outweighs any possible reward. At least, this is the case for a simple model. Complicating the model by assigning a lifetime to civilization and using human lives as our utility function, there is now a point in time ludicrously far into the future where the risk/reward ratio of delaying ASI is below 1. Further complicating our model by incorporating other existential risks that ASI could resolve but that we potentially could not (e.g., asteroid strikes), the point in time at which the ratio falls below 1 goes from being ludicrously far into the future to just being very far into the future. It's even feasible to place that point within the next few decades by making certain overoptimistic assumptions about the likelihood of a threat like nuclear war, and very conservative assumptions about whether ASI could solve it. This could also be accomplished by additionally complicating the model.
Overall, though, modelling risk/reward in a way that regards humanity as a super-organism typically places the point at which delaying ASI development is less rewarding than it is risky quite far out into the future.
Now let's try modelling humanity as a collection of individuals that exist in the present. We make one core assumption with this model: that the probability of reversing aging or achieving any other form of immortality has a low probability of occurring within the average lifespan of the current population. Under this model, risk is ultimately capped by the fact that individual humans have limited lifespans. In fact, for the average individual, the point at which the risk/reward for delaying ASI that could reverse aging may have already gone below 1.
Put simply, it's likely that every individual currently alive today is going to die of aging, which massively caps the downsides of accelerating the development of existentially risky ASI. In fact, such a model would suggest that the upsides of accelerating ASI development are infinitely higher than the downsides.
I believe such a model is useful as a rational alternative explanation for the current accelerationism and shirking of regulations in the AI sector, as opposed to proverbial "launch fever." It's also a worthwhile reminder that even collectively, the subset of humans living during the development of AGI and ASI are heavily misaligned with humanity as an extremely long-living super-organism.