charding

Fair enough & I appreciate the follow-up.

Though I will say --- It seems we need to find lessons somewhere in history, in part because we aren't smart enough as a species to reason purely from first principles. I'm certainly not smart enough for that, anyway.

When looking for lessons on AI, nuclear development may be the least worst historical analogy.

Thank you for explaining.

The societal readiness plot doesn't seem to have a log-ish y axis, considering the shapes of the trend lines.

If the alignment plots were also drawn without a log-ish y axis, then they might look as bad as the societal readiness plot or -- if not equally bad -- then at least substantially worse than they do now.

I'm questioning plotting decisions for fake plots ...I know. This may seem like splitting hairs. However, to me, there is a major difference between requiring linear vs. exponential power law improvement in alignment, to take us to "what we need".

The plots state that their x axes show capabilities on log scale, but what scales were intended for the y axes?

We might expect that the y axes are on linear (untransformed) scale. However, this would imply that multiplicative increases in AI capability can be addressed safely by making only additive amounts of progress in alignment (dashed green line on plots).

In general, multiplicative outmatches additive. How can we be confident that additive alignment progress would be enough?

Alternatively, we could view y axes as being on log scale. Yet, then the gap b... (read more)

Thanks & no apology needed : )

Thank you to your dad for offering to answer questions.

Sometimes people make the argument that the U.S. needs to race toward AGI-ASI as rapidly as possible, because if China obtains it first, then the risks to the U.S. are unacceptably high. However, this argument can also be an appealing excuse for people in the U.S. who would wish to go full-speed toward AGI-ASI even if there were no competition from China.

I imagine similar arguments are also made in China, with the roles of the U.S. and China reversed.

Does your dad have thoughts about these kinds of ar... (read more)

Thank you for your response!

The go/no-go model is not meant to show that a P(doom) of up to 97% is "acceptable" 

I should have been clearer, yes. I meant that the 97% deals are acceptable to your go/no-go model, not to you or your later models.

However, I think my arguments apply equally to your later models, just with P(doom) different from 97%. (See below.)

a series of more complicated models are introduced that take into account some of these other factors.

Thank you for running the more complicated models. (And, in case unclear, I did read all your article... (read more)

Bostrom's Footnote 21 seems innocuous, but to me it unravels a lot of the argument Bostrom is making.^[1]

Bostrom's central go/no-go model suggests a P(doom) of up to 97% is acceptable if life expectancy rises to 1,400 years post-AGI. 

Footnote 21 clarifies, ``more generally, we could take P(doom) to be the expected fraction of the human population that dies when superintelligence is launched.''

So, suppose one could extend life 1,400 years for 3% of humans at the cost of killing 97% right now. How should we reply to this deal? Bostrom's go/no-go model says to... (read more)

Bostrom's results seem very sensitive to deviations from a wholly person-affecting perspective. To investigate, I coded up the model from Appendix A with one modification: I supposed that, instead of being wholly self interested, people are willing to sacrifice 10% of life expectancy for the sake of all future generations.

My method was to calculate the launch time that is later than the optimal time-point according to a selfish view, but only so much that life expectancy is reduced 10% from the selfish optimum.[^1] This method is crude, but illustrates ho... (read more)