At this point timelines look short enough that you likely increase your personal odds of survival more by increasing the chance that AI goes well than by speeding up timelines. Also I don't see why you think cryonics doesn't make sense as alternative option.
- The timelines certainly still looked short enough a couple of months ago. But what prompted me to write this was the 13th observation: the seemingly snowballing Pause movement, which, once it reaches certain threshold, has a potential to significantly stifle the development of AI. Analogies: human genetic enhancement, nuclear energy. I'm not sure whether this is already past the point of countering the opposite forces (useful applications, Moore's law), but I'm also not sure that it isn't (or won't be soon).
- Cryonics is a very speculative tech. We don't understand how much information is lost in the process, scientific evidence seems lacking overall - consensus being it's in the ~few percent success probability region, future AI (future society) would have to want to revive humans instead of creating new ones, etc.
> consensus being it's in the ~few percent success probability region
Consensus among who? I haven't been able to find a class of experts I'd defer to. We have Alcor, who are too partial, we have the society of cryobiology who openly refuse to learn anything about the process and threaten to exile any member who does, and I have random members of the rationalist community who have no obligation to be right and just want to sound grounded.
I do not want the timing of AGI development to be based on my personal life-expectancy. Going further I think most decent people would be willing to sacrifice their own life to prevent their civilization from going extinct, and I think it would be a very honorable thing to do [1]. Every single generation to come is at stake, so I don't think my own life bears much on whether to defend all of theirs.
- ^
Not that I am aware of any such opportunity, to be clear.
Every single generation to come is at stake, so I don't think my own life bears much on whether to defend all of theirs.
Note that the idea that >10^50 lives are at stake is typically premised on the notion that there will be a value lock-in event, after which we will successfully colonize the reachable universe. If there is no value lock-in event, then even if we solve AI alignment, values will drift in the long-term, and the stars will eventually be colonized by something that does not share our values. From this perspective, success in AI alignment would merely delay the arrival of a regime of alien values, rather than prevent it entirely. If true, this would imply that positive interventions now are not as astronomically valuable as you might have otherwise thought.
My guess is that the idea of a value lock-in sounded more plausible back in the days when people were more confident there will be (1) a single unified AI that takes control over the world with effectively no real competition forever, and (2) this AI will have an explicit, global utility function over the whole universe that remains unchanging permanently. However, both of these assumptions seem dub...
The relevance of value drift here sounds to me analogous to the following exchange.
Alice, who is 10 years old: "Ah, I am about to be hit by a car, I should make sure this doesn't happen else I'll never get to live the rest of my life!"
Bob: "But have you considered that you probably won't really be the same person in 20 years? You'll change what you value and where you live and what your career is and all sorts. So I wouldn't say your life is at stake right now."
The concern about AI misalignment is itself a concern about value drift. People are worried that near-term AIs will not share our values. The point I'm making is that even if we solve this problem for the first generation of smarter-than-human AIs, that doesn't guarantee that AIs will permanently share our values in every subsequent generation. In your analogy, a large change in the status quo (death) is compared to an arguably smaller and more acceptable change over the long term (biological development). By contrast, I'm comparing a very bad thing to another similarly very bad thing. This analogy seems mostly valid only to the extent you reject the premise that extreme value drift is plausible in the long-term, and I'm not sure why you would reject that premise.
1Putting "cultural change" and "an alien species comes along and murders us into extinction" into the same bucket seems like a mistake to me. I understand that in each literally one set of values replaces another. But in the latter case, the route by which those values change is something like "an alien was grown in order to maximize a bunch of functions that we were able to define like stock price and next-token prediction and eventually overthrew us", and I think that is qualitatively different than "people got richer and wealthier and so what they wanted changed" in a way that is likely to be ~worthless from our perspective.
From reading elsewhere my current model (which you may falsify) is that you think that those values will be sufficiently close that they will still be very valuable from our perspective, or about as valuable as people from 2000 years ago would think us today. I don't buy the first claim; I think the second claim is more interesting but I'm not really confident that it's relevant or true.
(Consider this an open offer to dialogue about this point sometime, perhaps at my dialogues party this weekend.)
Yes, I believe this point has practical relevance. If what I'm saying is true, then I do not believe that solving AI alignment has astronomical value (in the sense of saving 10^50 lives). If solving AI alignment does not have astronomical counterfactual value, then its value becomes more comparable to the value of other positive outcomes, like curing aging for people who currently exist. This poses a challenge for those who claim that delaying AI is obviously for the greater good as long as it increases the chance of successful alignment, since that could also cause billions of currently existing people to die.
I've seen this kind of opinion before (on Twitter, and maybe reddit?), and I strongly suspect that the average person would react with extreme revulsion to it. It most closely resembles "cartoon villain morality", in being a direct tradeoff between everyone's lives and someone's immortality. People strongly value the possibility of their children and grandchildren being able to have further children of their own, and for things in the world to continue on. And of course, the statement plays so well into stereotypes of politically-oriented age differences: Old people not sufficiently caring about what happens after they die, so they'll take decisions that let young people deal with catastrophes, young people thinking they'll never die and being so selfish that they discount the broader world outside themselves, etc. If anything, this is a "please speak directly into the microphone" situation, where the framing would pull people very strongly in the direction of stopping AGI.
But yes, one of the pieces of evidence is how old people don't seem to particularly care about the future of civilisation.
I don't think this is true. Stereotype evidence: your average conservative old person loves their country and is mad that college campuses are liberal because it means elites will ruin it.
I agree with the Statement. As strongly as I can agree with anything. I think the hope of current humans achieving... if not immortality, then very substantially increased longevity... without AI doing the work for us, is at most a rounding error. And ASI that was even close to aligned, that found it worth reserving even a billionth part of the value of the universe for humans, would treat this as the obvious most urgent problem and solve death pretty much if there's any physically possible way of doing so. And when I look inside, I find that I simply don't care about a glorious transhumanist future that doesn't include me or any of the particular other humans I care about. I do somewhat prefer being kind / helpful / benificent to people I've never met, very slightly prefer that even for people who don't exist yet, but it's far too weak a preference to trade off against any noticeable change to the odds of me and everyone I care about dying. If that makes me a "sociopath" in the view of someone or other, oh well.
I've been a supporter of MIRI, AI alignment, etc. for a long time, not because I share that much with EA in terms of values, but because the path to the future having an...
Apparent moral alien here. Hi! I'm pretty astonished how many people apparently believe something like that statement. In my world, it's an off-the-deep-end perverse and sociopathic sort of value to hold. Of course, it's consistent. I can work with sociopaths, because to a first approximation I can work with anyone. But if there's enough delay, cooperation is going to become strained, because I'm quite happy with me dying, and you dying, and your loved ones dying (ideally painlessly having lived fulfilling lives!), if it turns out to be necessary in order for there to be a valuable future at all.
Despite introducing myself as an alien, I currently think most humans don't espouse your statement, because most humans' moral circle is big enough to preclude it. Other humans reject it on the basis of philosophical open/empty individualism. Others reject it for superstitious reasons or due to ethical injunctions. Others still for the deceptive reasons you mentioned. There are basically a lot of reasons either not buy it, or be quiet about it if you do!
As an aside, it appears that in certain circles, the deceptive/prudent reasons you mention to not espouse this view are inverted: a sort ...
1
1Many humans, given a choice between
A) they and their loved ones (actually everyone on earth) will live forever with an X-risk p
B) this happens after they and everyone they love dies with an X-risk less than p
Would choose A.
Abortion has a sort of similar parallel but with economic risk instead of X risk, and obviously no immortality yet many are pro choice.
I think valuing the lives of future humans you don’t know of over the lives of yourselves and your loved ones is the alien choice here.
I kind of agree with the statement, but I don't quite agree with the implications: my timelines are already quite short and I think that if we contract things more, we pay a high cost in risk for little benifit unless you expect yourself and many of your loved ones to die before then. In which case, what's the point of discussing it?
Now if timelines were long, then yeah, it would make sense to accelerate AI insofar as it contributed to the continued surival of me and my loved ones. That could be through creating a FAI, or it could be via improved longevity tech, better disease treatment, improved cryonics, mind uploading etc. Which many people in this community are quite in favour of, even now.
Fifth observation: Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.
I think you underestimate how weird people find the desire for immortality. Why does pretty much no-one sign up for cryonics? Because it is unlikely to work? Nah, the odd's are maybe a few percent for current tech. Plenty of people should be willing to take a bet for continued life at those odds. Is it because of cost? Again, no. Life insurance isn't that expensive. The real reason is that it is weird.
I wrote about this in Appendix A of this post.
------
One might look at the rough 50/50 chance at immortality given surviving AGI and think "Wow, I should really speed up AGI so I can make it in time!". But the action space is more something like:
- Work on AI safety (transfers probability mass from "die from AGI" to "survive AGI")
- The amount of probability transferred is probably at least a few microdooms per person.
- Live healthy and don't do dangerous things (transfers probability mass from "die before AGI" to "survive until AGI")
- Intuitively, I'm guessing one can transfer around 1 percentage point of probability by doing this.
- Do nothing (leaves probability distribution the same)
- Preserve your brain if you die before AGI (kind of transfers probability mass from "die before AGI" to "survive until AGI")
- This is a weird edge case in the model and it conditions on various beliefs about preservation technology and whether being "revived" is possible
- Delay AGI (transfers probability from "die from AGI" to "survive AGI" and from "survive until AGI" to "die before AGI")
- Accelerate AGI (transfer probability mass from "survive AGI" to "die from AGI" and from "die before AGI" to "survive until AGI")
I t...
Two days ago I wrote a tweet with similar idea and it get unexpected boost - which means that it is important thing for people: 'Perhaps, we don't need superintelligent AI to solve aging quickly. Some more mundane level of AI will be enough." https://twitter.com/turchin/status/1719315694813130896
EY wrote against the statement, saying that why not all these people not signing for cryonics
M. Batin wrote in favor of this statement and even made a counter-appearance during AI Pause meeting in SF https://twitter.com/MikhailBatin/status/1719948531233140754
I also agree with the statement. I'm guessing most people who haven't been sold on longtermism would too.
When people say things like "even a 1% chance of existential risk is unacceptable", they are clearly valuing the long term future of humanity a lot more than they are valuing the individual people alive right now (assuming that the 99% in that scenario above is AGI going well & bringing huge benefits).
Related question: You can push a button that will, with probability P, cure aging and make all current humans immortal. But with probability 1-P, all humans die. How high does P have to be before you push? I suspect that answers to this question are highly correlated with AI caution/accelerationsim
1You have a very strange understanding of politics, wherein you have laymen who want to advance their interests at the expense of other peoples, who realize that would be unpopular if they stated exactly what they were doing, and then as a consequence lie on Twitter about needing to do it for a different reason. This is insufficiently cynical. People almost always genuinely think their political platform is the Right and Just one. It's just that people also have strong instincts to align themselves with whatever tribe is saying they're the Good Guys, and th...
My limited impression of "e/accs", and you may think this is unfair, is that most of them seem not to have any gears-level model of the problem at all, and have instead claimed the mantle because they decided amongst each other it's the attire of futurism and libertarianism. George Hotz will show up to the Eliezer/Leahy debates with a giant American flag in the background and blurt out stuff like "Somalia is my preferred country", not because he's actually going to live there, but because he thinks that sounds based and the point of the discussion for him is to wave a jersey in the air. I don't think Hotz has made the expected value calculation you mention because I don't think he's even really gotten to the point of developing an inside view in the first place.
Based on personal experience, you are definitely not the only one thinking about that Statement.
The statement seems like it's assuming:
-
we know roughly how to build AGI
-
we decide when to do that
-
we use the time between now and then to increase chance of successful alignment
-
if we succeed in alignment early enough, you and your loved ones won't die
I don't think any of these are necessarily true, and I think the ways they are false is asymmetric in a manner that favors caution
I think most humans agree with this statement in an "I emotionally want this" sort of way. The want has been sublimated via religion or other "immortality projects" (see The Denial of Death). The question is, why is it taboo, and it is taboo in the sense you say? (a signal of low status)
I think these elements are at play most in peoples mind, from lay people to rationalists:
- It's too weird to think about: Considering the possibility of a strange AI-powered world where either complete extinction or immortality are possible feels "unreal". Our instinct that e
I have expressed versions of this Statement, or at least parts of it, to those close to me. They are not in the EA or LW communities and so I have to rephrase a lot. Mostly along the lines of, "One way or another, the last generation to die of old age has most likely already been born. I'm not quite sure which one, but if we manage to survive the century as a species, then I expect it's my own, and that my nieces and nephew will be able to live for millennia. I hope I'm wrong about which generation it is."
For me, both AI and medicine are far outside my areas of expertise, so I'm focusing on other problems that I have a better chance of influencing, hence the passive framing. This is even more true for my family members.
Off-topic, but somewhat related. I want to know if there is anyone reading these words who is willing to admit that he or she is kind of hoping humanity will go extinct because humanity has been unfair to him or her or because (for some other reason) humanity is bad or unworthy.
Honestly, I haven’t seen much about individual biological immortality, or even significant life-extension, in the last few years.
I suspect progress on computational consciousness-like mechanisms has fully eclipsed the idea that biological brains in the current iteration are the way of the future. And there’s been roughly no progress on upload, so the topic of immortality for currently-existing humans has mostly fallen away.
Also, if/when AI is vastly more effective than biological intelligence, it takes a lot of the ego-drive away for the losers.
Who said anything about slowly and painfully?? I don't think a fully reflective you would endorse that.
(FWIW I'm not whoever downvoted your comments, though these definitely aren't to my taste and could be interpreted as threatening)
I'm very much of an egoist but I don't really agree with this statement. Even if the last smidge of my altruism were deleted, I still wouldn't rob banks or run a fraudulent asset exchange because I have a very cushy position in society as it is and cooperating with humanity in fact makes my life much better than not. I similarly wouldn't do existentially crazy things to try to reach the singularity for me personally. Even if normal people wouldn't understand how I'm burning the commons, you guys would.
I'm like way more likely to do norm-breaking stuff (lik...
I think a lot of people haven't fully considered how amazing immortality is. Your 7th observation is probably very accurate too. Socially sophisticated people want others to believe they're fighting for the greater good rather than their own self interest. That doesn't mean they're necessarily lying.
I fully agree with the Statement but also support Pause/Stop. The reason is that I'm highly confident that everyone dies if we achieve AGI in the near future, so since I'm still relatively young I prefer getting to live some more years instead.
Not much to add content wise but just to be one data point: I strongly agree and have talked about it with exactly one person. I have no plans to advance the position openly and proactively, largely for reasons stated, but would quietly and strongly support efforts to that end should I become aware of them.
As I've put it to my friend: the worst day for a transcendent AI to emerge is the week after I would die and the best day is the week before. We then argue over where "tomorrow" fits on that spectrum
I'm very glad that you're raising this topic for discussion. I've recently been thinking about the same thing myself. If I could delay the advent of AGI/TAI by, say, 100 years, would I? There are at least two relevant considerations here: 1) the commonly held (but not, to me, obviously true) assumption that delaying AGI/TAI increases the probability of it being safely deployed and 2) the costs of delaying AGI/TAI.
(2) motivates thinking really hard about whether (1) is true. General concern about AI safety also motivates thinking hard about whether (1) is true, since if delaying AGI/TAI does not increase the probability of a safe deployment, then we should think about what would increase it.
You should consider cross-posting to EAF. Would be interesting to see the poll differences, but might be fiddly to set up.
If I take the statement face value as written, s-risks are very relevant to disagreement - a life lived longer without solving sources of suffering tends to result in accruing damage, and saving my life immediately before I would die is one of the more likely ways to force me to live with the maximum possible damage dealt by (what just nearly killed me, and cumulative health troubles in general).
I notice that "don't die" is rarely a terminal value. Most people who say they want immortality are making some assumptions about the quality of life it would entail -- both in an s-risk sense and in a "technically people who are permanently comatose on life support aren't actually dead" sense as well.
I wouldn't call "don't die" a terminal value in the cryonics case either, because in that case you do die for awhile but then you get better. Eventual reincarnation after a delay seems meaningfully distinct from immortality, because immortality implies continuity of life.
Statement: I want to deliberately balance the caution and the recklessness in developing AGI, such that it gets created in the last possible moment so that I and my close ones do not die.
This Statement confuses me. There are several observations I can make about it. There are also many questions I want to ask but have no idea how to answer. The goal of this post is to deconfuse myself, and to get feedback on the points that I raised (or failed to raise) below.
First observation: The Statement is directly relevant to LW interests.
It ties together the issues of immortality and AI risk, both of which are topics people here are interested in. There are countless threads, posts and discussions about high-level approaches to AI safety, both in the context of "is" (predictions) and "ought" (policy). At the same time, there is still a strong emphasis on the individual action, deliberating on which choices to make to improve the to marginal effects of living life in a certain way. The same is true for immortality. It has been discussed to death, both from the high-level and from the individual, how-do-I-sign-up-for-Alcor point of view. The Statement has been approached from the "is", but not from the "ought" perspective. At the same time:
Second observation: No one talks about the Statement.
I have never met anyone who expressed this opinion, neither in-person nor online, even after being a part (although, somewhat on the periphery) of the rationalist community for several years. Not only that, I have not been able to find any post or comment thread on LW or SSC/ACX that discusses it, argues for or against it, or really gives it any attention whatsoever. I am confused by this since the Statement seems to be fairly straightforward.
One reason might be the:
Third observation: Believing in the Statement is low status, as it constitutes an almost-taboo opinion.
Not only no one is discussing it, but the few times when I expressed the Statement in person (at EA-infiltrated rationalists meetups), it was treated with suspicion or hostility. Although to be honest, I'm not sure how much this is me potentially misinterpreting the reactions. I got the impression that it is seen as sociopathic. Maybe it is?
Fourth observation: Believing in the Statement is incompatible with long-termism, and it runs counter to significantly valuing future civilisation in general.
Fifth observation: Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.
Most people value their lives, and the lives of those around them to a much greater extent than those far away from them. This is even more true for the future lives. The revealed-preference discount factor is bounded away from 1.
Sixth observation: The Statement is internally consistent.
I don't see any problems with it on the purely logical level. Rational egoism (or variants thereof) constitutes a valid ethical theory, although it is potentially prone to self-defeat.
Seventh observation: Because openly admitting to believing in the Statement is disadvantageous, it is possible that many people in fact hold this opinion secretly.
I have no idea how plausible this is. Judging this point is one of my main goals in writing this post. The comments are a good place for debating the meta-level points, but, if I am right about the cost of holding this opinion - not so much for counting its supporters. An alternative is this anonymous poll I created - please vote if you're reading this.
Eighth observation: The Statement has the potential to explain some of the variance of attitudes to AI risk-taking.
One way of interpreting this observation might be that people arguing against pausing/for accelerating AI developments are intentionally hiding their motivations. But, using the standard Elephant-in-the-brain arguments about people's tendency to self-delude about their true reasons for acting or believing in certain things, it seems possible that some of the differences between Pause-aligned and e/acc-aligned crowd come down to this issue. Again, I have no idea how plausible this is. (People in e/acc are - usually - already anonymous, which is a point against my theory).
Ninth observation: The Statement is not self-defeating in the sense of failure of ethical theories to induce coordination in the society/escaping prisoner dilemmas.
Although the Statement relies on the ethical judgement of gambling the entire future of our civilisation on my meagre preferences for my life and life of people around me, everyone's choices are to a greater or lesser extent aligned (depending on the similarity of life expectancy). Most probably we, as the "current population" survive together or die together - with everyone dying being the status quo. Trading off future lives does not create coordination problems (ignoring the possibility of acausal trade), leaving only the conflict between different age groups.
This leads me to the...
Tenth observation: Openly resolving the topic of the Statement leads to a weird political situation.
As we said, the Statement implicitly refers to the Pareto frontier of AI existential risk-taking, which is parametrised by the life expectancy of the individual. Therefore, assuming a sufficiently powerful political system, it is a negotiation about the cut-off point - who gets left behind and who gets to live indefinitely. There does not seem to be much negotiation space - fairly compensating the first group seems impossible. Thus, this seems like a high-conflict situation. In a real-life not-so-powerful system, it might lead to unilateral action of people left below the cut-off, but they might not have enough resources to carry out the research and development alone. Also, the probabilistic nature of that decision would probably not trigger any strong outrage response.
Eleventh observation: Much of the "working EA theory" might break when applied to the Statement.
How much do people posture altruism, and how much do they really care? EA signalling can be a good trade (10% of your income in exchange for access to the network and being well-liked overall). This was the argument SSC was making in one of the posts: the beauty of the EA today is that the inside doesn't really matter - being motivated by true altruism and being motivated by status are both still bringing the same result at the end. This can, however, break if the stakes are raised sufficiently high. (I personally approach the problem of death with a pascal-wager-like attitude, but even with a discount rate of 0.9, the choice comes down to a 10x life value).
Twelfth observation: The Statement does not trivialise either way: the optimal amount of neither recklessness nor caution is zero.
The latter is a consensus (at least on LW). The former is not. The only take I ever heard advocating for this was that cryonics + certainly aligned AI is worth more in expectation than the chance of aligned AI in our lifetimes. For obvious reasons, I consider this to be a very weak argument.
Thirteenth observation: For a median believer of the Statement, we might be passing the point where the balance is being too heavily weighted towards pause.
The fundamental difference between the pause side and the accelerate side is that pause has a much bigger social momentum to it. Governments are good at delaying and introducing new regulations - not so much at speeding up and removing them (at least not recently). I can't find a source, but I think EY admitted this in one of the interviews - the only good thing about the government is that it will grind everything to a complete stop. After seeing literally all major governments take interest (US executive order, UK summit), the academia heavily leaning towards pause, and the general public being more and more scared of AI - I worry that the pause is going to be too entrenched, will have too much political momentum to stop and reverse course. I don't know how much should I consider the market forces being a counterweight.
Fourteenth observation: Taking into account s-risks in addition to x-risk might change the tradeoff.
A final argument about which I am unsure - it seems plausible that the Statement should not be endorsed by anyone because of the potential of unaligned AI to cause suffering. But I do not think s-risks are sufficiently probable to pause indefinitely.
So: Am I the only one thinking those thoughts? Am I missing any obvious insights? Is that really a taboo view? Was it debated before and resolved definitely one way or another? Is this some standard philosophical problem that already has a Google-able name? Is there any downside to openly discussing this whole thing that I don't see?