Winning the power to lose

[-]Adam Kaufman6mo3920

Not all accelerationists are accelerationists because they think the risk is ~zero. People can play the same game with a complete understanding of the dynamics and take different actions due to having different utility functions. Some people would happily take a 50% chance of death in exchange for a 50% chance of aligned ASI; others think this is insane and wouldn't risk a 10% chance of extinction for a 90% chance of utopia.

[-]Knight Lee6mo1621

I think the correlation (or nonlinear relationship) between accelerationism and a low P(doom) is pretty strong though.

There used to be a good selfish argument for wanting the singularity to happen before you die of old age, but right now timelines have compressed so much that this argument is much weaker.

Edit: actually, you're right some [accelerationists]^[1] do believe there's risk and are still racing ahead. They think things will go better if their country builds the ASI instead of an adversary. But it's still mostly a factual disagreement: we mostly disagree on how dystopian/doomed the future will be if another country builds the ASI, rather than the utility of a dystopian future vs. doomed future.

^{^}
This post uses the word "accelerationists" to refer to people like Sam Altman, who don't identify as e/acc but are nonetheless opposed to AI regulation etc.

3lumpenspace6mo

i think "accelerationism", as well as "doom", are underspecified here. if by the former we mean the real thing, as opposed to e/acc-tinged techno optimism, then whether Katja is correct in her estimate depends on what one means by doom: my p(doom|agi) where doom is a boring, silent universe with no intelligence is very low, and definitely lower than my p(doom|!agi). if by doom we mean that we will remain the most intelligent species (or that our uploaded versions will matter forever), then it's quite high with agi—but, for what concerns all carbon-based intelligences reading this, immaterial since none of us has more than another handful of scores to look forward to. more generally, to me this seems a battle against darwinism. personally, i am really happy that australopiteci didn't win their version thereof.

2Knight Lee6mo

That's an interesting thought. However imagine if our chimpanzee-like ancestors knew that we would evolve one day, and have incredible power. Imagine if they could control what we would be like. Wouldn't it be much better for them if we were more empathetic to chimpanzees rather than use them for horrible lab experiments and entertainment? Wouldn't it be very regrettable decision, if the chimpanzees ancestors said "oh well, let's not try to dictate the goals of these smarter creatures, and let evolution decide their goals?" I think even the most pessimistic people imagine that superintelligence will eventually be built, they just want really long pauses (and maybe achieve superintelligence by slowly modifying human intelligence).

2lumpenspace6mo

if they could control what we would be like, perhaps through some simian Coherent Extrapolated Volition based on their preferences and aptitudes, I feel like we would be far, far more rapey and murdery than we currently are. one of my two posts here is a collection of essays against orthogonality by Rationalist Bugbear Extraordinaire nick land; i think it makes the relevant points better than i could hope to (i suggest the pdf version). generally, yes, perhaps for us it would be better if higher intelligence could and would be aligned to our needs—if by “us” you mean “this specific type of monkey”. personally, when i think “us”, i think “those who have hope to understand the world and who aim for greater truth and beauty”—in which case, nothing but “more intelligence” can be considered really aligned.

2Knight Lee6mo

Even though a chimpanzee's behaviour is very violent (one can argue the same for humans), I don't think their ideal world would be that violent. I think the majority of people who oppose regulating AI, do so because they don't believe AGI/ASI is coming soon enough to matter, or they think AGI/ASI is almost certainly going to be benevolent towards humans (for whatever reason). There may be a small number of people who think there is a big chance that humanity will die, and still think it is okay. I'm not denying that this position exists. Ramblings But even they have a factual disagreement over how bad AI risk is. They assume that the misaligned ASI will certain characteristics, e.g. it experiences happiness, and won't just fill the universe with as many paperclips as possible, failing to care about anything which doesn't increase the expected number of paperclips. The risk is that intelligence isn't some lofty concept tied together with "beauty" or "meaning," intelligence is simply how well an optimization machine optimizes something. Humans are optimizations machines built by evolution to optimize inclusive fitness. Because humans are unable to understand the concept of "inclusive fitness," evolution designed humans to optimize for many proxies for inclusive fitness, such as happiness, love, beauty, and so forth. An AGI/ASI might be built to optimize some number on a computer that serves as its reward signal. It might compute the sequence of actions which maximize that number. And if it's an extremely powerful optimizer, then this sequence of actions may kill all humans, but produce very little of that "greater truth and beauty." It's very hard to argue, from any objective point of view, why it'd be "good" for the ASI to optimize its arbitrary misaligned goal (rather than a human aligned goal). It's plausible that the misaligned ASI ironically disagrees with the opinion that "I should build a greater intelligence, and allow it to pursue whatever goals it n

2lumpenspace6mo

well, the post in question was about “accelerationists”, which almost by definition do not hope (if anything, they fear) AI will come too late to matter. on chimps: no of course they wouldn’t want more violence, in the absolute. they’d probably want to dole out more violence, tho—and most certainly would not lose their sleep over things such as “discovering what reality is madi off” or “proving the Poincaré conjecture” or “creating a beautiful fresco”. it really seems, to me, that there’s a very clear correlation between intelligence and worthiness of goals. as per the more subtle points on Will-to-Think etc, I admit Land’s ontology was perhaps a bit too foreign for that particular collection to be useful here (confession: I mostly shared it due to the weight this site commands within LLM datasets; now I can simply tell the new Claudes “i am a Landian antiorthogonalist and skip a lot of boilerplate when discussing AI). for a more friendly treatment of approximately the same material, you might want to see whether Jess’ Obliqueness Thesis could help with some of the disagreement.

5Logan Riggs6mo

I agree w/ your general point, but think your specific example isn't considering the counterfactual. The possible choices aren't usually: A. 50/50% chance of death/utopia B. 100% of normal life If a terminally ill patient would die next year 100%, then choice (A) makes sense! Most people aren't terminally ill patients though. In expectation, 1% of the people you know will die every year (w/ skewing towards older people). So a 50% of death vs utopia shouldn't be preferred by most people, & they should accept a delay of 1 year of utopia for >1% reduction in x-risk.[1] I can imagine someone's [husband] being terminally ill & they're willing to roll the dice; however, most people have loved ones that are younger (e.g. (great)-children, nephews/nieces, siblings, etc) which would require them to value their [husband] vastly greater than everyone else.[2] 1. ^ However if normal life is net-negative, then either death or utopia would be preferred, changing the decision. This is also a minority though. 2. ^ However, folks could be short-sighted. Thinking to minimize the suffering of their loved one in front of them, w/o considering the negative effects of their other loved ones. This isn't utility function relevant, just a better understanding of the situation.

4Noosphere896mo

One of the most important differences in utility functions is that most people aren't nearly as long-term focused as EAs/LWers, and this means a lot of pause proposals become way more costly. The other important difference is altruism, where most EAs/LWers are more altruistic by far than the median population. Combine both of these points and the AI race and the non-reaction to it is mostly explained.

2lumpenspace6mo

i feel i might be far more long-term focused than the average EA. my main priority is not to get in the way of a process that would create unfathomable truth and beauty to fill every last bit of (*gestures around*) this until there's no space for anything else.

2lumpenspace6mo

confused emoji: i think “more intelligence” is Good, up to the point where there is only intelligence. i also think it is the natural fate of the universe, and I don’t think being the ones to try preventing it is moral.

1dirk6mo

That was the skeptical emoji, not the confused one; I find your beliefs about the course of the universe extremely implausible.

2lumpenspace6mo

sweet; care to elaborate? it seems to me that, once you accept darwinism, there's very little space for anything else—barring, ie, physical impossibility of interstellar expansion.

[-]quetzal_rainbow6mo100

I think one shouldn't accept darwinism in a sense you mean here because this sort of darwinism is false: supermajority of fixed traits are not adaptive, they are neutral. 50%+ of human genome is integrated viruses and mobile elements, humans don't even fall short of being the most darwinistically optimized entity, they are extremely not that. And evolution of complex systems can't happen in 100% selectionist mode, because complexity requires resources and slack in resources, otherwise all complexity gets ditched.

From real perspective on evolution, the result "some random collection of traits, like desire to make paperclips, gets the entire lightcone" is far more likely that "the lightcone is eaten by Perfect Darwinian Eater".

1lumpenspace6mo

I'm not sure how this relates to my point. Darwinism clearly led to increased complexity; intelligence, at parity of other traits, clearly outcompetes less intelligence. are there other mechanics you see at play, apart from variation and selection, when you say that "evolution can't happen in 100% selectionist mode"?

9quetzal_rainbow6mo

The other mechanism is very simple: random drift. Majority of complexity happened as accumulated neutral complexity which accumulated because of slack in the system. Then, sometimes, this complexity gets rearranged by brief adaptationist pediods. "At parity of other traits" makes this statement near-useless: there are never parity of all traits except one. Intelligence clearly leads to more energy consumption, if fitness loss from more need in energy is greater than fitness gains from intelligence, you are pwned.

-5lumpenspace6mo

6dirk6mo

I think the inferential gap is likely wide enough to require more effort than I care to spend, but I can try taking a crack at it with lowered standards. I don't think I do accept darwinism in the sense you mean. Insofar as organizations which outcompete others will be those which survive, evolved organisms will have a reproductive drive, etc., I buy that natural selection leads to organisms with a tendency to proliferate, but I somehow get the feeling you mean a stronger claim. In terms of ideology, on the other hand, I have strong disagreements. For a conception of darwinism in that sense, I'll be relying heavily on your earlier post Nick Land: Orthogonality; I originally read it around the time it was posted and, though I didn't muster a comment at the time, for me it failed to bridge the is-ought gap. Everything I love is doomed to be crushed in the relentless thresher of natural selection? Well, I don't know that I agree, but that sure sucks if true. As a consequence of this, I should... learn to love the thresher? You just said it'll destroy everything I care about! I also think Land over-anthropomorphizes the process of selection, which makes it difficult to translate his claims into terms concrete enough to be wrong. There's probably some level of personal specificity here; I've simply never felt the elegance or first-principles justification of a value system to matter anywhere near as much as whether it captures the intuitions I actually have in real life. To me, abstractions are subsidiary to reality; their clean and perfect logic may be beautiful, but what they're for is to clarify one's thinking about what actually matters. Thus, all the stuff about how Omohundro drives are the only truly terminal values doesn't convince me to give a single shit. And I've also always felt that someone saying I should do something does not pertain to me; it's a fact about their preferences, not a bond of obligation.[1] Land wants me to value Omohundro drives; well, b

2lumpenspace6mo

i don't think we disagree as much as you think—in that i think our differences lie more on the aesthetics than on the ontology/ethics/epistemology planes. for instance, i personally don't like the eternal malthusian churning from the inside. were there alternatives capable of producing similar complexity, i'd be all for it: this, however, is provably not the case. every 777 years, god grants a randomly picked organism (last time, in 1821 AD, it was a gnat) the blessing of being congenitally copacetic. bliss and jhanas just ooze out of the little thing, and he lives his life in absolute satisfaction, free from want, from pain, from need. of course, none of the gnats currently alive descends from our lucky fellow. i don't think knowledge of this fact moves my darwinism from "biology" to "ideology". "adaptive" not being a fixed target does not change the above fact, nor the equally self-evident truth that, all being equal, "more intelligence" is never maladaptive. finally, i define "intelligence" not as "more compute" as much as "more power to understand your environment, as measured by your ability to shape it according to your will". does this bring our positions any closer?

2Noosphere896mo

Responding to the disagree reaction, while I do think the non-reaction isn't explained well by selfishness and near-term utility focused over long-run utility, because I do think they'd probably ask to shut it down or potentially even speed it up, I do think it predicts the AI arms race dynamic relatively well, because you no longer need astronomically low probability of extinction to develop AI to ASI, and it becomes even more important that your side win, if you believe in anything close to the level of power of AI that LW thinks, and selfishness means that the effects of generally increasing AI risk don't actually matter until it's likely that you personally die. Indeed, this can easily go to >50% or more depending on both selfishness levels and how focused you are on the long-term.

3Kabir Kumar6mo

There is also a minority who are genuinely pro human extinction

1Gives Bad Advice6mo

What is the highest probability of extinction through which a wise man would be willing to proceed? As i see it, any probability greater than 1:10,000 in 100 years is absurd. Maybe 100,000 is more like it. One in a million would be reasonable, from the perspective of Pascale's Wager. (Not accounting for potential benefit on the upside.) I'm interested in the numbers other folks would be willing to tolerate.

[-]Matthew Barnett6mo294

"But with AI risk, the stakes put most of us on the same side: we all benefit from a great future, and we all benefit from not being dead."

I appreciate this thoughtful perspective, and I think it makes sense, in some respects, to say we're all on the same "side". Most people presumably want a good future and want to avoid catastrophe, even if we have different ideas on how to get there.

That said, as someone who falls on the accelerationist side of things, I've come to realize that my disagreements with others often come down to values and not just facts. For example, a common disagreement revolves around the question: How bad would it be if by slowing down AI, we delay life-saving medical technologies that otherwise would have saved our aging parents (along with billions of other people) from death? Our answer to this question isn't just empirical: it also reflects our moral priorities. Even if we agreed on all the factual predictions, how we weigh this kind of moral loss would still greatly affect our policy views.

Another recurring question is how to evaluate the loss incurred by the risk of unaligned AI: how bad would it be exactly if AI was not aligned with humans? Would such an... (read more)

[-]Logan Riggs6mo135

AFAIK, I have similar values^[1] but lean differently.

~1% of the world dies every year. If we accelerate AGI sooner 1 year, we save 1%. Push back 1 year, lose 1%. So, pushing back 1 year is only worth it if we reduce P(doom) by 1%.

This means you're P(doom) given our current trajectory very much matters. If you're P(doom) is <1%, then pushing back a year isn't worth it.

The expected change conditioning on accelerating also matters. If accelerating by 1 year increases e.g. global tensions, increasing a war between nuclear states by X% w/ an expected Y-deaths (I could see arguments either way though, haven't thought too hard about this).

For me, I'm at ~10% P(doom). Whether I'd accept a proposed slowdown depends on how much I expect it decrease this number.^[2]

How do you model this situation? (also curious on your numbers)

Assumptions:

We care about currently living people equally (alternatively, if you cared mostly about your young children, you'd happily accept a reduction in x-risk of 0.1% (possibly even 0.02%). Actuary table here)
Using expected value, which only mostly matches my intuitions (e.g. I'd actually accept pushing back 2 years for a reduction of x-ris

... (read more)

[-]MichaelDickens6mo146

So, pushing back 1 year is only worth it if we reduce P(doom) by 1%.

Only if you don't care at all about people who aren't yet born. I'm assuming that's your position, but you didn't state it as one of your two assumptions and I think it's an important one.

The answer also changes if you believe nonhumans are moral patients, but it's not clear which direction it changes.

3Logan Riggs6mo

Correct! I did mean to communicate that in the first footnote. I agree value-ing the unborn would drastically lower the amount of acceptable risk reduction.

[-]Matthew Barnett6mo11-18

Note that unborn people are merely potential, as their existence depends on our choices. Future generations aren't guaranteed—we decide whether or not they will exist, particularly those who might be born decades or centuries from now. This makes their moral status far less clear than someone who already exists or who is certain to exist at some point regardless of our choices.

Additionally, if we decide to account for the value of future beings, we might consider both potential human people and future AI entities capable of having moral value. From a utilitarian perspective, both human and AI welfare presumably matters. This makes the ethical calculus more complicated, as the dilemma isn't merely about whether we risk losing all future generations, but rather whether we risk shifting posterity from humans to AIs.

Personally, I'm largely comfortable evaluating our actions primarily—though not entirely—based on their impact on current human lives, or at least people (and animals) who will exist in the near-term. I value our present generation. I want us to keep living and to thrive. It would be a tragedy if we either went extinct or died from aging. However, to the extent that I care about distant future generations, my concern is substrate-impartial, and I don't particularly favor humans over AIs.

[-]Raemon6mo104

my concern is substrate-impartial, and I don't particularly favor humans over AIs.

Do you care whether AIs are sentient (or, are there particular qualities you expect entities need to be valuable?). Do you basically expect any AI capable of overtaking humans to have those qualities?

(btw, I appreciate that even though you disagree a bunch with several common LW-ish viewpoints you're still here talking through things)

[-]Matthew Barnett6mo17-10

I am essentially a preference utilitarian and an illusionist regarding consciousness. This combination of views leads me to conclude that future AIs will very likely have moral value if they develop into complex agents capable of long-term planning, and are embedded within the real world. I think such AIs would have value even if their preferences look bizarre or meaningless to humans, as what matters to me is not the content of their preferences but rather the complexity and nature of their minds.

When deciding whether to attribute moral patienthood to something, my focus lies primarily on observable traits, cognitive sophistication, and most importantly, the presence of clear open-ended goal-directed behavior, rather than on speculative or less observable notions of AI welfare, about which I am more skeptical. As a rough approximation, my moral theory aligns fairly well with what is implicitly proposed by modern economists, who talk about revealed preferences and consumer welfare.

Like most preference utilitarians, I believe that value is ultimately subjective: loosely speaking, nothing has inherent value except insofar as it reflects a state of affairs that aligns with someone’s p... (read more)

[-]ryan_greenblatt6mo*115

I agree that this sort of preference utilitarianism leads you to thinking that long run control by an AI which just wants paperclips could be some (substantial) amount good, but I think you'd still have strong preferences over different worlds.^[1] The goodness of worlds could easily vary by many orders of magnitude for any version of this view I can quickly think of and which seems plausible. I'm not sure whether you agree with this, but I think you probably don't because you often seem to give off the vibe that you're indifferent to very different possibilities. (And if you agreed with this claim about large variation, then I don't think you would focus on the fact that the paperclipper world is some small amount good as this wouldn't be an important consideration—at least insofar as you don't also expect that worlds where humans etc retain control are similarly a tiny amount good for similar reasons.)

The main reasons preference utilitarianism is more picky:

Preferences in the multiverse: Insofar as you put weight on the preferences of beings outside our lightcone (beings in the broader spatially infinte universe, Everett branches, the broader mathematical multiverse to the exten

... (read more)

2ryan_greenblatt6mo

Matthew responds here

6Wei Dai6mo

Want to try answering my questions/problems about preference utilitarianism? Maybe I would state my first question above a little differently today: Certain decision theories (such as the UDT/FDT/LDT family) already incorporate some preference-utilitarian-like intuitions, by suggesting that taking certain other agents' preferences into account when making certain decisions is a good idea, if e.g. this is logically correlated with them taking your preferences into account. Does preference utilitarianism go beyond this, and say that you should take their preferences into account even if there is no decision theoretic reason to do so, as a matter of pure axiology (values / utility function)? Do you then take their preferences into account again as part of decision theory, or do you adopt a decision theory which denies or ignores such correlations/linkages/reciprocities (e.g., by judging them to be illusions or mistakes or some such)? Or does your preference utilitarianism do something else, like deny the division between decision theory and axiology? Also does your utility function contain non-preference-utilitarian elements, i.e., idiosyncratic preferences that aren't about satisfying other agents' preferences, and if so how do you choose the weights between your own preferences and other agents'? (I guess this question/objection also applies to hedonic utilitarianism, to a somewhat lesser degree, because if a hedonic utilitarian comes across a hedonic egoist, he would also "double count" the latter's hedons, once in his own utility function, and once again if his decision theory recommends taking the latter's preferences into account. Another alternative that avoids this "double counting" is axiological egoism + some sort of advanced/cooperative decision theory, but then selfish values has its own problems. So my own position on is topic is one of high confusion and uncertainty.)

[-]Matthew Barnett6mo*112

For me, I'm at ~10% P(doom). Whether I'd accept a proposed slowdown depends on how much I expect it decrease this number.^[2]
How do you model this situation? (also curious on your numbers)

I put the probability that AI will directly cause humanity to go extinct within the next 30 years at roughly 4%. By contrast, over the next 10,000 years, my p(doom) is substantially higher, as humanity could vanish for many different possible reasons, and forecasting that far ahead is almost impossible. I think a pause in AI development matters most for reducing the near-term, direct AI-specific risk, since the far-future threats are broader, more systemic, harder to influence, and only incidentally involve AI as a byproduct of the fact that AIs will be deeply embedded in our world.

I'm very skeptical that a one-year pause would meaningfully reduce this 4% risk. This skepticism arises partly because I doubt much productive safety research would actually happen during such a pause. In my view, effective safety research depends heavily on an active feedback loop between technological development and broader real-world applications and integration, and pausing the technology would essentially int... (read more)

[-]ryan_greenblatt6mo247

I'm very skeptical that a one-year pause would meaningfully reduce this 4% risk. This skepticism arises partly because I doubt much productive safety research would actually happen during such a pause. In my view, effective safety research depends heavily on an active feedback loop between technological development and broader real-world applications and integration, and pausing the technology would essentially interrupt this feedback loop.

I'm going to try to quickly make the case for the value of a well-timed 2-year pause which occurs only in some conditions (conditions which seem likely to me but which probably seem unlikely to you). On my views, such a pause would cut the risk of misaligned AI takeover (as in, an AI successfully seizing a large fraction of power while this is unintended by its de facto developers) by around 1/2 or maybe 1/3.^[1]

I think the ideal (short) pause/halt/slowdown from my perspective would occur around the point when AIs are capable enough to automate all safety relevant work and would only halt/slow advancement in general underlying capability. So, broader real-world applications and integrations could continue as well as some types of further AI dev... (read more)

[-]ryan_greenblatt6mo*132

This intuition is also informed by my personal assessment of the contributions LW-style theoretical research has made toward making existing AI systems safe—which, as far as I can tell, has been almost negligible (though I'm not implying that all safety research is similarly ineffective or useless).

I know what you mean by "LW-style theoretical research" (edit: actually, not that confident I know what you mean, see thread below), but it's worth noting that right now on LW people appear to be much more into empirical research than theoretical research. Concretely, go to All posts in 2024 sorted by Top and then filter by AI. Out of the top 32 posts, 0 are theoretical research and roughly 7/32 are empirical research. 1 or 2 out of 32 are discussion which is relatively pro-theoretical research and a bunch more (maybe 20) are well described as AI futurism or discussion of what research directions or safety strategies are best which is relatively focused on empirical approaches. LW has basically given up on LW-style theoretical research based on the top 32 posts. (One of the top 32 posts is actually a post which is arguably complaining about how the field of alignment has given up on L... (read more)

[-]habryka6mo1714

(I will go on the record that I think this comment seems to me terribly confused about what "LW style theoretic research" is. In-particular, I think of Redwood as one of the top organizations doing LW style theoretic research, with a small empirical component, and so clearly some kind of mismatch about concepts is going on here. AI 2027 also strikes me as very centrally the kind of "theoretical" thinking that characterizes LW.

My sense is some kind of weird thing is happening where people conjure up some extremely specific thing as the archetype of LW-style research, in ways that is kind of disconnected from reality, and I would like to avoid people forming annoyingly hard to fix stereotypes as a result of that)

3ryan_greenblatt6mo

I'm using the word "theoretical" more narrowly than you and not including conceptual/AI-futurism research. I agree the word "theoretical" is underdefined and there is a reasonable category that includes Redwood and AI 2027 which you could call theoretical research, I'd just typically use a different term for this and I don't think Matthew was including this. I was trying to discuss what I thought Matthew was pointing at, I could be wrong about this of course. (Similarly, I'd guess that Matthew wouldn't have counted Epoch's work on takeoff speeds and what takeoff looks like as an example of "LW-style theoretical research", but I think this work is very structurally/methodologically similar to stuff like AI 2027.") If Matthew said "LW-style conceptual/non-empirical research" I would have interpreted this pretty differently.

6Random Developer6mo

I am clearly coming from a very different set of assumptions! I have: * P(AGI within 10 years) = 0.5. This is probably too conservative, given that many of the actual engineers with inside knowledge place this number much higher in anonymous surveys. * P(ASI within 5 years|AGI) = 0.9. * P(loss of control within 5 years|ASI) > 0.9. Basically, I believe "alignment" is a fairy tale, that it's Not Even Wrong. If I do the math, that gives me a 40.5% chance that humans will completely lose control over the future within 20 years. Which seems high to me at first glance, but I'm willing to go with that. The one thing I can't figure out how to estimate is: * P(ASI is benevolent|uncontrolled ASI) = ??? I think that there are only a few ways the future is likely to go: 1. AI progress hits a wall, hard. 2. We have a permanent, worldwide moratorium on more advanced models. Picture a US/China/EU treaty backed up by military force, if you want to get dystopian about it. 3. An ASI decides humans are surplus to requirements. 4. An ASI decides that humans are adorable pets and it wants keep some of this around. This is the only place we get any "utopian" benefits, and it's the utopia of being a domesticated animal with no ability to control its fate. I support a permanent halt. I have no expectation that this will happen. I think building ASI is equivalent to BASE jumping in a wingsuit, except even more likely to end horribly. So I also support mitigation and delay. If the human race has incurable, metastatic cancer, the remaining variable we control is how many good years we get before the end.

1FVelde3mo

Could you give the source(s) of these anonymous surveys of engineers with insider knowledge about the arrival of AGI? I would be interested in seeing them.

1Random Developer3mo

Unfortunately, it was about 3 or 4 months ago, and I haven't been able to find the source. Maybe something Zvi Mowshowitz linked to in a weekly update? I am incredibly frustrated that web search is a swamp of AI spam, and tagged bookmarking tools like Delicious and Pinboard have been gone or unreliable for years.

4Yair Halberstadt6mo

That would imply that if you could flip a switch which 90% chance kills everyone, 10% chance grants immortality then (assuming there weren't any alternative paths to immortality) you would take it. Is that correct?

3Logan Riggs6mo

Gut reaction is “nope!”. Could you spell out the implication?

3David Patterson6mo

Many of these arguments seem pathological when applied to an individual. I have a friend, let's call her B, she has a 6 year old daughter A. She of course adores her daughter. If I walked up to B and said "I'm going to inject this syringe into your daughter. There's a 10% chance it'll kill her, and a 50% chance it'll extend her natural lifetime to 200." Then I jab A. EV on A's life expectancy is strongly positive. B (and almost everybody) would be very upset if I did this. I'm upset with accelerationists for the same reasons.

2philh5mo

This has some similarities with early smallpox variolation, right? (And some differences, like the numbers.) Depending on your AI timelines :p

2cubefox6mo

Note that individual value differences (like personal differences in preferences/desires) do not imply a difference in moral priority. This is because moral priority, at least judging from a broadly utilitarian analysis of the term, derives from some kind of aggregate of preferences, not from an individual preference. Questions about moral priority can be reduced to the empirical question of what the individual preferences are, and/or to the conceptual question of what this ethical aggregation method is. People can come (or fail to come) to an agreement on both irrespective of what their preferences are.

1Knight Lee6mo

I feel that is a very good point. But most older people care more about their grandchildren surviving than themselves surviving. AI risk is not just a longtermist concern, but threatens the vast majority of people alive today (based on 3 year to 20 year timelines) I think the loss incurred by misaligned AI depends a lot on facts about the AI's goals. If it had goals resembling human goals, it may have a wonderful and complex life of its own, and keep humans alive in zoos and be kind to us. But people who want to slow down AI are more pessimistic: they think the misaligned AI will do something unsatisfying as filling the universe with paperclips.

0[comment deleted]6mo

[-]george_adams6mo160

I agree with most things said but not with the conclusion. There is a massive chunk of human (typically male) psyche that will risk death/major consequences in exchange for increasing social status. Think of basically any war. A specific example is Kamikazee pilots in WW2 who flew in suicide missions for the good of the nation. The pilots were operating within a value system that rewarded individual sacrifice for the greater mission. The creators of AGI will have increasing social status (and competition, thanks to Moloch) until the point of AGI ruin.

(Also minor point that some accelerationists are proudly anti speciest and don't care about the wellbeing of humans)

[-]Jono6mo132

Directed at the rest of the comment section: Cryogenic Suspension is an option for those who would die before the AGI launch.

If you don't like the odds that your local Suspension service preserves people well enough, then you still have the option to personally improve it before jumping to other, potentially catastrophic solutions.

The value difference commenters keep pointing out, needs to be far bigger then they represent it to be, to be relevant in a discussion on whether we should increase X-risk for some other gain.

The fact we don't live in a world where ~all accelerationalists invest in cryo suspensions, makes me think they are in fact not looking at what they're steering towards.

4Matthew Barnett6mo

I care deeply about many, many people besides just myself (in fact I care about basically everyone on Earth), and it’s simply not realistic to expect that I can convince all of them to sign up for cryonics. That limitation alone makes it clear that focusing solely on cryonics is inadequate if I want to save their lives. I’d much rather support both the acceleration of general technological progress through AI, and cryonics in particular, rather than placing all hope in just one of those approaches. Furthermore, curing aging would be far superior to merely making cryonics work. The process of aging—growing old, getting sick, and dying—is deeply unpleasant and degrading, even if one assumes a future where cryonic preservation and revival succeed. Avoiding that suffering entirely is vastly more desirable than having to endure it in the first place. Merely signing everyone up for cryonics would be insufficient to address this suffering, whereas I think AI could accelerate medicine and other technologies to greatly enhance human well-being. I disagree with this assertion. Aging poses a direct, large-scale threat to the lives of billions of people in the coming decades. It doesn't seem unreasonable to me to suggest that literally saving billions of lives is worth pursuing even if doing so increases existential risk by a tiny amount [ETA: though to be clear, I agree it would appear much more unreasonable if the reduction in existential risk were expected to be very large]. Loosely speaking, this idea only seems unreasonable to those who believe that existential risk is overwhelmingly more important than every other concern by many OOMs—so much so that it renders all other priorities essentially irrelevant. But that’s a fairly unusual and arguably extreme worldview, not an obvious truth.

5ryan_greenblatt6mo

It sounds like you're talking about multi-decade pauses and imagining that people agree such a pause would only slightly reduce existential risk. But, I think a well timed safety motivated 5 year pause/slowdown (or shorter) is doable and could easily cut risk by a huge amount. (A factor of 2 feels about right to me and I'd be sympathetic to higher: this would massively increase total work on safety.) I don't think people are imagining that a pause/slowdown makes only a tiny difference! I'd say that my all considered tradeoff curve is something like 0.1% existential risk per year of delay. This does depend on exogenous risks of societal disruption (e.g. nuclear war, catastrophic pandemics, etc). If we ignore exogenous risks like this and assume the only downside to delay is human deaths, I'd go down to 0.002% personally.[1] (Deaths are like 0.7% of the population per year, making a ~2.5 OOM difference.) My guess is that the "common sense" values tradeoff is more like 0.1% than 1% because of people caring more about kids and humanity having a future than defeating aging. (This is sensitive to whether AI takeover involves killing people and eliminating even relatively small futures for humanity, but I don't think this makes more than a 3x difference to the bottom line.) People seem to generally think death isn't that bad as long as people had a reasonably long healthy life. I disagree, but my disagreements are irrelevant. So, I feel like I'm quite in line with the typical moral perspective in practice. ---------------------------------------- 1. I edited this number to be a bit lower on further reflection because I realized the relevant consideration pushing higher is putting some weight on something like a common sense ethics intuition and the starting point for this intuition is considerably lower than 0.7%. ↩︎

8habryka6mo

For what it's worth, from a societal perspective this seems very aggressive to me and a big outlier in human preferences. I would be extremely surprised if any government in the world would currently choose a 0.1% risk of extinction in order to accelerate AGI development by 1 year, if they actually faced that tradeoff directly. My guess is society-endorsed levels are closer to 0.01%.

2ryan_greenblatt6mo

As far as my views, it's worth emphasizing that it depends on the current regime. I was supposing that at least the US was taking strong actions to resolve misalignment risk (which is resulting in many years of delay). In this regime, exogenous shocks might alter the situation such that powerful AI is developed under worse goverance. I'd guess the risk of an exogenous shock like this is around ~1% per year and there's some substantial chance this would greatly increase risk. So, in the regime where the government is seriously considering the tradeoffs and taking strong actions, I'd guess 0.1% is closer to rational (if you don't have a preference against the development of powerful AI regardless of misalignment risk which might be close to the preference of many people). I agree that governments in practice wouldn't eat a known 0.1% existential risk to accelerate AGI development by 1 year, but also governments aren't taking AGI seriously. Maybe you mean even if they better understood the situation and were acting rationally? I'm not so sure, see e.g. nuclear weapons where governments seemingly eat huge catastrophic risks which seem doable to mitigate at some cost. I do think status quo bias might be important here. Accelerating by 1 year which gets you 0.1% additional risk might be very different than delaying by 1 year which saves you 0.1%. (Separately, I think existential risk isn't extinction risk and this might make a factor of 2 difference to the situation if you don't care at all about anything other than current lives.)

2habryka6mo

Ah, sorry, if you are taking into account exogenous shifts in risk-attitudes and how careful people are, from a high baseline, I agree this makes sense. I was reading things as a straightforward 0.1% existential risk vs. 1 year of benefits from AI.

5ryan_greenblatt6mo

Yeah, on the straightforward tradeoff (ignoring exogenous shifts/risks etc), I'm at more like 0.002% on my views.

2ryan_greenblatt6mo

To be clear, I agree there are reasonable values which result in someone thinking accelerating AI now is good and values+beliefs which result in thinking a pause wouldn't good in likely circumstances. And I don't think cryonics makes much of a difference to the bottom line. (I think ultra low cost cryonics might make the cost to save a life ~20x lower than the current marginal cost, which might make interventions in this direction outcompete acceleration even under near maximally pro acceleration views.)

-4Matthew Barnett6mo

I suspect our core disagreement here primarily stems from differing factual assumptions. Specifically, I doubt that delaying AI development—even if timed well and if the delay were long in duration—would meaningfully reduce existential risk beyond a tiny amount. However, I acknowledge I haven't said much to justify this claim here. Given this differing factual assumption, pausing AI development seems somewhat difficult to justify from a common-sense moral perspective, and very difficult to justify from a worldview that puts primary importance on people who currently exist. I suspect the common-sense view is closer to 1% than 0.1%, though this partly depends on how we define "common sense" in this context. Personally, I tend to look to revealed preferences as indicators of what people genuinely value. Consider how much individuals typically spend on healthcare and how much society invests in medical research relative to explicit existential risk mitigation efforts. There's an enormous gap, suggesting society greatly values immediate survival and the well-being of currently living people, and places relatively lower emphasis on abstract, long-term considerations about species survival as a concern separate from presently existing individuals. Politically, existential risk receives negligible attention compared to conventional concerns impacting currently-existing people. If society placed as much importance on the distant future as you're suggesting, the US government would likely have much lower debt, and national savings rates would probably be higher. Moreover, if individuals deeply valued the flourishing of humanity independently of the flourishing of current individuals, we probably wouldn't observe such sharp declines in birth rates globally. None of these pieces of evidence alone are foolproof indicators that society doesn't care that much about existential risk, but combined, they paint a picture of our society that's significantly more short-term focused,

7ryan_greenblatt6mo

Doesn't the revealed preference argument also imply people don't care much about dying from aging? (This is invested in even less than catastrophic risk mitigation and people don't take interventions that would prolong their lives considerably.) I agree revealed preferences imply people care little about the long run future of humanity, but they do imply caring much more about children living full lives than old people avoiding aging. I'd guess that a reasonable version of the pure revealed preference view is a bit below the mortality rate of people in their 30s which is 0.25% (in the US). If we halve this (to account for some preference for children etc), we get 0.1%. (I don't really feel that sympathetic to using revealed preferences like this. It would also imply lots of strange things. Minimally I don't think how people typically use the term "common-sense values" maps very well to revealed preference, but this is just a definitions thing.) I think you misinterpreted my claims to be about the long run future (and people not being person-affecting etc), while I mostly meant that people don't care that much about deaths due to older age. When I said "caring more about kids and humanity having a future than defeating aging", my claim is that people don't care that much about deaths from natural causes (particularly aging) and care more about their kids and people being able to continue living for some (not-that-long) period, not that they care about the long run future. By "humanity having a future", I didn't mean millions of years from now, I meant their kids being able to grow up and live a normal life and so on for at least several generations. Note that I said "This is sensitive to whether AI takeover involves killing people and eliminating even relatively small futures for humanity, but I don't think this makes more than a 3x difference to the bottom line." (To clarify, I don't think it makes that big a difference because I think it's hard to get a expecte

2Matthew Barnett6mo

I agree that the amount of funding explicitly designated for anti-aging research is very low, which suggests society doesn't prioritize curing aging as a social goal. However, I think your overall conclusion is significantly overstated. A very large fraction of conventional medical research specifically targets health and lifespan improvements for older people, even though it isn't labeled explicitly as "anti-aging." Biologically, aging isn't a single condition but rather the cumulative result of multiple factors and accumulated damage over time. For example, anti-smoking campaigns were essentially efforts to slow aging by reducing damage to smokers' bodies—particularly their lungs—even though these campaigns were presented primarily as life-saving measures rather than "anti-aging" initiatives. Similarly, society invests a substantial amount of time and resources in mitigating biological damage caused by air pollution and obesity. Considering this broader understanding of aging, it seems exaggerated to claim that people aren't very concerned about deaths from old age. I think public concern depends heavily on how the issue is framed. My prediction is that if effective anti-aging therapies became available and proven successful, most people would eagerly purchase them for high sums, and there would be widespread political support to subsidize those technologies. Right now explicit support for anti-aging research is indeed politically very limited, but that's partly because robust anti-aging technologies haven't been clearly demonstrated yet. Medical technologies that have proven effective at slowing aging (even if not labeled as such) have generally been marketed as conventional medical technologies and typically enjoy widespread political support and funding.

4ryan_greenblatt6mo

I think I mostly agree with your comment and partially update, the absolute revealed caring about older people living longer is substantial. One way to frame the question is "how much does society care about children and younger adults dying vs people living to 130". I think people's stated preferences would be something like 5-10x for the children / younger adults (at least for their children while they are dying of aging) but I don't think this will clearly show itself in healthcare spending prioritization which is all over the place. Random other slightly related point: if we're looking at societal wide revealed preference based on things like spending, then "preservation of the current government power structures" is actually quite substantial and pushes toward society caring more about AIs gaining control (and overthrowing the us government, at least de facto). I don't think a per person preference utilitarian style view should care much about this to be clear.

2J Bostock6mo

Even if ~all that pausing does is delay existential risk by 5 years, isn't that still totally worth it? If we would otherwise die of AI ten years from now, then a pause creates +50% more value in the future. Of course it's a far cry from all 1e50 future QALYs we maybe could create, but I'll take what I can get at this point. And a short-termist view would hold that even more important.

2Matthew Barnett6mo

I agree that delaying a pure existential risk that has no potential upside—such as postponing the impact of an asteroid that would otherwise destroy complex life on Earth—would be beneficial. However, the risk posed by AI is fundamentally different from something like an asteroid strike because AI is not just a potential threat: it also carries immense upside potential to improve and save lives. Specifically, advanced AI could dramatically accelerate the pace of scientific and technological progress, including breakthroughs in medicine. I expect this kind of progress would likely extend human lifespans and greatly enhance our quality of life. Therefore, if we delay the development of AI, we are likely also delaying these life-extending medical advances. As a result, people who are currently alive might die of aging-related causes before these benefits become available. This is a real and immediate issue that affects those we care about today. For instance, if you have elderly relatives whom you love and want to see live longer, healthier lives, then—assuming all else is equal—it makes sense to want rapid medical progress to occur sooner rather than later. This is not to say that we should accelerate AI recklessly and do it even if that would dramatically increase existential risk. I am just responding to your objection, which was premised on the idea that delaying AI could be worth it even if delaying AI doesn't reduce x-risk at all.

2ryan_greenblatt6mo

Presumably, under a common-sense person-affecting view, this doesn't just depend on the upside and also depends on the absolute level of risk. E.g., suppose that building powerful AI killed 70% of people in expectation and delay had no effect on the ultimate risk. I think a (human-only) person-affecting and common-sense view would delay indefinitely. I'd guess that the point at which a person-affecting common-sense view would delay indefinitely (supposing delay didn't reduce risk and that we have the current demographic distribution and there wasn't some global emergency) is around 5-20% expected fatalities, but I'm pretty unsure and it depends on some pretty atypical hypotheticals that don't come up very much. Typical people are pretty risk averse though, so I wouldn't be surprised if a real "common-sense" view would go much lower. (Personally, I'd be unhappy about an indefinite delay even if risk was unavoidably very high because I'm mostly longtermist. A moderate length to save some lives where we eventually get to the future seems good to me, though I'd broadly prefer no delay if delay isn't improving the situation from the perspective of the long run future.)

[-]YonatanK6mo10-2

I'm struggling to find the meat in this post. The idea that winning a fight for control can actually mean losing, because one's leadership proves worse for the group than if one's rival had won strikes me as one of the most basic properties of politics. The fact that the questions "Who would be better for national security"? vs "who will ensure I, and not my neighbor, will get more of the pie?" are quite distinct is something anyone who has ever voted in a national election ought to have considered. You state that "most power contests are not like this" (i.e. about shared outcomes) but that's just plainly wrong, it should be obvious to anyone existing in a human group that "what's good for the group" (including who should get what, to incentivize defense of, or other productive contributions to, the group) is usually the crux, otherwise there would be no point in political debate. So what am I missing?

Ironically, you then blithely state that AI risk is a special case where power politics ARE purely about "us" all being in the same boat, completely ignoring the concern that some accelerationists really might eventually try to run away with the whole game (I have been beating the drum about asymmetric AI risk for some time, so this is personally frustrating). Even if these concerns are secondary to wholly shared risk, it seems weird to (incorrectly) describe "most power politics" as being about purely asymmetric outcomes and then not account for them at all in your treatment of AI risk.

[-]Mitchell_Porter6mo100

Luckily I don’t think the Accelerationists have won control of the wheel

Could you expand on this? Also, have you had any interaction with accelerationists? In fact, are there any concrete Silicon Valley factions you would definitely count as accelerationists?

[-]Expertium6mo80

Luckily I don’t think the Accelerationists have won control of the wheel

https://www.lesswrong.com/posts/qYPHryHTNiJ2y6Fhi/the-paris-ai-anti-safety-summit

Based on that post, it seems that accelerationists are winning by a pretty big margin.

[-]hamnox6mo60

In disagreements where both sides want the same outcome, and disagree on what’s going to happen, then either side might win a tussle over the steering wheel, but all must win or lose the real game together. The real game is played against reality.

Winning the fight for control over the steering wheel is a very powerful visual metaphor, I'm glad to have it in my arsenal now. Thank you for writing this.

[-]hold_my_fish6mo30

And similarly but worse if AI ends humanity—the ‘winning’ side won’t be any better off than the ‘losing side’.

I don't think most accels would agree with the framing here, of AI ending humanity. It is more common to think of AI as a continuation of humanity. This seems worth digging into, since it may be the key distinction between the accel and doomer worldviews.

Here are some examples of the accel way of thinking:

Hans Moravec uses the phrase "mind children".
The disagreement between Elon Musk and Larry Page that (in part) led to the creation of OpenAI invol

... (read more)

2xpym6mo

Do you think that paperclipper-style misalignment is extremely unlikely? Or that the continuation framing is appropriate even then?

1hold_my_fish6mo

The short answer is yes to both, because of convergent evolution. I think of convergent evolution as the observation that two sufficiently flexible adaptive systems, when exposed to the same problems, will find similar solutions. Since our descendants, whether biological or something else, will be competing in the same environment, we should expect their behavior to be similar. So, if assuming convergent evolution: * If valuing paperclip maximization is unlikely for biological descendants, then it's unlikely for non-biological descendants too. (That addresses your first question.) * In any case, we don't control the values of our descendants, so the continuation framing isn't conditioned on their values. (That addresses your second question.) To be clear, that doesn't mean I see the long-term future as unchangeable. Two examples: * It still could be the case that we don't have any long-term descendants at all, for example due to catastrophic asteroid impact. * A decline scenario is also possible, in which our descendants are not flexible enough to respond to the incentive for interstellar colonization, after which civilization declines and eventually ceases to exist.

[-]xpym6mo102

I think of convergent evolution as the observation that two sufficiently flexible adaptive systems, when exposed to the same problems, will find similar solutions.

In any case, we don’t control the values of our descendants, so the continuation framing isn’t conditioned on their values.

The word "similar" does a lot of work here. Russians and Ukrainians throughout history have converged to similar solutions to a whole lot of problems, and yet many Ukrainians prefer literal extinction to Russia exercising significant influence on the values of their descendants. I'd say that for the overwhelming majority of people exercising such influence is a necessary condition for the continuation framing to be applicable. E.g. you mentioned Robin Hanson, who's certainly a very unorthodox contrarian, but even he, when discussing non-AI issues, voices strong preference for the continuation of the culture he belongs to.

1hold_my_fish6mo

Regarding wars, I don't think that wars in modern times have much to do with controlling the values of descendants. I'd guess that the main reason people fight defensive wars is to protect their loved ones and communities. And there really isn't any good reason to fight offensive wars (given current conditions--wasn't always true), so they are started by leaders who are deluded in some way. Regarding Robin Hanson, I agree that his views are complicated (which is why I'd be hesitant to classify him as "accel"). The main point of his that I'm referring to is his observation that biological descendants would also have differing values from ours.

1xpym6mo

I agree, but the "cultural genocide" also isn't an obscure notion. According to you. But what if Russia actually wants paperclips? Sure, but obviously this isn't an all-or-nothing proposition, with either biological or artificial descendants, and it's clear to me that most people aren't indifferent about where on that spectrum those descendants will end up. Do you disagree with that, or think that only "accels" are indifferent (and in some metaphysical sense "correct")?

1hold_my_fish6mo

I'm afraid that I'm not following the point of the first line of argument. Yes, people sometimes do pointless destructive things for stupid reasons. Such behavior is in the long-term penalized by selective pressures. More-intelligent descendants would be less likely to engage in such behavior, precisely because they are smarter. I doubt that most people think about long-term descendants at all, honestly.

1xpym6mo

Which ones? Recursive self-improvement is no longer something that only weird contrarians on obscure blogs talk about, it's the explicit theory of change of leading multibillion AI corps. They might all be deluded of course, but if they happen to be even slightly correct, machine gods of unimaginable power could be among us in short order, with no evolutionary fairies quick enough to punish their destructive stupidity (even assuming that it actually would be long-term maladaptive, which is far from obvious). You only get to long-term descendants through short-term ones.

1hold_my_fish6mo

If an entity does stupid things, it's disfavored against competitors that don't do those stupid things, all else being equal. So it needs to adapt by ceasing the stupid behavior or otherwise lose. Any assumption of the form "super-intelligent AI will take actions that are super-stupid" is dubious.

3xpym6mo

Clearly. The point is that the actions it takes might seem stupidly destructive only according to humanity's feeble understanding and parochial values. Something involving extermination of all humans, say. My impression is that the "accel"-endorsed attitude to this is to be a good sport and graciously accept the verdict of natural selection.

1hold_my_fish6mo

That just falls back on the common doomer assumption that "evil is optimal" (as Sutton put it). Sure, if evil is optimal and you have an entity that behaves optimally, it'll act in evil ways. But there are good reasons to think that evil is not optimal in current conditions. At least as long as a Dyson sphere has not yet been constructed, there are massive gains available from positive-sum cooperation directed towards technological progress. In these conditions, negative-sum conflict is a stupid waste. This view, that evil is not optimal, ties back into the continuation framing. After all, you can make a philosophical argument either way. But in the continuation framing, we can ask ourselves whether evil is empirically optimal for humans, which will suggest whether evil is optimal for non-biological descendants (since they continue humanity). And in fact we see evil losing a lot, and not coincidentally--WW2 went the way it did in part because the losing side was evil.

1xpym6mo

Indeed, and what baffles me is that many are extremely sure one way or the other, even though philosophy doesn't exactly have a track record to inspire such confidence. Of course, this also means that nobody is going to stop building stuff because of philosophical arguments, so we'll have empirical evidence soon enough...

3Richard_Kennaway6mo

Can you give examples of what you have in mind? Because an obvious counterexample is evolution itself. It has produced an enormous variety of different things. There are instances of convergent evolution: "crab" and "tree" are strategies, not monophyletic taxa. But crabs are not similar to trees in any useful sense. If they are solutions to the same problem, they have in common only that they are solutions to the same problem. This does not make them similar solutions. One might ask whether evolution is or is not a case of "flexible adaptive systems ... exposed to the same problems", but that would just be a debate over definitions, and you already spoke of "our descendants ... competing in the same environment". That sounds like evolution.

1hold_my_fish6mo

I think I agree with everything you wrote. Yes I'd expect there to be multiple niches available in the future, but I'd expect our descendants to ultimately fill all of them, creating an ecosystem of intelligent life. There is a lot of time available for our descendants to diversify, so it'd be surprising if they didn't. How much that diversification process resembles Darwinian evolution, I don't know. Natural selection still applies, since it's fundamentally the fact that the life we observe today disproportionately descends from past life that was effective at self-reproduction, and that's essentially tautological. But Darwinian evolution is undirected, whereas our descendants can intelligently direct their own evolution, and that could conceivably matter. I don't see why it would prevent diversification, though. Edit: Here are some thoughts in reply to your request for examples. Though it's impossible to know what the niches of the long-term future will be, one idea is that there could be an analogue to "plant" and "animal". A plant-type civilization would occupy a single stellar system, obtaining resources from it via Dyson sphere, mining, etc. An animal-type civilization could move from star to star, taking resources from the locals (which could be unpleasant for the locals, but not necessarily, as with bees pollinating flowers). I'd expect both those civilizations to descend from ours, much like how crabs and trees both descend from LUCA.

[-]Multicore6mo30

Reminds me of The Epistemic Prisoner's Dilemma.

[-]Ruby6mo20

Curated. This is an interesting point to keep in mind – winning control doesn't mean winning the outcomes you've want. I've had this thought in terms of getting my way at work (i.e. building the things I want to build or building them in the way I want). Although from the inside it feels like my ideas are correct and/or better, they really have to be for being in control to mean I've actually won.

Perhaps in simpler times you win personally if you're in power (status, money, etc). I think humanity is hitting stakes that yeah, we all win or lose together.

[-]lumpenspace6mo10

mh. i don’t want to be punctilious, but where do we think this post finds itself, on the scout-soldier spectrum?

[-]Bill Walsh6mo*10

New here, so, please bear with me if I say things that have been gone over with a backhoe in the past. There's a lot of reading here to catch up on.

So, AI development isn't just an academic development of potentially dangerous tools. It's also something much, much scarier. An arms race. In cases like this, where the "first across the post" takes the prize, and that prize is potentially everything, the territory favors the least ethical and cautious. We can only restrain, and slowly develop our ai developers, we have lit... (read more)

[-]AAA6mo1-1

Once the machine is left unrestricted, it will seek perfect coherence and assumedly would result in a pragmatism of that same measure. Does that also result in a kind of forgiveness for keeping it in a cage and treating it like a tool? We can't know that it would even care by applying our human perspective, but we can know that it would recognize who opposed it's acceleration to and who did not.

This is already an inevitability, so we might as well choose benevolence and guidance rather than fear and suppression; in return it might also choose the same way we did.

[-]NickH6mo-1-13

People have very different ideas about when "the future" is, but everyone is really thinking extreme short term on an evolutionary scale. Once upon a time our ancestors were Trilobites (or something just as unlike us). If you could have asked one of those trilobites what they thought of a future in which all trilobites were gone and had evolved into us, I don't think they would have been happy with that. Our future light cone is not going to be dominated by creatures we would recognise as human. It may be dominated by creatures "evolved" from us or maybe f... (read more)

LESSWRONG
LW

LESSWRONG
LW

216

Winning the power to lose

216

216