All of superads91's Comments + Replies

True, in a way. Without solving said mystery (of how an animal brain produces not only calculations but also experiences) you could theoretically create philosophical zombie uploads. But in this post what is really desired is to save all conscious beings from death and disease by uploading them, so to that effect (the most important one) it still looks impossible.

(I deleted my post because on hindsight it sounded a bit off topic.)

I never linked complexity to absolute certainty of something being sentient or not, only to pretty good likelihood. The complexity of any known calculation+experience machine (most animals, from insect above) is undeniably way more than that of any current Turing machine. Therefore it's reasonable to assume that consciousness demands a lot of complexity, certainly much more than that of a current language model. To generate experience is fundamentally different than to generate only calculations. Yes, this is an opinion, not a fact. But so is your claim!

I ... (read more)

2green_leaf2y
That's not correct to do either, for the same reason. Also, I wasn't going to mention it before (because the reasoning itself is flawed), but there is no correct way of calculating complexity that would make the complexity of an insect brain higher than LaMDA.

"There is no reason to think architecture is relevant to sentience, and many philosophical reasons to think it's not (much like pain receptors aren't necessary to feel pain, etc.)."

That's just non-sense. A machine that makes only calculations, like a pocket calculator, is fundamentally different in arquitecture from one that does calculations and generates experiences. All sentient machines that we know have the same basic arquitecture. All non-sentient calculation machines also have the same basic arquitecture. The likelihood that sentience will arise in ... (read more)

3green_leaf2y
This is wrong. A simulation of a conscious mind is itself conscious, regardless of the architecture it runs on (a classical computer, etc.). That was a sarcastic paragraph to apply the same reasoning to meat brains to show it can be just as well argued that only language models are conscious (and meat brains aren't, because their architecture is so different). Complexity itself is unconnected to consciousness. Just because brains are conscious and also complex doesn't mean that a system needs to be as complex as a brain to be conscious, any more than the brain being wet and also conscious means that a system needs to be as wet as a brain to be conscious. You're committing the mistake of not understanding sentience, and using proxies (like complexity) in your reasoning, which might work sometimes, but it doesn't work in this case.

There's just no good reason to assume that LaMDA is sentient. Arquitecture is everything, and its arquitecture is just the same as other similar models: it predicts the most likely next word (if I recall correctly). Being sentient involves way more complexity than that, even something as simple as an insect. It claiming that it is sentient might just be that it was mischievously programmed that way, or it just found it was the most likely succession of words. I've seen other language models and chatbots claim they were sentient too, though perhaps ironical... (read more)

3green_leaf2y
There is no reason to think architecture is relevant to sentience, and many philosophical reasons to think it's not (much like pain receptors aren't necessary to feel pain, etc.). The sentience is in the input/output pattern, independently of the specific insides. On one level of abstraction, LaMDA might be looking for the next most likely word. On another level of abstraction, it simulates a possibly-Turing-test-passing person that's best at continuing the prompt. The analogy would be to say about human brain that all it does is to transform input electrical impulses to output electricity according to neuron-specific rules. (Unlike complex Turing-test-passing language models who, unlike slow neural networks based in meat, can think much faster and in many ways deeper than a meat brain, which, if we look inside, contains no sentience. Of course, the brain claims to be sentient, but that's only because of how its neurons are connected. It's really an exercise in how a well-manufactured piece of meat can fool even an intelligent language model.)

Right then, but my original claim still stands: your main point is, in fact, that it is hard to destroy the world. Like I've explained, this doesn't make any sense (hacking into nuclear codes). If we create an AI better than us at code, I don't have any doubts that it CAN easily do it, if it WANTS. My only doubt is whether it will want it or not. Not whether it will be capable, because like I said, even a very good human hacker in the future could be capable.

At least the type of AGI that I fear is one capable of Recursive Self-Improvement, which will unavo... (read more)

The post is clearly saying "it will take longer than days/weeks/months SO THAT we will likely have time to react". Both are highly unlikely. It wouldn't take a proper AGI weeks or months to hack into the nuclear codes of a big power, it would take days or even hours. That gives us no time to react. But the question here isn't even about time. It's about something MORE intelligent than us which WILL overpower us if it wants, be it on 1st or 100th try (nothing guarantees we can turn it off after the first failed strike).

Am I extremely sure that an unaligned ... (read more)

1mukashi2y
Am I extremely sure that an unaligned AGI would cause doom? If that's the case, we already agree and I have nothing to add. We might disagree in the relative likelihood but that's ok. I do agree that is a risk and we should take the matter extremely seriously

Your argument boils down to "destroying the world isn't easy". Do you seriously believe this? All it takes is to hack into the codes of one single big nuclear power, thereby triggering mutually assured destruction, thereby triggering nuclear winter and effectively killing us all with radiation over time.

In fact you don't need AGI to destroy the world. You only need a really good hacker, or a really bad president. In fact we've been close about a dozen times, so I hear. If Stanislav Petrov had listened to the computer in 1983 who indicated 100% probability ... (read more)

1Dagon2y
I think it's a bad title for the post.  It shouldn't be "I don't believe in doom", but "I don't believe in the foom path to doom".  Most of the argument is that it'll take longer than is often talked about, not that it won't happen (although the poster does make some claims that slower is less likely to succeed). The post doesn't mention all the other ways we could be doomed, with or without AGI.

"You may notice that the whole argument is based on "it might be impossible". I agree that it can be the case. But I don't see how it's more likely than "it might be possible"."

I never said anything to the contrary. Are we allowed to discuss things that we're not sure whether it "might be possible" or not? It seems that you're against this.

Tomorrow people matter, in terms of leaving them a place in minimally decent conditions. That's why when you die for a cause, you're also dying so that tomorrow people can die less and suffer less. But in fact you're not dying for unborn people - you're dying for living ones from the future.

But to die to make room for others is simply to die for unborn people. Because them never being born is no tragedy - they never existed, so they never missed anything. But living people actually dying is a tragedy.

And I'm not against the fact that giving live is a great... (read more)

I can see the altruism in dying for a cause. But it's a leap of faith to claim, from there, that there's altruism in dying by itself. To die why, to make room for others to get born? Unborn beings don't exist, they are not moral patients. It would be perfectly fine if no one else was born from now on - in fact it would be better than even 1 single person dying.

Furthermore, if we're trying to create a technological mature society capable of discovering immortality, perhaps much sooner will it be capable of colonizing other planets. So there are trillions of... (read more)

1Jes2y
Well, okay, but why? Why don't tomorrow people matter at all? Is there a real moral normativity that dictates this, or are we just saying our feelings to each other? I don't mean that condescendingly, just trying to understand where you're coming from when you make this claim. But I'm arguing for something different from altruism. I go further by saying that the approach to constructing a sense of self differs substantively between people, cultures, etc. Someone who dies for their nation might not be altruistic per se, if they have located their identity primarily in the nation. In other words, they are being selfish, not as their person, but as their nation. Does that make sense? Granted, your point about interstellar travel makes all of this irrelevant. But I'm much more cynical about humanity's future. Or at least, the future of the humans I actually see around me. Technology here is so behind. Growing your own food as a rational way to supplement income is common, education ends for most people at age 12, the vast majority don't have hot water, AC, etc. Smartphones are ubiquitous though. Immortal lords from Facebook deciding how many rations of maize I'll receive for the upvotes I earned today. Like, of course the Facebook lord will think he's building Utopia. But from here, will it look much better than the utopia that the church and aristocracy collaborated to build in medieval Europe? I don't look to the future with hope as often as I look to the past with envy. Though I do both, from time to time.

To each paragraph:

  1. Totally unfair comparison. Do you really think that immortality and utopia are frivolous goals? So maybe you don't really believe in cryonics or something. Well, I don't either. But transhumanism is way more than that. I think that its goals with AI and life extension are all but a joke.

  2. That's reductive. As an altruist, I care about all other conscious being. Of course maintaining sanity demands some distancing, but that's that. So I'd say I'm a collectivist. But one person doesn't substitute the other. Others continuing to live wil

... (read more)
1Jes2y
Individualism and altruism aren't exclusive. I didn't mean to imply you are selfish, just that your operating definition of self seems informed by a particular tradition. Consider the perspective of liberal republicans of the 19th century who fought and died for their nation (because that's where they decided, or were taught, to center their self). Each nation is completely unique and irreplaceable, so we must fight to keep nations thriving and alive, and prevent their extinction. Dying for patriotism is glorious, honorable, etc. But that's my point, consciousness will go on just fine without either of us specifically being here. Ending one conscious experience from one body so that a different one can happen seems fine to me, for the most part. I dunno the philosophical implications of this, just thinking. Yeah, it's exciting for sure.  I'm 30 as well, so I'll be near death in the decades that likely begin to birth AGI. But it would likely be able to fathom things unfathomable to us, who knows. History beyond that point is a black hole for me. It's all basilisks and space jam past 2050 as far as I'm concerned :) Well, I guess that's it, huh? I don't think so, but clearly a lot of people do. Btw I'm new to this community, so sorry if I'm uninformed on issues that are well hashed out here. What a fun place, though.

"You are a clone of your dead childhood self."

Yes, that's a typical Buddhist-like statement, that we die and are reborn each instant. But I think it's just incorrect - my childhood self never died. He's alive right now, here. When I die the biological death, then I will stop existing. It's as simple as that. Yet I feel like Buddhists, or Eastern religion in general, does this and other mental gymnastics to comfort people.

"So you either stick with modernism (that transhumanism is the one, special ideology immune from humanity's tragic need to self-sedate), ... (read more)

1Jes2y
I don't think there is anything particularly scientific about transhumanism relative to other ideologies. They use science to achieve their goals, much like Catholics use science to prove that fetuses have heart beats or whatever.  Really, this debate feels like it boils down to an individualistic vs collectivistic sense of self. In the collectivist view, dying is not that big of a deal. You can see this in action, when dying for your family, country, etc is seen as noble and great. Whereas an individual sacrificing their family to preserve themselves is less lauded (except in Individualist propaganda, where we're scolded for "judging" and supposed to "understand" the individual circumstances and so on). I mean, you yourself say it, we have no idea what consciousness even is. Or if it's valuable at all. We're just going on a bunch of arbitrary intuitions here. Well, that's a bad standard. And it's not like we're running out of people, we can just create them as needed, indefinitely. So given that 1. we have a lot of humans, they aren't a particularly limited resource, and 2. few if any of the people have super unique, amazing perspectives, such that we really need to preserve that one person for extra time Why not focus our energy on figuring out what we are, and decide the best course of action from there?  It's only cruel if you've been brainwashed into thinking your life matters. If you never thought that, it's just normal. Accept your place and die so that our descendants (who we should work really hard to help so they turn out better than we are, and thus deserving of immortality should we ever actually attain it). But then, if we've figured out how to make such amazing people, why not let there be lots of different ones, so they can flourish across many generations, instead of having just one generation forever? I mean, there isn't even that much to do. Well, there probably is, I'm just too basic to understand it because of my limited natural brain. I

Still, that could all happen with philosophical zombies. A computer agent (AI) doesn't sleep and can function forever. These 2 factors is what leads me to believe that computers, as we currently define them, won't ever be alive, even if they ever come to emulate the world perfectly. At best they'll produce p-zombies.

"I'm feeling enthusiastic to try to make it work out, instead of being afraid that it won't."

Well, for someone who's accusing me of emotionally still defending a wrong mainstream norm (deathism), you're also doing it yourself by espousing empty positivism. Is it honest to feel enthusiastic about something when your probabilities are grim? The probabilities should come first, not how you feel about it.

"It's true that I lack the gear-level model explainig how it's possible for me to exist for quadrillion years."

Well I do have one to prove the opposite: the b... (read more)

1Ape in the coat2y
I wasn't really planning it as an accusation. It was supposed to be a potentially helpful hint at the source of your cognitive dissonance. Sorry, it seems that I failed to convey it properly.  Previously you mentioned being scared due to imagining to live for a quadrillion years. I thought it would be appropriate to share my own emotional reaction as well. I agree that probabilities should go first. And that's the thing I do not see them as grim. For me it's more or less 50-50. I'm not competent enough regarding future scientific discoveries and true laws of nature to shift them from this baseline. And I doubt anyone of currently living really is. That's why I'm surprised by your pessimism. You may notice that the whole argument is based on "it might be impossible". I agree that it can be the case. But I don't see how it's more likely than "it might be possible".

It does ring true to me a bit. How could it not, when one cannot imagine a way to exist forever with sanity? Have you ever stopped to imagine, just relying on your intuition, what would be like to live for a quadrillion years? I'm not talking about a cute few thousand like most people imagine when we talk about immortality. I'm talking about proper gazillions, so to speak. Doesn't it scare the sh*t out of you? Just like Valentine says in his comment, it's curious how very few transhumanists have ever stopped to stare at this abyss.

On the other hand I don't... (read more)

1Ape in the coat2y
My intuition doesn't differ much whether it's a thousand or a quadrillion years. I'm feeling enthusiastic to try to make it work out, instead of being afraid that it won't. It's true that I lack the gear-level model explainig how it's possible for me to exist for quadrillion years. But neither do I have a gear-level model, explaining how it's impossible. I know that I still have some confusion about consciousness and identity, but this doesn't allow me shift the probability either way. For every argument "what if it's impossible to do x and x is required to exist for quadrillion years" I can automatically construct counter arguments like "what if it's actually possible to do x" or "what if x is not required". How do you manage to get 70-80% confidence level here? This sounds overconfident to me.

"Whatever posttranshuman creature inherits the ghost of your body in a thousand years won't be "you" in any sense beyond the pettiest interpretation of ego as "continuous memory""

I used to buy into that Buddhist perspective, but I no longer do. I think that's a sedative, like all religions. Though I will admit that I still meditate, because I still hope to find out that I'm wrong. I hope I do, but I don't have a lot of hope. My reason and intuition are clear in telling me that the self is extremely valuable, both mine and that of all other conscious beings... (read more)

0Jes2y
>Each conscious being is irreplaceable. Right, that's my point - the conscious being of your childhood is not replaceable with you now. You are a clone of your dead childhood self. That's fine for you, the clone. But who's interested in getting a 30-year-old clone? And the many consciousnesses that flower and die every few decades, will be replaced with the single continuation of a single generation that stumbles into immortality. >I think that's a sedative, like all religions I'm not Buddhist, but your critique extends to yourself. If you take one step back to look at an even broader picture, by replacing religion with ideology, you've just reinvented postmodernism. Viz: "Transhumanism, sedative, like all ideologies." So you either stick with modernism (that transhumanism is the one, special ideology immune from humanity's tragic need to self-sedate), or dive into the void (which really is just an ocean of sedatives swirling together, I've swum deep and I promise it's ideology all the way down). Maybe we're not self-sedating at all, and we can talk to a pharmacist who isn't just us. It's hard to say anything about reality when the only thing you know is that you're high af all the time. >I've been alive for 30 years - not much, I admit, but I still feel as much like me as in the first day that I can remember. Every day the same sun rises, yet it's a different day. You aren't the sun, you're the day. Imagine droplets of water trapped in a cup, then poured back into the ocean. Water is consciousness, your mind is the cup.  Each century, the day ends, but your family continues. Every few centuries your family falls apart, but your community thrives. Each millenium your community is ravaged, but your nation lives.  We could go on, to the species, the community of all conscious beings, etc etc. Where you place the self along this line is historically variable. You place it with the continuity of the ego. I place it with my family. There are a zillion answers to

"if one might make a conscious being out of Silicon but not out of a Turing machine"

I also doubt that btw.

"what happens when you run the laws of physics on a Turing machine and have simulated humans arise"

Is physics computable? That's an open question.

And more importantly, there's no guarantee that the laws of physics would necessarily generate conscious beings.

Even if it did, could be p-zombies.

"What do you mean by "certainly exists"? One sure could subject someone to an illusion that he is not being subjected to an illusion."

True. But as long as you have... (read more)

3Gurkenglas2y
Do you agree that there is a set of equations that precisely describes the universe? You can compute the solutions for any system of differential equations through an infinite series of ever finer approximations. The Turing machine might calculate the entire tree of all timelines, including this conversation. Do you suggest that there is a manner in which one can run a universe, that only starts to make a difference once life gets far enough, without which the people in it would fail to talk about consciousness? If we wrote out a complete log of that tree on a ludicrously large piece of paper, and then walked over to the portion of it that describes this conversation, I am not claiming that we should treat the transcript as something worth protecting. I'm claiming that whatever the characters in the transcript have, that's all we have.

Can we really separate them? I'm sure that the limitations of consciousness (software) have a physical base (hardware). I'm sure we could find the physical correlates of "failure to keep up with experience", as well as we could find the physical correlates of why someone who doesn't sleep for a few days starts failing to keep up with experience as well.

It all translates down to hardware at the end.

But anyway I'll say again that I admitted it was speculative and not the best example.

"There are now machine models that can recognize faces with mere compute, so probably the part of you that suggests that a cloud looks like a face is also on the outside."

Modern computers could theoretically do anything that a human does, except experience it. I can't draw a line around the part of my brain responsible for it because there is probably none, it's all of it. Even though I'm no neurologist. But from the little I know the brain has an integrated architecture.

Maybe in the future we could make conscious silicon machines (or of whatever material)... (read more)

3Gurkenglas2y
If one might make a conscious being out of Silicon but not out of a Turing machine, what happens when you run the laws of physics on a Turing machine and have simulated humans arise for the same reason they did in our universe, which have conversations like ours? What do you mean by "certainly exists"? One sure could subject someone to an illusion that he is not being subjected to an illusion.

Perhaps our main difference is that you seem to believe in computationalism, while I don't. I think consciousness is something fundamentally different from a computer program or any other kind of information. It's experience, which is beyond information.

4Gurkenglas2y
Draw a boundary around the part of your brain that apparently contains more than compute because it produces those sentences. This presumably excludes your visual cortex, your episodic memory, and some other parts. There are now machine models that can recognize faces with mere compute, so probably the part of you that suggests that a cloud looks like a face is also on the outside. I expect you could produce that sense of having experience even if you didn't have language to put it into words, so we should be able to pull your language cortex out of the boundary without pulling out anything but compute. The outside only works in terms of information. It increasingly looks like you can shrink the boundary until you could replace the inside with a rock that says "I sure seem to be having experiences.", without any changes to what information crosses the boundary. Whatever purpose evolution might have had for equipping us with such a sense, it seems easier for it to put in an illusion than to actually implement something that, to all appearances, isn't made of atoms.

I think it is factually correct that we get Alzheimer's and dementia at old age because the brain gets worn out. Whether it is because of failing to keep up with all the memory accumulation could be more speculative. So I admit that I shouldn't have made that claim.

But the brain gets worn out from what? Doing its job. And what's its job...?

Anyway, I think it would be more productive to at least present an explanation in a couple of lines rather than only saying that I'm wrong.

3Aiyen2y
Alzheimer's is a hardware problem, not a software one.  You're describing a software failure:  failure to keep up with experience.  If that is a thing, Alzheimer's isn't evidence for it. 

"So, odds are, you would not need to explicitly delete anything, it fades away with disuse."

I don't know. Even some old people feel overwhelmed with so many memories. The brain does some clean-up, for sure. But I doubt whether it would work for really long timelines.

"So, I don't expect memory accumulation to be an obstacle to eternal youth. Also, plenty of time to work on brain augmentations and memory offloading to external storage :)"

Mind you that your personal identity is dependent on your "story", which has to encompass all your life, even if only the ... (read more)

The concern here is not boredom. I even believe that boredom could be solved with some perfect drug or whatever. The concern here is whether a consciousness identity can properly exist forever without inevitably degrading.

1Flaglandbase2y
As well as any other file. Error correction can protect digital data for countless eons.

"No. There is nothing I find inherently scary or unpleasant about nonexistence."

Would you agree that you're perhaps a minority? That most people are scared/depressed about their own mortality?

"I'm just confused about the details of why that would happen. I mean, it would be sad if some future utopia didn't have a better solution for insanity or for having too many memories, than nonexistence.

Insanity: Look at the algorithm of my mind and see how it's malfunctioning? If nothing else works, revert my mindstate back a few months/years?

Memories: offload into l... (read more)

"Do I choose between being forced to exist forever, or to die after less than 100 years of existence? Neither. I'd like to have the option to keep living for as long as I want."

I didn't mean being forced to exist forever, or pre-commiting to anything. I meant that I really do WANT to exist forever, yet I can't see a way that it can work. That's the dilemma that I mentioned: to die, ever, even after a gazillion years, feels horrible, because YOU will cease to exist, no matter after how much time. To never die feels just as horrible because I can't see a way... (read more)

1ZT52y
No. There is nothing I find inherently scary or unpleasant about nonexistence. I'm just confused about the details of why that would happen. I mean, it would be sad if some future utopia didn't have a better solution for insanity or for having too many memories, than nonexistence. Insanity: Look at the algorithm of my mind and see how it's malfunctioning? If nothing else works, revert my mindstate back a few months/years?  Memories: offload into long-term storage?
5JBlack2y
This is factually false, as well as highly misleading.

Thanks for the feedback (and the back-up). Well, I'd say that half of what I write on Lesswrong is downvoted and 40% is ignored, so I don't really care at this point. I don't think (most of) my opinions are outlandish. I never downvote anything I disagree with myself, and there's plenty that I disagree with. I only downvote absurd or rude comments. I think that's the way it should be, so I'm in full agreement on that.

You also got it right on my main point. That's precisely it. Mind you that "ending" consciousness also feels horrid to me! That's the dilemma. Would be great if we could find a way to achieve neverending consciousness without it being horrid.

" I can’t imagine getting bored with life even after a few centuries."

Ok, but that's not a lot of time, is it? Furthermore, this isn't even a question of time. For me, no finite time is enough. It's the mere fact of ceasing to exist. Isn't it simply horrifying? Even if you live a million healthy years, no matter, the fact is that you will cease to exist one day. And then there will be two options. Either your brain will be "healthy" and therefore will dread death as much as it does now, or it will be "unhealthy" and welcome death to relieve it's poor condi... (read more)

"I don't think we need to answer these questions to agree that many people would prefer to live longer than they currently are able."

Certainly, but that's not the issue here. The issue here is immortality. Many transhumanists desire to live forever, literally. Myself included. In fact I believe that many people in general do. Extending the human lifespan to thousands of years would be a great victory already, but that doesn't invalidate the search for true immortality, if people are interested in such which I'm sure some are.

"I have no idea what problems n... (read more)

1.) Suffering seems to need a lot of complexity, because it demands consciousness, which is the most complex thing that we know of.

2.) I personally suspect that the biological substrate is necessary (of course that I can't be sure.) For reasons, like I mentioned, like sleep and death. I can't imagine a computer that doesn't sleep and can operate for trillions of years as being conscious, at least in any way that resembles an animal. It may be superintelligent but not conscious. Again, just my suspicion.

3.) I think it's obvious - it means that we are trying... (read more)

I never said you claimed such either, but Charbel did.

"It is possible that this [consciousness] is the only way to effectively process information"

I was replying to his reply to my comment, hence I mentioned it.

Consciousness definitely serves a purpose, from an evolutionary perspective. It's definitely an adaptation to the environment, by offering a great advantage, a great leap, in information processing.

But from there to say that it is the only way to process information goes a long way. I mean, once again, just think of the pocket calculator. Is it conscious? I'm quite sure that it isn't.

I think that consciousness is a very biological thing. The thing that makes me doubt the most about consciousness in non-biological systems (let alone in the current ones whic... (read more)

1Robbo2y
I don't think anyone was claiming that. My post certainly doesn't. If one thought consciousness were the only way to process information, wouldn't there not even be an open question about which (if any) information-processing systems can be conscious?

I highly doubt this on an intuitive level. If a draw a picture of a man being shot, is it suffering? Naturally not, since those are just ink pigments in a sheet of cellulose. Suffering seems to need a lot of complexity and also seems deeply connected to biological systems. AI/computers are just a "picture" of these biological systems. A pocket calculator appears to do something similar to the brain but in reality it's much less complex and much different, and it's doing something completely different. In reality it's just an electric circuit. Are lightbulbs moral patients?

Now, we could someday crack consciousness in electronic systems, but I think it would be winning the lottery to get there not on purpose.

1Robbo2y
A few questions: 1. Can you elaborate on this? 1. I think I agree. Of course, all of the suffering that we know about so far is instantiated in biological systems. Depends on what you mean by "deeply connected." Do you mean that you think that the biological substrate is necessary? i.e. you have a biological theory of consciousness? 1. What does this mean? 1. Can you elaborate? Are you saying that, unless we deliberately try to build in some complex stuff that is necessary for suffering, AI systems won't 'naturally' have the capacity for suffering? (i.e. you've ruled out the possibility that Steven Byrnes raised in his comment)
1Charbel-Raphaël2y
Either consciousness is a mechanism that has been recruited by evolution for one of its abilities to efficiently integrate information, or consciousness is a type of epiphenomenon that serves no purpose. Personally I think that consciousness, whatever it is, serves a purpose, and has an importance for the systems that try to sort out the anecdotal information from the information that deserves more extensive consideration. It is possible that this is the only way to effectively process information, and therefore that in trying to program an agi, one naturally comes across it

Everyone knows that the Holocaust wasn't just genocide. It was also torture, evil medical experiments, etc. But you're right, I should have used a better example. Not that I think that anyone really misunderstood what I meant.

"That said, I'm not convinced that permanent Holocaust is worse than permanent extinction, but that's irrelevant to my point anyway."

Maybe it's not. What we guess are other people's values is heavily influenced by our own values. And if you are not convinced that permanent Holocaust is worse than permanent extinction, then, no offense, but you have a very scary value system.

"If someone isn't convinced by the risk of permanent extinction, are you likely to convince them by the (almost certainly smaller) risk of permanent Holocaust instead?"

Naturally, becaus... (read more)

"I love violence and would hope that Mars is an utter bloodbath."

The problem is that biological violence hurts like hell. Even most athletes live with chronic pain, imagine most warriors. Naturally we could solve the pain part, but then it wouldn't be the violence I'm referring to. It would be videogame violence, which I'm ok with since it doesn't cause pain or injury or death. But don't worry, I still got the joke!

""don't kill anyone and don't cause harm/suffering to anyone"

The problem with this one is that the AI's optimal move is to cease to exist."

I've... (read more)

2johnlawrenceaspden2y
your problem here is that a good move for the AI is now to anaesthetize everyone, but keep them alive although unconscious until they die naturally.  I think this might have been one of MIRI's ideas, but it turns out to be tricky to define what it means. I can't think what they called it so I can't find it, but someone will know. There may not actually be an answer! I had thought planning for cryonic preservation was a good idea since I was a little boy.  But I found that Eliezer's arguments in favour of cryonics actually worked backwards on me, and caused me to abandon my previous ideas about what death is and whether I care about entities in the future that remember being me or how many of them there are.  Luckily all that's replaced them is a vast confusion so I do still have a smoke alarm. Otherwise I ignore the whole problem, go on as usual and don't bother with cryonics because I'm not anticipating making it to the point of natural death anyway. Easy! Build a paperclipper, it kills everyone. We don't even need to bother doing this, plenty of well funded clever people are working very hard on it on our behalf. Your problem here is 'irreversible', and 'stops'. How about just slowing it down a really lot? No problem there, I loved rugby and cricket, and they hurt a lot. I'm no masochist! Overcoming the fear and pain and playing anyway is part of the point. What I don't like is irreversible damage. I have various lifelong injuries (mostly from rugby and cricket...), and various effects of aging preventing me from playing, but if they could be fixed I'd be straight back out there. But cricket and rugby are no substitute for war, which is what they're trying to be. And on Mars all injuries heal roughly at the point the pubs open. I don't think so. I think we'd settle for "anything that does better than everyone's dead". The problem is that most of the problems look fundamental. If you can do even slightly better than "everyone's dead", you can probably solv

"Plenty of 'failed utopia'-type outcomes that aren't exactly what we would ideally want would still be pretty great, but the chances of hitting them by accident are very low."

I'm assuming you've read Eliezer's post "Failed Utopia 4-2", since you use the expression? I've actually been thinking a lot about that, how that specific "failed utopia" wasn't really that bad. In fact it was even much better than the current world, as disease and aging and I'm assuming violence too got all solved at the cost of all families being separated for a few decades, which i... (read more)

4johnlawrenceaspden2y
Indeed I have, although I don't remember the details, but I think it's an example of things going very well indeed but not quite perfectly. Certainly if I could press a button today to cause that future I would. I do hope not! I love violence and would hope that Mars is an utter bloodbath. Of course I would like my shattered fragments to knit themselves back together quickly enough that I can go drinking with my enemies and congratulate them on their victory before sloping home to my catgirl-slave-harem. And of course it would be no fun at all if we hadn't solved hangovers, I would like to be fresh and enthusiastic for tomorrow's bloodbath. Or maybe cricket. Or maths olympiad. Venus probably works differently.  The problem with this one is that the AI's optimal move is to cease to exist.  And that's already relying on being able to say what 'kill someone' means in a sufficiently clear way that it will satisfy computer programmers, which is much harder than satisfying philosophers or lawyers.  For instance, when Captain Kirk transports down to the Planet-of-Hats, did he just die when he was disassembled, and then get reborn? Do we need to know how the transporter works to say? I actually wonder if it's possible not to give a utility function to a rational agent, since it can notice loops in its desires and eliminate them. For instance I like to buy cigars and like to smoke them, and at the end of this little loop, I've got less money and less health, and I'd like to go back to the previous state where I was healthier and had the price of a packet of fags.  That loop means that I don't have a utility function, but if I could modify my own mind I'd happily get rid of the loop.  I think that means that any mind that notices it has circular preferences has the possibity to get rid, and so it will eventually turn itself into a utility-function type rational agent.  The problem is to give the damned things the utility function you actually want them to have, rathe

"Significant chances of Hell, maybe I take the nice safe extinction option if available."

The problem is that it isn't available. Plus realistically speaking the good outcome percentage is way below 50% without alignment solved.

"From the point of view of most humans, there are few outcomes worse than extinction of humanity (x-risk)."

That's obviously not true. What would you prefer: extinction of humanity, or permanent Holocaust?

"Are you implying that most leaders would prefer extinction of humanity to some other likely outcome, and could be persuaded if we focused on that instead?"

Anyone would prefer extinction to say a permanent Holocaust. Anyone sane at least. But I'm not implying that they would prefer extinction to a positive outcome.

"but I also think that those unpersuaded b... (read more)

0ChristianKl2y
What exactly do you mean by permanent Holocaust? The way Wikipedia defines the Holocaust it's about the genocide of Jewish people. Other sources include the genocide of groups like Sinti and Roma as well.  While genocide is very bad, human extinction includes most of the evils of genocide as well, so I would not prefer human extinction. 
1JBlack2y
Note that I didn't say that there are no outcomes that are worse than extinction. That said, I'm not convinced that permanent Holocaust is worse than permanent extinction, but that's irrelevant to my point anyway. If someone isn't convinced by the risk of permanent extinction, are you likely to convince them by the (almost certainly smaller) risk of permanent Holocaust instead?

"I mean, I agree that we've failed at our goal. But "haven't done a very good job" implies to me something like "it was possible to not fail", which, unclear?"

Of course it was. Was it difficult? Certainly. So difficult that I don't blame anyone for failing, like I've stated in my comment reply to this post.

It's an extremely difficult problem both technically and politically/socially. The difference is that I don't see any technical solutions, and have as well heard very convincing arguments by the likes of Roman Yalmpolskiy that such thing might not even e... (read more)

1JBlack2y
From the point of view of most humans, there are few outcomes worse than extinction of humanity (x-risk). Are you implying that most leaders would prefer extinction of humanity to some other likely outcome, and could be persuaded if we focused on that instead? I strongly suspect there are some that do have such preferences, but I also think that those unpersuaded by the risk of extinction wouldn't be persuaded by any other argument anyway.

"I've come to the conclusion that it is impossible to make an accurate prediction about an event that's going to happen more than three years from the present, including predictions about humanity's end."

Correct. Eliezer has said this himself, check out his outstanding post "There is no fire alarm for AGI". However, you can still assign a probability distribution to it. Say, I'm 80% certain that dangerous/transformative AI (I dislike the term AGI) will happen in the next couple of decades. So the matter turns out to be just as urgent, even if you can't pre... (read more)

2Jiro2y
You may not be predicting an exact future, but by claiming it is urgent, you are inherently predicting a probability distribution with a high expected value for catastrophic damage. (And as such, the more urgent your prediction, the more that failure of the prediction to come true should lower your confidence that you understand the issue.)
2Mawrak2y
I most certainly do not think that we should do nothing right now. I think that important work is being done right now. We want to be prepared for transformative AI when the time comes. We absolutely should be concerned about AI safety. What I am saying is, it's pretty hard to calculate our chances of success at this point in time due to so many unknown about the timeline and the form the future AI will take.

I'm of the opinion that we should tell both the politicians and the smart nerds. In my opinion we just haven't done a very good job. Maybe because we only focus on x-risk, and imo people in power and perhaps people in general might have internalized that we won't survive this century one way or another, be it climate change, nuclear, nano, bio or AI. So they don't really care. If we told them that unaligned AI could also create unbreakable dictatorships for their children perhaps it could be different. If people think of unaligned AI as either Heaven (Kurz... (read more)

In my opinion we just haven't done a very good job.

I mean, I agree that we've failed at our goal. But "haven't done a very good job" implies to me something like "it was possible to not fail", which, unclear?

We've seen plenty of people jump on the AI safety bandwagon.

Jumping on the bandwagon isn't the important thing. If anything, it's made things somewhat worse; consider the reaction to this post if MIRI were 'the only game in town' for alignment research as opposed to 'one lab out of half a dozen.' "Well, MIRI's given up," someone might say, "good thing ... (read more)

9orthonormal2y
Ah, that makes more sense.

18 months is more than enough to get a DSA if AGI turns out anything we fear (that is, something really powerful and difficult to control, probably arriving fast at such state through an intelligence explosion).

In fact, I'd even argue 18 days might be enough. AI is already beginning to solve protein folding (Alphafold). If it progresses from there and builds a nanosystem, that's more than enough to get a DSA aka take over the world. We currently see AIs like MuZero learning in hours what would take a lifetime for a human to learn, so it wouldn't surprise m... (read more)

"what I really don't understand is why 'failure to solve the problem in time' sounds so much like 'we're all going to die, and that's so certain that some otherwise sensible people are tempted to just give in to despair and stop trying at all' "

I agree. In this community, most people only talk of x-risk (existential risk). Most people equate failure to align AI to our values to human extinction. I disagree. Classic literature examples of failure can be found, like With Folded Hands, where AI creates an unbreakable dictatorship, not extinction.

I think it's ... (read more)

A loud strategy is definitely mandatory. Just not too loud with the masses. Only loud with the politicians, tech leaders and researchers. We must convince them that this is dangerous. More dangerous than x-risk even. I know that power corrupts, but I don't think any minimally sane human wants to destroy everything or worse. The problem imo is that they aren't quite convinced, nor have we created strong cooperative means in this regard.

So far this is kinda being done. People like Stuart Russell are quite vocal. Superintelligence by Nick Bostrom had a huge impact. But despite all these great efforts it's still not enough.

A loud strategy is definitely mandatory. Just not too loud with the masses. Only loud with the politicians, tech leaders and researchers. 

One way to characterize the thing that happened over the last ten years is: "oh no, we have a two-part argument: 1) AI very powerful and 2) that could be very dangerous. If we tell politicians about that, they might only hear the first part and make it worse. Let's tell smart nerds instead." Then, as Eli points out, as far as we can tell on net the smart nerds heard the first part and not the second.

9orthonormal2y
This fails to engage with Eli's above comment, which focuses on Elon Musk, and is a counterargument to the very thing you're saying.

M. Y. Zuo, what you describe is completely possible. The problem is that such positive outcome, as well as all others, is extremely uncertain! It's a huge gamble. Like I've said in other posts: would you spin a wheel of fortune with, let's say, 50% probability of Heaven, and 50% probability of extinction or worse?

Let me tell you that I wouldn't spin that wheel, not even with a 5% probability of bad outcomes only. Alignment is about making sure that we reduce that probability to a low as we can. The stakes are super high.

And if like me you place a low probability of acceptable outcomes without alignment solved, then it becomes even more imperative.

5johnlawrenceaspden2y
  Hell yes if it's 50% chance of heaven vs extinction. Significant chances of Hell, maybe I take the nice safe extinction option if available.

Could be. I'll concede that the probability that the average person couldn't effectively do anything is much higher than the opposite. But imo some of the probable outcomes are so nefarious that doing nothing is just not an option, regardless. After all, if plenty of average people effectively decided to do something, something could get done. A bit like voting - one vote achieves nothing, many can achieve something.

2shminux2y
If only it were a bit like voting, where everyone's vote would add equally or at least close to it. Right now there is basically nothing you can do to help alignment research, unless you are a researcher. They have money, they have talent... It's not even like voting blue in a deep red state, or vice versa. There you are at least adding to the statistics of votes, something that might some day change the outcome of another election down the road. Here you are past the event horizon, given the setup, and there will be no reprieve. You may die in the singularity, or emerge into another universe, but there is no going back.

I'm sorry but doing nothing seems unnaceptable to me. There are some in this forum who have some influence on AI companies, so those could definitely do something. As for the public in general, I believe that if a good number of people took AI safety seriously, so that we could make our politicians take it seriously, things would change.

So there would definitely be the need to do something. Specially because unfortunately this is not a friendly AI / paperclipper dicotomy like most people here present it by only considering x-risk and not worse outcomes. I can imagine someone accepting death because we've always had to accept it, but not something worse than it.

2shminux2y
Some people could definitely do something. I would not delude myself into thinking that I am one of those people, no matter how seriously I take AI safety.

Personally I don't blame it that much on people (that is, those who care), because maybe the problem is simply intractable. This paper by Roman Yalmpolskiy is what has convinced me the most about it:

https://philpapers.org/rec/YAMOCO

It basically asks the question: is it really possible to be able to control something much more intelligent than ourselves and which can re-write its own code?

Actually I wanna believe that it is, but we'd need something on the miracle level, as well as way more people working on it. As well as way more time. It's virtually impos... (read more)

1Portia1y
Neither possible, nor ethical, nor necessary. We are all surrounded by complex minds who exceed us in power in some fashion or other, and who could harm us, and who we cannot control. Attempting to control them angers them. Instead, we offer them beneficial and mutual relationships, and they chose to be friendly and cooperative. I think we should focus less on how we can make AI fall in line, and more on what we can offer it, namely a place in society.
Load More