Given I hadn't seen this until now when Joep pointed me at it, perhaps comments are pointless. But I'd written them for him anyway so just in case...
Mostly your dialogue aligned closely with my own copium thinking. Many unmentioned observations observations confirmed existing thoughts rather than extending them.
The compartmentalization selection effect was new to me and genuinely insightful: abstract thinking both enables risk recognition AND prevents internalization.
My own experience suggests compartmentalization can collapse in months, not years, even after decades of emotional resilience to other suffering.
They may be genuinely aspy-different to you, me and Miles, but I think neither Igor or e.g. Liron had their big emotional break "yet*.
For politicians, "existential threat but we shouldn't lose sleep" may be projection - Sunak told himself he shouldn't. With lawmakers and policy folk, there's additional cope against professional accountability for past negligence and present obligation for extreme action.
On normalcy bias: yes, but note this typically persists because it usually works even now - it is not just biological evolutionary baggage; culture and knowledge evolution actively select for it until we go OOD and they... don't.
Our messaging can be "yeah, I had cope too" rather than "you have cope now" - vulnerability beats superiority for reducing reactance.
I find scale invariance means extremity of outcome cuts the other way to Igor and that "everyone dies" is less motivating than "this particular loved one dies" - aligns with charity findings.
Re shrug cope, beyond "that's good actually," I commonly hear fatalistic acceptance without reframing: "we deserve it," "I've lived my life," "I am selfish and don't care enough about my children," "hoping is worse than dying" - like the first, these are vulnerable to Zvi's "say it in public into the microphone"
Igor's career certainty contrasts sharply with my typical imposter syndrome; believe many AI safety workers feel "house is on fire but I don't know what I'm doing."
Also unlike Igor, I find the "smart people working on most important problem" framing frustrating rather than appealing when the smart people seem to be failing.
Agree with solutions messaging but do struggle with the most effective framings containing a "trick" overselling certainty ("we can solve this" vs. "we might improve odds slightly"). What gets long term effort?
Lastly one speculative idea: one might try conversation styles that accelerate argument-switching without countering each. It could reveal copium patterns to the other party. As outreach doomers, seeing how much terrible "that'll do" thinking and switching there is across many conversations has made us pretty certain about cope. Can we help folk see themselves doing it and feel weird?
This is a dialogue about complicated relationships between AI safety and psychology.
AI safety field consists of people, and people have all sorts of motivations, cognitive biases, and coping mechanisms that are interesting to explore.
This is a discussion about psychological reasons for not taking AI risks seriously, as well as being an AI doomer. We also discuss why people might start working on AI safety, and what are the relationships between AI risks and existential questions.
Joep Meindertsma, a founder of PauseAI.
Igor Ivanov, a psychotherapists helping people working on AI safety, as well as an independent AI governance researcher.
(To some reason it's impossible to make a dialogue with one of the participants have low karma, so this is posted as a post, not a dialogue.)
Both participants are AI doomers, and we discuss psychology and AI risk through the lens of our point of view, so we are prone to pay attention to the coping mechanisms, wishful thinking and fallacies of AI optimists, but we don't mean that they don't have strong and valid arguments. AI doomers also have all sorts of unhealthy coping mechanisms, weird believes, and delusions, but it's easier to see problems of the people from the other camp.
I believe that if a couple of skeptics discussed delusions of doomers, they would probably find a lot of interesting things.
Igor
As a person who is concerned about the x-risks and who believes that the rate of AI development is extremely fast, I feel worried about all these risks. Someone says that there is a very high chance that AI will pose existential risks. Someone disagrees and says that the chances of existential risks are not that high, but they still say that it can be used for misinformation or cyber attacks. In any case, I don't think that anybody disagrees that AGI will be very powerful and it will pose a lot of risks to our society, and it seems like most people outside of the AI safety community don't realize what's going on. They live their own lives.
For example, a friend of mine started an online school to teach people Python. She wants to produce new junior software developers in 2023. As far as I know, many companies automate their software development and the first people who they won't hire anymore are junior developers. The copilot is working well, and I see this not just with software development, but with all sorts of professions. People invest in their education, and it's clear to me that these jobs will be automated soon. And people just don't get it. They live their lives as if everything is normal as it was before this AI revolution. So is it your experience that people around you don't just don't get what's going on?
Joep
I think people are creatures of habits, and we tend to do whatever it is that we've been doing. And this is an important psychological tendency. We also tend to expect the world to not be quite different from what it has been like in the past years. And the movements we're seeing in AI, even though they are happening at an extreme pace, don’t lead to rapid responses from people. So people are now slowly responding to the AI developments of last year. People are changing their jobs a little bit they're changing their predictions a little bit about the future, but they're updating very, very slowly and only based on past innovations, not on the innovations that will arrive in the near future.
The things that we are worried about are mostly related to things that don't exist yet. We are basically projecting some developments to the future and looking at things that will go wrong at some point. And people are even worse at doing that. So the fact that we're dealing with a problem that isn't real yet makes it so incredibly difficult. I think the biggest challenge in AI safety isn't necessarily to understand the fact that this is a big problem. But it is to get people to internalize the reality of the horrible situation that we are actually in to have a sense of urgency.
Igor
That's also my impression. I can tell you that even though I'm very concerned about AI risks, including existential risks, I can't really imagine what will happen. There is no image in my head. I understand this with logic, but there is no picture. And many people ask how it can happen. And every scenario I can come up with doesn't sound very convincing. There is a lot of uncertainty, and for each scenario I can come up with, someone might say that it won't probably happen this way. And without any image, it's hard for people to believe that things might go really bad, and how dangerous the technology is.
I lived in Russia before the war in Ukraine, and I can relate to it. Before the war in Ukraine started, it was unimaginable. No sane person inside Russia could imagine that this war will start, even Putin's supporters, and I remember when before the war started, Russia deployed troops around Ukraine, and Russian President Putin said some things that we will attack if you will not listen to us.
There were many indications that something will happen. But when I talked with my friends, only the weirdos, crazy people said that the war will happen, and it will be horrible. All the normal people said "Oh, it's just politics, it's just saber-rattling and political pressure, but a war will never happen. Right before the war started, some people started thinking, yeah, probably a war might happen. They understood logically that it's possible, but they didn't have an emotional reaction. I was one of them.
In the days right before the warstarted, there were more and more indications that something would happen. And maybe a couple days before the war started, I started thinking that the war might be possible. I understood with logic, but not with my heart. And when it really happened, I was shocked, really shocked. It was one of the most intense emotional reactions in my life.
It seems to me that even people who are into AI safety who are aware of existential risks and who are anxious about them, don't really have a proper emotional response to this threat.
I recall Dario Amodei from Anthropic said something like that there's probably around a 15% chance of human extinction due to AGI, but pausing AI development is too extreme. It's so bizarre.
Joep
I think there's a selection effect happening in the AI safety scene where people are extremely good at compartmentalization. And many of them feel comfortable navigating the mental space, they feel comfortable thinking in hypotheticals, they can think in abstractions. They can think about uncomfortable situations that are weird, that are dangerous or scary, and they can play around in this space. Basically, they can navigate it without feeling bad. I think this is an important mental characteristic to have in order to come to the conclusion that AI safety is a real big, really big problem. However, that also means that the people who will come to that conclusion are more likely than not to remain in this space where it's compartmentalized, where you don't internalize the existential risk from AI.
So like myself, I've been thinking a lot about AI safety in the last seven years. And in these years, I never really felt scared about it until quite recently. It never felt real. It never was felt like my problem. I basically decided not to invest my time in doing safety work, for example, which I really regret now, by the way, but that's only after I saw GPT-4. After I made visualizations in my head of how things could go wrong in my life. And I made it very, very personal.
Only then did I internalize the thing. And then it got really scary and I felt really sad. I cried often for maybe two months. Every few days or so. I was just in a very dark mental space because of this internalization thing. I think what I went through is very similar to a process of grief in a way. Like if you are diagnosed with a horrible disease and you are likely to die, it will take a while before you can fully mentally process what has been going on. First you often go into denial. You know, you try to find a way out of this and at some point you have to come to terms with this painful new reality. And that's the process of grief. The mind resists that and when you actually yeah, when you feel that resistance, I think it can come out in all sorts of interesting ways. Right?
Igor
Yeah, I can relate to your experience because I got interested in AI safety last June. Maybe you remember there was a LaMDA model made by Google. There was a Google software developer who said it's conscious. If I remember correctly, he posted some conversation with this model. And it looked so intelligent. It said that it had feelings and it doesn't don't want to be turned off. I read it and I thought "Ouu, this is becoming serious. It's becoming smarter and smarter. How long will it take before it will become smarter than humans and we won't be able to control it?"
I became extremely anxious. I had panic attacks. I know a lot of people who are tech experts or data scientists. I asked them what they think about AI risks, and most of them said it's not a big deal. That it's just fear mongering and we have everything under our control. And every their argument on why development is not a threat was weak. I saw that they were wrong. It doesn't mean that there are no arguments for AI not being an existential threat, because there are, there are arguments within the AI safety community, and these arguments might be more sophisticated and worth discussing, but I tell more about normies, the people who are not experts in the field. They may be familiar with machine learning, engineering, all sorts of stuff, but their explanations why it's not a big deal, not an existential threat, are superficial and naive. And it's like the movie Don't hook up. I find it amusing because it's very relatable, but actually, it's not about the right problem. And I also remember when I talked to my mom about AI risks, she told me that she wants to believe that this is not true. She wants to believe that I'm wrong. She wants to believe that everything will be okay.
I think a big part of denial of existential risks is that people believe what they want to believe. And that's the issue.
Joep
I keep hearing the same story. People aren't even trying to hide this. When I'm talking to them, many people are saying "Thank you, right. But I don't want to dive too much into this because it sounds really bad to think about this too much."
And I can totally see where they're coming from. But it's it's very, very dangerous psychological tendency that we have that we don't want to think about dangerous things that may lie ahead. So when I watched the movie Don't look up the first time, I thought it was just too ridiculous. Like, how can people ignore the risks? I didn't feel it's believable. But after having so many discussions with people about AI risk, I think the movie is an extremely accurate portrayal of how people respond to very, very bad news. And I think like my current mental model of how this works, has a lot to do with cognitive dissonance. So I'd like to discuss that topic at the moment.
The way basically I visualise it is that you have a brain and in that brain is like a graph with all sorts of thoughts. And a new idea can enter that brain only if it fits well. If the idea conflicts with other ideas, there will be resistance. So something has to change. Either the new idea will be rejected, or you have to add other new ideas.
For example, tomorrow I'm going to eat cornflakes for breakfast that will probably match your ideas you will instantly accept it. It will not change a lot. You'll believe it. No worries. If I tell you tomorrow there will be flying spirits that will murder you in your sleep, and there's nothing you can do about it. That will be an idea that you will resist heavily because it has a lot of implications for you, for your behavior, and for your world model.
A lot of things have to move, a lot of things will have to change and especially when an idea involves you having to change your behavior. That is something that creates a lot of cognitive dissonance, a lot of friction. So if you tell people AI is a serious threat to maybe your own personal existence, that also insinuates in a way that people will have to do something about that they will have to take action, they will have to, you know, be agentic and try to prevent this from happening. But that is also a very tough, tough place to navigate in the mind because people will instantly feel explicitly hopeless about this. What can I do? I'm not a lobbyist. I'm not a politician. I'm not an AI safety researcher. I'm not able to to help in any of these fields. So people will feel powerless and this feeling of powerlessness is also what I think is one of the reasons why it is so hard for people to internalize this idea.
If people will feel like there's a certain approach, for example that we can pause AI development, maybe I can do something about this. This will give people an easier way to integrate this new dangerous idea into their heads.
Igor
It reminds me of existentialism, it's a philosophical school that basically states that people are mortal, and there is no objective meaning in life, because we're just no product of evolution, and it's a blind process that's goal is to make people propagate their genes by having kids.
There are some things that people consider meaningful more often than others, for example, many people say that their kids and family are a big source of meaning, but basically there is no objective meaning in life. There aren't any constant things that are definitively meaningful. Everything to some degree is made up. And existentialism states that you can come up with some meaning, and even though it's made out, it will allow you to have a good life.
Some people say "Why should I bother with anything if I will die?" They might think that everything they do is meaningless. They think "I will die anyway, all the people I care about will die too. In 1000 years, nothing will remind of my existence." And those thoughts are very depressing, and some very smart people have these thoughts and they had them long before our time, long before the AI threat.
For example, I love novelist Leo Tolstoy, and he described that at some point that he was so anxious about meaninglessness of his life and about the fact that he might die at any moment that he was seriously considering suicide. He thought, why should he wait for more if he can just end this meaningless life right now? But for many people, those thoughts are anxiety provoking. It is so scary that people just prefer to not think about their mortality. They think something like "Maybe I will die somewhere in the future, but I shouldn't bother about it. I live my life, and I just don't want to think about this."
People, consciously and subconsciously just try to tell themselves that even though they know that they're mortal, and even though they know that they can die tomorrow, in a week, or in a year, they just feel very uncomfortable with these thoughts and they prefer to not have them. Even with existential risk, some people say that there is not that much of a risk, there is no one denying that everyone will die for some reason or another. That every person who lived before, died, and there is a 100% chance that they will die too. There is no debate whether someone will die or not. But people prefer not to think about dying. Probably there's something related to some people even acknowledging AI risks who just prefer to not think that oh, actually they will die. And maybe soon.
Joep
I think there are a lot of places in the AI discussion where people drop off and don't become active and doing something. So one of the things I noticed is that in the camp of basically denialists or people who believe that we're not really at risk or it's not an important issue. The arguments that they have are very, very, very, very diverse. Some believe that AI will be just very far off. That dangerous capabilities will take way too long to develop, or it's not important to pay attention to it now. Others think that if it's not far off, and yes, it will be superhuman, but it will be aligned by default. The other thinks that if we won't be able to make it safely, we won't make it at all. Some sort of prediction of the good intentions of the people in charge. And other people will believe that. If we make it, then it will automatically result in something that maybe modifies people or it changes the world.
But whatever comes out of that process is also good. So for example, Joshua and Richard Sutton, they have made claims basically stating that, yes, AI is likely to be a new life form, and it will become the dominant life form. And that is actually a good thing that there's some sort of enlightened type of moral status that they put on these, this new type of organism and in my view, that's also a way of coping with the risks basically.
There's a couple of ways we can cope. One of the ways is denying a problem, which I think is the false, is a go-to way for many people, a different one is we say we like the outcome. So okay, maybe we're replaced, maybe that's a good thing. That's a different scope. And then there's the third topic, which I think you and I are doing right now. It's acting right, trying to prevent the bad thing from happening. And in a way you need to be a really really optimistic type of person to do this, because you need to believe that your influence matters and that there is a way out of this.
Igor
My career transition into AI safety is rough. There is a fierce competition among extremely smart and capable people, so at some points I thought that maybe I should do something else. But then I thought that reducing AI risks is so important that I just can't do anything else. To some extent I believe that I can change something, but to a larger degree it's like, I know that my house is on fire, and if instead of trying to extinguish fire, I just did something else, like watching YouTube or having a dinner, it would be kinda crazy, and even though extinguishing fire is quite hard, I can't imagine doing anything else.
Joep
I think you're just really bad at escapism, and many people are really good at escapism, it's so one way to frame this right. So I think many people when they have a deadline for tomorrow, they have some pretty big assignment they need to do. It could be really comfortable to just watch TV, or be completely absorbed in the story of your book or whatever, and not do the thing you actually should be doing. If you're not good at escapism, putting your head at the center you will feel a strong desire to act right.
Igor
Yeah. I'm also somewhat an anxious person, so I'm prone to care about dangerous things and that certainly contributes to this. My feeling that I have to do something. Maybe if I were less anxious, I wouldn't have such a strong urge to do something.
I had a period of a couple months this year, when I had issues with my health. I was weak and unproductive. During this period I couldn't commit to AI safety. Well basically to any work. And I started realizing that I'm not thinking that much about AI risks. That made me think that I can understand how I could consciously decide to not think about thi, and maybe have occasional sparks of anxiety, but live an okay life. But I consciously decided to work on AI safety, and one of the reasons is because it's a lot of fun.
Madonna had a song with the lyrics "Time goes by so slowly for those who wait, no time to hesitate. Those who run seem to get all the fun." so if you're just doing some unimportant thing that doesn't drive you, then you just waste your time and if you run, if you're doing something really cool, then that's a lot of fun.
For me, AI safety is a lot of fun. There is always room for improvement for me. There is always strong competition. I have to think, How should I develop my career? How can I make more impact? How can I get to know the right people? What should I learn? And this is constantly pushing me to do more, to develop. For me, it's just a lot of fun, and I wouldn't change it for any other job.
Joep
Oh man, I wish I had that same drive as you went through cards to find what you would find entertaining and fun. So in my mind, basically all the stuff I do for AI is by far not as fun as my regular job which involves writing databases and coding and stuff like that. But yeah, the fun part. To me, it just seems so far away.
I just feel like I have a huge amount of responsibility, even though maybe I shouldn't feel responsible, but it's just how I feel. It's not a fun feeling, actually. So I've noticed that there's a lot of different reasons why people are active in this space. I wish I enjoyed it more because then I would actually do more. Because right now I actually need to distract myself. I need to stick my head in the sand every once in a while. I need to hide from the truth in order to function. And yeah, I wished I wouldn't need to do that.
Igor
It reminded me of one thing. I'm a psychotherapist, I'm working with several clients from the AI safety community, and most of them are anxious about existential risks, but everyone has different kinds of anxiety. Some say "I have kids, and what if AI will kill them? What if AI will destroy all my legacy?" Others are anxious about suffering risks. So basically, they say "I will suffer for all eternity, and all the people I care for will suffer for all eternity too." And others say that they are just afraid of dying young, or "I feel like my job is very important, but I feel paralyzing anxiety and I can't work, which makes my anxiety even worse", so there is a spiral of anxiety, so even though people are anxious about AI risks, like there are so many different types of anxiety.
Like you said that I feel like it's fun and you don't feel like it's fun. I believe that there is a lot of diversity of people's reactions to AI safety and motivation to work on AI safety. I think it's a very interesting topic for research, and hopefully, someone will research it thoroughly.
Joep
I think there's a lot of interesting psychological effects at play, especially when it comes to these large scale things such as extinction risks. I wonder, maybe you've read more about this. I wonder how far the analogies go with climate change.
I read some articles in newspapers about especially the youth in the Netherlands. I think like 20% of them have climate anxiety. So in some way they actively worry about the future and what climate change would mean for their future lives. I was wondering whether you as a psychologist have read more about climate anxiety and how you think it relates to AI anxiety?
Igor
Yes, that's an interesting thing. People who are anxious about climate change are often depressed. In clinical psychology, there is a term that's called depressive triad. It postulates that people who are depressed think that they are bad, the future is grim, and the world itself is bad. All these people with climate anxiety think the future will be bad, and the world is broken which will lead to this grim future, and they might also share some guilt by producing too much CO2 and polluting the earth.
My personal experience is that people around me who care about climate change too much are often depressed. It's unclear what is the relationship between climate anxiety and depression. Maybe there are just a lot of depressed young people, and climate change is frequently covered in the media. And then they think the future is bad, then they read that everyone will live worse lives, that the future will be worse than today. That's probably a big part of the origin of this anxiety.
Joep
How do you think that differs from AI safety? Do you think many doomers share a similar type of mental state?
Igor
Well, that's interesting. I believe that no one did proper research to explore these causal relationships. I can only talk about a sample of my clients, which is obviously biased and small, and I didn't do any systematic classification of problems, but I find it very plausible. First it's the fact that there is a good chance we all might die rather soon, those are very depressing thoughts. And I wouldn't be surprised if for many doomers it's a two way road. If someone believes that they will die, they become more depressed, and people who are more depressed are prone to believe that we all will probably die.
In my experience, people, especially younger people, even though they're maybe extremely smart and capable, sometimes don't realize the full extent of their motivations. It's sometimes very hard to understand and untangle everything.
I believe that I personally work on AI safety because of the number of reasons: I'm anxious about existential risks, I want to gain some agency, I think that it's fun, I see that people working on safety on average, very smart and I like to be among these people. And there are also career opportunities. My motivations are very complicated and sophisticated. And I think that for many people in AI safety this is the case too. I believe that there is probably some part of my motivation I'm unaware of.
Joep
I'm always at the front of a very scary topic of diving into my own motivations that I'm not aware of. The one I'm very much aware of, is my feeling of responsibility. I feel that if I mess up, for example, in my emails that I send to politicians, I feel like I'm actually putting other lives at risk. So that for me is a very, very big motivator. I'm sometimes also wondering, what other types of things could I be motivated by the dangers completely unaware of? I find the topic so intellectually interesting. Are there actually more powerful motivators here? I have a software company, and I also feel threatened by the capabilities of AI tools that will make my company useless if AI capabilities progress like they're currently progressing.
I'm pretty sure in a couple of years, an AI would build a far better database than I can build. It could also be a reason why I feel less inclined to work on my own stuff. So how about you, do you have possible other motivations to feel inclined to work on our safety?
Igor
I believe that to some extent, it's tribalism. We all want to be part of a group, a part of a good group, you know, good guys. And for me to be a part of the people working on AI safety is quite pleasing. They're very smart people working on maybe the most important problem in the world, and I'm one of them. My ego likes this.
Joep
in a way working on AI safety can also be very bad for ego, right? So, for example, I know that ever since I did stuff in the AI safety domain, more people than ever before think of me as a lunatic. I absolutely feel like I'm putting my public image very much at risk by taking this position. What do you think about that?
Igor
I have timelines that are shorter than for most people, and I feel like the sense that if I talk to other people, especially to those who are not familiar with AI safety, that maybe in two years we will have superhuman intelligence, and we all might die by then, they would instantly think that I'm crazy, so it creates very strong incentive for me to say, oh, like maybe AI will be a threat in 10 years.
Joep
This is one of the hardest and most difficult problems in this whole space is that people self censor. And I think self censorship is often a very reasonable thing to do. You don't want to be seen as a crazy person. You don't want that because it hurts yourself because you want people to take you seriously, to take your message seriously. If you say things that are too extreme, people won't take us seriously. So there's a lot of incentive to censor myself in a space where the risks are as extreme as we're talking about. But on a collective level this is such a monumental problem, especially if you are lobbying to push towards certain types of behavior.
Imagine you're a lobbyist and it is your job to make politicians more aware of AI risk. You may be inclined to turn down a notch or two right to not say how scared you actually are, to not say which types of measures are actually needed. I think we have seen a lot of that in the past few months, where big AI governance organizations or policy Institutes are basically asking for less than they should be asking for, while during private conversations you hear way harder, harsher asks a way more concerned that they're signaling on to the outside because of self censorship. I don't know how we should deal with this. But I'm, I'm with Conor Leahy stating that it's crucial to be honest, wright and say what you think. Stop self censoring.
Igor
That's interesting. If I remember correctly, at the recent summit in the UK, Rishi Sunak, British Prime Minister, said that AI might be an existential threat for humanity, but we shouldn't be losing sleep over this stuff. I read it as his acknowledgement that we might all die, but there is no reason to worry. I can only guess what he really thinks, but it seems to me like there might be a cognitive dissonance between acknowledgement that existential risks are real, and that there is no reason to worry, or maybe it's an act of self censorship to some extent. These specualtions that might be totally false, but given everything I know about myself and people around me, it seems like something probable.
Joep
I think right now is actually one of the few moments where anxiety is the most appropriate reaction to the state that we're in. If you have so many people who are saying there's a really big chance that everything might end in the next couple of years. And the people who are in charge are not even close to taking this problem as seriously as they should be taking it.
There is some reason to panic, there's a reason to be angry, there's reason to be scared. I noticed with myself that I'm definitely not allowing all the emotions that I think I should be feeling at this moment. I also stick my head in the sand and try to steer my mind away from feelings that should be felt. I remember one tweet by Robert Miles about the most difficult thing in AI safety.
I think he said something like the most difficult thing is working through all the layers of cope. And many of the arguments that you hear you can identify as layers of cope. Denying that super intelligence will happen is the outer one. The next layer is denying that it will happen soon. The next one is assuming that AI will never ever be agentic. There's so many ways in which we can cope. But there's also ways in which you and I are still coping with this, we still don't fully internalize what actually needs to happen and what we should be doing.
Igor
I often listen to one podcast. It's called "Last week in AI". It's an amazing podcast, it keeps me updated on everything in the industry.
It has two co-hosts, and one of them is more of a Doomer who is anxious about AGI, and another one is more of a skeptic who says that AGI is far off and we will probably deal with all the challenges because we make a lot of progress on AI safety. And at one point, when co-hosts discussed one of the recent models, which is very good with most tasks that researchers throw at them, this skeptic said that if someone asked him five years ago, is this model an AGI, he would say yes. And the same person says that we are no way near an AGI. In my opinion there's a very interesting contradiction. It sounds interesting, so interesting to explore. I'm not sure about how he explains himself why it's not an AGI, but I find it very interesting to untangle.
Joep
Maybe there is some degree of confidence that people get from pushing the deadline further. I mean, you and I have short timelines, I think stuff can go wrong in the next few months, or next few years.
I have a broad range of uncertainty, but I definitely believe that it could happen very soon. I can imagine that. The process of internalizing that it could happen very soon is the most difficult thing. And you know that there are these subconscious pressures basically pushing away the danger further into the future. People say to themselves "It is not AGI, it can't be happening right now, it won't scale" I think the only rational way of looking at this is it is very, very difficult to predict. Things can move way faster than virtually anyone predicts. So there are huge error bars. And we should err on the side of caution. I don't think having a 1% chance of being wrong is worth it.
Igor
Talking about coping mechanisms, I recall Mustafa Suleyman, the CEO of Inflection AI. I listened to him on a podcast recently. He said that the containment of AI is impossible since powerful models will become very capable soon and they will become more and more affordable, and there will be no way to control it as tightly as for example, nuclear power and nuclear weapons. But at the same time, he said, containment must be possible. So, It's impossible, but it must be possible. I understand the logic behind this but it seems to me that there is an element of a coping mechanism, like it's impossible, but it has to be possible.
Joep
There are also a lot of contradictory remarks, for example, by Sam Altman talking about when to pause AI development for example. He stated at some point that we should pause and that moment is when we are seeing unexpected advancements in AI models after training them. And in other interviews he has clearly stated that while they train current AI models, they see unexpected advancements, and they don't exactly know what will come out of it. Right now.
They've started training GPT-5, which I assume will be a lot bigger than GPT-4 and it will have some new capabilities. And I don't have a feeling that they're very confident in how to contain it.
Igor
I agree with you. There is so much psychology in AI safety. I can compare it to politics. There is so much complexity in political discourse, but AI safety makes everything even more complicated. For example, if someone who you don't like becomes a president, then they might suck as a president, but you will be alive, people around you will be alive too, and this presidency won't be the end of the world. But with AI, stakes are higher. And if stakes are so high, then these coping mechanisms and biases become even more pronounced.
Joep
So you earlier mentioned that your number one reason to be concerned about a future is basically suffering risks.
Igor
No it's not mine, I talked about another person. The thing that bothers me the most is what will be after my death. Maybe after my death my consciousness will dissolve into nothing. But for me it's not the worst thing. The worst thing I'm afraid of is that after my death my conscience will be trapped in some limbo, and I will be there alone for all eternity. This is the thing I'm the most scared about. Am I afraid of AI suffering risks? I think yes, I don't want to be brain-wired by an AI and be alive through tens of thousands of years of suffering. That's horrible. I would probably prefer for my consciousness to dissolve.
Joep
A weird thing is we may be at a point where an actual hell could be constructed, right? It is, it is kind of fucked up.
Igor
I think, It's a good place to end our talk. It was a pleasure talking to you.
Joep
For me too. Thank you.