Reflections on My Own Missing Mood

If these paths are viable, I desire to believe that they are viable.

If these paths are nonviable, I desire to believe that they are nonviable.

Does it do any good, to take well-meaning optimistic suggestions seriously, if they will in fact clearly not work? Obviously, if they will work, by all means we should discover that, because knowing which of those paths, if any, is the most likely to work is galactically important. But I don't think they've been dismissed just because people thought the optimists needed to be taken down a peg. Reality does not owe us a reason for optimism.

Generally when people are optimistic about one of those paths, it is not because they've given it deep thought and think that this is a viable approach, it is because they are not aware of the voluminous debate and reasons to believe that it will not work, at all. And inasmuch as they insist on that path in the face of these arguments, it is often because they are lacking in security mindset - they are "looking for ways that things could work out", without considering how plausible or actionable each step on that path would actually be. If that's the mode they're in, then I don't see how encouraging their optimism will help the problem.

Is the argument that any effort spent on any of those paths is worthwhile compared to thinking that nothing can be done?

edit: Of course, misplaced pessimism is just as disastrous. And on rereading, was that your argument? Sorry if I reacted to something you didn't say. If that's the take, I agree fully. If one of those approaches is in fact viable, misplaced pessimism is just as destructive. I just think that the crux there is whether or not it is, in fact, viable - and how to discover that.

[-]Jeff Rose4y120

I have a reasonably low value for p(Doom). I also think these approaches (to the extent they are courses of action) are not really viable. However, as long as they don't increase the probability of p(Doom) its fine to pursue them. Two important considerations here: an unviable approach may still slightly reduce p(Doom) or delay Doom and the resources used for unviable approaches don't necessarily detract from the resources used for viable approaches.

For example, "we'll pressure corporations to take these problems seriously", while unviable as a solution will tend to marginally reduce the amount of money flowing to AI research, marginally increase the degree to which AI researchers have to consider AI risk and marginally enhance resources focused on AI risk. Resources used in pressuring corporations are unlikely to have any effect which increases AI risk. So, while this is unviable, in the absence of a viable strategy suggesting the pursuit of this seems slightly positive.

[-]FeepingCreature4y70

Resources used in pressuring corporations are unlikely to have any effect which increases AI risk.

Devil's advocate: If this unevenly delays corporations sensitive to public concerns, and those are also corporations taking alignment at least somewhat seriously, we get a later but less safe takeoff. Though this goes for almost any intervention, including to some extent regulatory.

[-]Lone Pine4y50

Yes. An example of how this could go disastrously wrong is if US research gets regulated but Chinese research continues apace, and China ends up winning the race with a particularly unsafe AGI.

[-]Lone Pine4y30

What is p(DOOM | Action), in your view?

[-]FeepingCreature4y40

I'm actually optimistic about prosaic alignment for a takeoff driven by language models. But I don't know what the opportunity for action is there - I expect Deepmind to trigger the singularity, and they're famously opaque. Call it 15% chance of not-doom, action or no action. To be clear, I think action is possible, but I don't know who would do it or what form it would take. Convince OpenAI and race Deepmind to a working prototype? This is exactly the scenario we hoped to not be in...

edit: I think possibly step 1 is to prove that Transformers can scale to AGI. Find a list of remaining problems and knock them down - preferably in toy environments with weaker models. The difficulty is obviously demonstrating danger without instantiating it. Create a fire alarm, somehow. The hard part for judgment on this action is that it both helps and harms.

edit: Regulatory action may buy us a few years! I don't see how we can get it though.

[-]Lone Pine4y40

I'm definitely on board with prosaic alignment via language models. There's a few different projects I've seen in this community related to that approach, including ELK and the project to teach a GPT-3-like model to produce violence-free stories. I definitely think these are good things.

I don't understand why you would want to spend any effort proving that transformers could scale to AGI. Either they can or they can't. If they can, then proving that they can will only accelerate the problem. If they can't, then prosaic alignment will turn out to be a waste of time, but only in the sense that every lightbulb Edison tested was a waste of time (except for the one that worked). This is what I mean by a shotgun approach.

[-]FeepingCreature4y50

I don’t understand why you would want to spend any effort proving that transformers could scale to AGI.

The point would be to try and create common knowledge that they can. Otherwise, for any "we decided to not do X", someone else will try doing X, and the problem remains.

Humanity is already taking a shotgun approach to unaligned AGI. Shotgunning safety is viable and important, but I think it's more urgent to prevent the first shotgun from hitting an artery. Demonstrating AGI viability in this analogy is shotgunning a pig in the town square, to prove to everyone that the guns we are building can in fact kill.

We want safety to have as many pellets in flight as possible. But we want unaligned AGI to have as few pellets in flight as possible. (Preferably none.)

[-]Shmi4y*220

Just a side note that climate change would not lead to the biosphere collapse even in the worst case. Earth is pretty cold and low on life now compared to 55 million years ago, when it was hotter than the worst projections of the climate models. If anything, global warming will be good for the biosphere, though maybe not so much for humans.

[-]Lone Pine4y60

I agree with you, but we should acknowledge that the problem is more serious than just how hot it is -- the problem is industrial pollution, habitat destruction, microplastics, ocean acidification, etc. I see the biosphere as anti-fragile and resilient, but the environmentalist's fear is that there is some tipping point that nature cannot recover from. And even if p(Dead Earth) is low, we wouldn't want to exit the twenty first century with half as much biomass as we started with. That's not a good thing.

[-]jmh4y50

Why would having half the biomass necessarily be a bad thing from a system level view? Is that the point the entire ecosystem, or at least the humans/human civilization, spiral into a dead planet?

[-]Lone Pine4y00

Would you rather have more biomass or less?

[-]antanaclasis4y100

I see it in a similar light to “would you rather have more or fewer cells in your body?”. If you made me choose I probably would rather have more, but only insofar as having fewer might be associated with certain bad things (e.g. losing a limb).

Correspondingly, I don’t care intrinsically about e.g. how much algae exists except insofar as that amount being too high or low might cause problems in things I actually care about (such as human lives).

[-]Ben Pace4y100

Here's a simple story for the emotional state of a pessimist. Not sure it's true but it's a story I have in mind.

In disaster movies, everything happens all at once. In reality climate disasters are more gradual, tsunamis and hurricanes and so on happen at different places at different times, but suppose for this story the disasters caused by climate change are all happening in a single period stretching on a few months. 100s of millions of people will be dead in a couple of months.

It's about a week before the disaster hits, and it's been shown very clearly (to most people's satisfaction) that it's about to hit.

Some people say "it is clearly time to do something, let us gather the troops and make lots of great plans". And that's true, in the world where there's work to be done. But it's also true that... the right time to solve this problem was 20+ years prior. Then there were real opportunities you could take that would have prevented the entire thing. Today there is no action that can prevent it, even though the threat is in the future. Fortunately it is not an existential threat, and there is triage to be done, and marginal effort can reliably save lives. But the time to solve it was decades ago, the opportunities lay unused, they were lost to time, and the catastrophe is here.

Returning to the reality we ourselves live in, insofar as someone is like "Huh, seems like these companies are producing what will soon be wildly lucrative products, and the search mechanism of gradient descent seems like it has the potential to produce a generally capable agent, and given the competitive nature of our economy and lack of political coordination it doesn't seem like we can stop lots of people from just pouring in money and having high-IQ people scale up their search mechanisms and potentially build an AGI — this is a big problem" I would say, hopefully without fear of contradiction, that the best time to avoid this catastrophe was decades ago, and that at this point the opportunities to avoid being in this situation have too been lost.

(When reading the description above, I feel that Quirinus Quirrell would say the right thing to do is not get into that situation, and head it off way earlier in the timeline, because it sounds like an extremely doomed situation that required a miracle to avert catastrophe, and plans that successfully cause a miracle to happen are pretty fragile and typically fail.)

I'm not saying this story about being a week before disaster is fully analogous to the one we're in, but it's one I've been thinking about, and I think has been somewhat missing when people talk about the emotional state of pessimists.

[-]Lone Pine4y90

It's interesting that climate change keeps coming up as an analogy. Climate change activists have always had a very specific path that they favored: global government regulation (or tax) of carbon emissions. As the years went on and no effective regulation emerged, they grew increasingly frustrated and pessimistic. (Similarly, alignment pessimists are frustrated that their preferred path, technical alignment, is not paying off.) In my view, even though they did not get what specifically wanted, climate change activist's work did pay off in the form of Obama's "All of the Above" energy strategy (shotgun approach), which led to solar innovation and scaled up manufacturing, which led to dramatically falling prices for solar energy, which is now leading to fossil fuels being out-competed in the market.

[-]Vaniver4y70

In disaster movies, everything happens all at once.

And, in general, I think I've seen very few movies where the villainous plan is thwarted before it succeeds, instead of afterwards; the villain gets the MacGuffin that gives him ultimate power, and then the heroes have to defeat him anyways, as opposed to that being the "well, now we've lost" moment.

[-]Vaniver4y50

In an attempt to make optimists take the problem more seriously, pessimists in this debate (and in particular EY) convinced themselves that all these paths are non-viable.

I feel like this is really not the correct causal story? My sense is that 'pessimists' evaluated those strategies hoping to find gold, and concluded they were just pyrite, and it was basically not motivated cognition, like this seems to imply.

[-]Joe Collman4y40

A few thoughts:

"p(DOOM | Action)" seems too coarse a view from which to draw any sensible conclusions. The utility of pessimism (or realism, from some perspectives) is that it informs decisions about which action to take. It's not ok that we're simply doing something.
Yes, the pessimistic view has a cost, but that needs to be weighed against the (potential) upside of finding more promising approaches - and realising that p(DOOM | we did things that felt helpful) > p(DOOM | we did things that may work).
Similarly, convincing a researcher that a [doomed-to-the-point-of-uselessness] approach is doomed is likely to be net positive, since they'll devote time to more worthwhile pursuits, likely increasing p(AWESOME...). Dangers being:
1. Over-confidence that it's doomed.
2. Throwing out wrong-but-useful ideas prematurely.
3. A few researchers may be discouraged to the point of quitting.
In general, the best action to take will depend on p(Doom); if researchers have very different p(Doom) estimates, it should come as no surprise that they'll disagree on the best course of action at many levels.

Specifically, the upside/downside of 'pessimism' will relate to our initial p(Doom). If p(Doom) starts out at >99.9%, then p(Doom | some vaguely plausible kind of action) is likely also >99%. High values of p(Doom) result from the problem's being fundamentally difficult - with answers that may be hidden in a tiny region of a high-dimensional space.
If the problem is five times as hard as we may reasonably hope, then a shotgun may work.
If the problem is five orders of magnitude harder, then we want a laser.
1. Here I emphasize that we'd eventually need the laser. By all means use a shotgun to locate promising approaches, but if the problem is hard we can't just babble: we need to prune hard.
2. It occurs to me that it may be useful to have more community-level processes/norms to signal babbling, so that mistargeted pruning doesn't impede idea generation.
Pivotal act talk does seem to be disastrous PR, but I think that's the only sense in which it's negative. By Critch's standards, the arguments in that post are weak. When thinking about this, it's important to be clear what we're saying on the facts, and what adjustments we're making for PR considerations. React too much to the negative PR issue (which I agree is real), and there's the danger of engaging in wishful thinking.
1. Should we try to do tackle the extremely hard coordination problems? Possibly - but we shouldn't be banking on such approaches working. (counterfactual resource use deserves consideration)
2. We should be careful of any PR-motivated adaptations to discourse, since they're likely to shape thinking in unhelpful ways. (even just talking less about X, rather than actively making misleading claims about X may well have a negative impact on epistemics)

[-]Lone Pine4y20

I agree with pretty much all of this, thanks for the clarification. I think the problem is that we need both a laser and a shotgun. We need a laser shotgun. It’s analogous to missile defense, where each individual reentry vehicle is very difficult to shoot down, and we have to shoot down every one or else we’ve failed. I appreciate now why it’s such a hard problem.

I definitely think there needs to be a better introductory material for alignment, particularly regarding what approaches have been investigated and what names they go by. I know the community already knows there is a problem with public communication so I’m not saying anything novel here.

I’d like to see a kind of “choose your own adventure” format webpage where lay people can select by multiple choice how they would go about addressing the problem, and the webpage will explain in not-too-technical language how much progress has been made and where to go for more information. I’m thinking about putting something like this together, I’ll use a wiki format since I don’t have all the answers of course.

[-]Joe Collman4y*40

Agreed on the introductory material (and various other things in that general direction).

I'm not clear on the case for public communication. We don't want lay people to have strong and wildly inaccurate opinions, but it seems unachievable to broadly remedy the "wildly inaccurate" part. I haven't seen a case made that broad public engagement can help (but I haven't looked hard - has such a case been made?).

I don't see what the lay public is supposed to do with the information - even supposing they had a broadly accurate high-level picture of the situation. My parents fit this description, and thus far it's not clear to me what I'd want them to do. It's easy to imagine negative outcomes (e.g. via politicization), and hard to imagine realistic positive ones (e.g. if 80% of people worldwide had an accurate picture, perhaps that would help, but it's not going to happen).

It does seem important to communicate effectively to a somewhat narrower audience. The smartest 1% of people being well informed seems likely to be positive, but even here there are pitfalls in aiming to achieve this - e.g. if you get many people as far as [AGI will soon be hugely powerful] but they don't understand the complexities of the [...and hugely dangerous] part, then you can inadvertently channel more resources into a race. (still net positive though, I'd assume EDIT: the being-well-informed part, that is)

I'll grant that there's an argument along the lines of:
Perhaps we don't know what we'd want the public to do, but there may come a time when we do know. If time is short it may then be too late to do the public education necessary. Therefore we should start now.

I don't think I buy this. The potential downsides of politicization seem great, and it's hard to think of many plausible "...and then public pressure to do X saved the day!" scenarios.

This is very much not my area, so perhaps there are good arguments I haven't seen.
I'd just warn against reasoning of the form "Obviously we want to get the public engaged and well-informed, so...". The "engaged" part is far from obvious (in particular we won't be able to achieve [engaged if and only if well-informed]).

[-]Lone Pine4y20

I’ll have to think about this more, but if you compare this to other things like climate change or gain of function research, it’s a very strange movement indeed that doesn’t want to spread awareness. Note that I agree that regulation is probably more likely to be harmful than helpful, and also note that I personally despise political activists, so I’m not someone normally amenable to “consciousness raising.” It’s just unusual to be against it for something you care about.

[-]Joe Collman4y10

Absolutely - but it's a strange situation in many respects.

It may be that spreading awareness is positive, but I don't think standard arguments translate directly. There's also irreversibility to consider: err on the side of not spreading info, and you can spread it later (so long as there's time); you can't easily unspread it.

More generally, I think for most movements we should ask ourselves, "How much worse than the status-quo can things plausibly get?".

For gain-of-function research, we'd need to consider outcomes where the debate gets huge focus, but the sensible side loses (e.g. through the public seeing gof as the only way to prevent future pandemics). This seems unlikely, since I think there are good common-sense arguments against gof at most levels of detail.

For climate change, it's less clear to me: there seem many plausible ways for things to have gotten worse. Essentially because the only clear conclusion is "something must be done", but there's quite a bit less clarity about what - or at least there should be less clarity. (e.g. to the extent that direct climate-positive actions have negative economic consequences, to what extent are there downstream negative-climate impacts; I have no idea, but I'm sure it's a complex situation)

For AGI, I find it easy to imagine making-things-worse and hard to see plausible routes to making-things-better.

Even the expand-the-field upside needs to be approached with caution. This might be better thought of as something like [expand the field while maintaining/improving the average level of understanding]. Currently, most people who bump into AI safety/alignment will quickly find sources discussing the most important problems. If we expanded the field 100x overnight, then it becomes plausible that most new people don't focus on the real problems. (e.g. it's easy enough only to notice the outer-alignment side of things)

Unless time is very short, I'd expect doubling the field each year works out better than 5x each year - because all else would not be equal. (I have no good sense what the best expansion rate or mechanism is - just that it's not [expand as fast as possible])

But perhaps I'm conflating [aware of the problem] with [actively working on the problem] a bit much. Might not be a bad idea to have large amounts of smart people aware of the problem overnight.

[-]Vaniver4y30

Amusingly, MacKay often redraws the cartoon with the edits, so having 'biodiversity collapse' (the original purple wave) being surpassed by a fifth AGI wave seems like it reflects the situation more (tho, ofc, it requires more art effort ;) ).

[-]JBlack4y20

P(DOOM from AGI | No Action) = 0. We will not die by AGI if we take no action to develop AGI, at least in less than astronomical timescales.

This is obviously not what will happen. Many groups will rush to develop AGI in the fastest and most powerful manner that they can possibly arrange, with safety largely treated as a public relations exercise and not a serious issue. My personal feeling is that any AGI developed in the next 30 years will very likely (p > 0.9) be sufficiently misaligned that humanity would be extinguished if it were scaled up to superintelligence.

My hopes for our species rest on one of three foundations: that I'm hopelessly miscalibrated about how hard alignment is, that AGI takes so much longer to develop that alignment is adequately solved, or that scaling to superintelligence is for some reason very much harder than developing AGI.

The second seems less likely each year, and the third seems an awfully flimsy hope. Both algorithm advances and hardware capabilities seem to offer substantial improvements well into the superhuman realm for all of the many narrow tasks that machines can now do better than humans, and I don't see any reason to expect that they cannot scale at least as well for more general ones once AGI is achieved.

I do have some other outlier possibilities for hope, but they seem more based in fiction than reality.

So that mainly leaves the first hope, that alignment is actually a lot simpler and easier than it appears to me. It is at least worth checking.

[-]Flaglandbase4y10

From all the stuff I've been reading here lately, it seems the only solution would be total worldwide oversight control of all activities.

LESSWRONG
LW

LESSWRONG
LW

53

Reflections on My Own Missing Mood

53

53

Crisis Fatigue

Pessimism is not Productive

Self-Fulfilling Prophecies

We Need a Shotgun Approach

Moving Forward