I have just published my new book on s-risks, titled Avoiding the Worst: How to Prevent a Moral Catastrophe. You can find it on Amazon, read the PDF version, or listen to the audio version.

The book is primarily aimed at longtermist effective altruists. I wrote it because I feel that s-risk prevention is a somewhat neglected priority area in the community, and because a single, comprehensive introduction to s-risks did not yet exist. My hope is that a coherent introduction will help to strengthen interest in the topic and spark further work.

Here’s a short description of the book:

From Nineteen Eighty-Four to Black Mirror, we are all familiar with the tropes of dystopian science fiction. But what if

... (read 297 more words →)

Replying toWhat failure looks like

Tobias_Baumann7y

What failure looks like

I agree with you that the "stereotyped image of AI catastrophe" is not what failure will most likely look like, and it's great to see more discussion of alternative scenarios. But why exactly should we expect that the problems you describe will be exacerbated in a future with powerful AI, compared to the state of contemporary human societies? Humans also often optimise for what's easy to measure, especially in organisations. Is the concern that current ML systems are unable to optimise hard-to-measure goals, or goals that are hard to represent in a computerised form? That is true but I think of this as a limitation of contemporary ML approaches rather than a... (read more)

Replying toThoughts on Human Models

Tobias_Baumann7y

Thoughts on Human Models

Thanks for elaborating. There seem to be two different ideas:

1), that it is a promising strategy to try and constrain early AGI capabilities and knowledge

2), that even without such constraints, a paperclipper entails a smaller risk of worst-case outcomes with large amounts of disvalue, compared to a near miss. (Brian Tomasik has also written about this.)

1) is very plausible, perhaps even obvious, though as you say it's not clear how feasible this will be. I'm not convinced of 2), even though I've heard / read many people expressing this idea. I think it's unclear what would result in more disvalue in expectation. For instance, a paperclipper would have no qualms to threaten... (read more)

Replying toThoughts on Human Models

Tobias_Baumann7y

Thoughts on Human Models

Another risk from bugs comes not from the AGI system caring incorrectly about our values, but from having inadequate security. If our values are accurately encoded in an AGI system that cares about satisfying them, they become a target for threats from other actors who can gain from manipulating the first system.

I agree that this is a serious risk, but I wouldn't categorise it as a "risk from bugs". Every actor with goals faces the possibility that other actors may attempt to gain bargaining leverage by threatening to deliberately thwart these goals. So this does not require bugs; rather, the problem arises by default for any actor (human or AI), and... (read more)

Replying toReframing Superintelligence: Comprehensive AI Services as General Intelligence

Tobias_Baumann7y

Reframing Superintelligence: Comprehensive AI Services as General Intelligence

Upvoted. I've long thought that Drexler's work is a valuable contribution to the debate that hasn't received enough attention so far, so it's great to see that this has now been published.

I am very sympathetic to the main thrust of the argument – questioning the implicit assumption that powerful AI will come in the shape of one or more unified agents that optimise the outside world according to their goals. However, given our cluelessness and the vast range of possible scenarios (e.g. ems, strong forms of biological enhancement, merging of biological and artificial intelligence, brain-computer interfaces, etc.), I find it hard to justify a very high degree of confidence in Drexler's model in particular.

Why I expect successful (narrow) alignment

Tobias_Baumann

Summary

I believe that advanced AI systems will likely be aligned with the goals of their human operators, at least in a narrow sense. I’ll give three main reasons for this:

The transition to AI may happen in a way that does not give rise to the alignment problem as it’s usually conceived of.
While work on the alignment problem appears neglected at this point, it’s likely that large amounts of resources will be used to tackle it if and when it becomes apparent that alignment is a serious problem.
Even if the previous two points do not hold, we have already come up with a couple of smart approaches that seem fairly likely to lead to successful alignment.

This argument lends some support to work on non-technical interventions like moral circle expansionor improving AI-related policy, as well as work on special aspects of AI safety like decision theory or worst-case AI safety measures.

Replying toShaping economic incentives for collaborative AGI

Tobias_Baumann8y

Shaping economic incentives for collaborative AGI

I agree that establishing a cooperative mindset in the AI / ML community is very important. I'm less sure if economic incentives or government policy are a realistic way to get there. Can you think of a precedent or example for such external incentives in other areas?

Also, collaboration between the researchers that develop AI may be just one piece of the puzzle. You could still get military arms races between nations even if most researchers are collaborative. If there are several AI systems, then we also need to ensure cooperation between these AIs, which isn't necessarily the same as cooperation between the researchers that build them.

Is Evidential Decision Theory presumptuous?

Tobias_Baumann

I recently had a conversation with a staunch defender of EDT who maintained that EDT gives the right answer in the Smoker’s Lesion and even Evidential Blackmail. I came up with the following, even more counterintuitive, thought experiment:

By doing research, you've found out that there is either

(A) only one universe or

(B) a multiverse.

You also found out that the cosmological theory has a slight influence (via different physics) on how your brain works. If (A) holds, you will likely decide to give away all your money to random strangers on the street; if there is a multiverse, you will most likely not do that. Of course, causality flows in one direction only, i.e.... (read 195 more words →)

Replying toDid EDT get it right all along? Introducing yet another medical Newcomb problem

Tobias_Baumann9y

Did EDT get it right all along? Introducing yet another medical Newcomb problem

What exactly do you think we need to specify in the Smoking Lesion?

LESSWRONG
LW

LESSWRONG
LW

Tobias_Baumann

New book on s-risks

An introduction to worst-case AI safety

Why I expect successful (narrow) alignment

A framework for thinking about AI timescales

Tobias_Baumann

New book on s-risks

Why I expect successful (narrow) alignment

Thoughts on short timelines

Mechanism Design for AI

What does the stock market tell us about AI timelines?

An introduction to worst-case AI safety

A framework for thinking about AI timescales

Tobias_Baumann

New book on s-risks

An introduction to worst-case AI safety

Why I expect successful (narrow) alignment

A framework for thinking about AI timescales

Tobias_Baumann

New book on s-risks

Why I expect successful (narrow) alignment

Thoughts on short timelines

Mechanism Design for AI

What does the stock market tell us about AI timelines?

An introduction to worst-case AI safety

A framework for thinking about AI timescales

Summary