Michele Campolo

With enough knowledge, any conscious agent acts morally

Thank you for this suggestion, I appreciate it! I’ve read the review I found here and it seems that parts of that account of ethics overlap with some ideas I’ve discussed in the post, in particular the idea of considering the point of view of all conscious (rational) agents. Maybe I’ll read the entire book if I decide to reformulate the argument of the post in a different way, which is something I was already thinking about.

How did you find that book?

Replying toWith enough knowledge, any conscious agent acts morally

With enough knowledge, any conscious agent acts morally

This type of argument has the problem that other peoples negative experiences aren't directly motivating in the way that yours are...there's a gap between bad-for-me and morally-wrong.

What type of argument is my argument, from your perspective? I also think that there is a gap between bad-for-me and bad-for-others. But both can affect action, as it happens in the thought experiment in the post.

To say that something is morally-wrong is to say that I have some obligation or motivation to do something about.

I use a different working definition in the argument. And working definitions aside, more generally, I think morality is about what is important, better/worse, worth doing, worth guiding action, which is... (read more)

Replying toWith enough knowledge, any conscious agent acts morally

With enough knowledge, any conscious agent acts morally

Consider this stamp collector construction: It sends and receives internet data, it has a magically accurate model of reality, it calculates how many stamps would result from each sequence of outputs, and then it outputs the one that results in the most stamps.

I’m not sure why you left out the “conscious agent” part, which is the fundamental premise of the argument. If you are describing something like a giant (artificial) neural network optimised to output actions that maximise stamps while receiving input data about the current state of the world, that seems possible to me and the argument is not about that kind of AI. You can also have a look at... (read 444 more words →)

Replying toDoing good... best?

But you were arguing for them, weren't you? It is the arguments that fail to convince me. I was not treating these as bald assertions.

No, I don’t argue that “a sufficiently intelligent intelligence would experience valence, or for that matter that it would necessarily even be conscious”. I think those statements are false.

Replying toDoing good... best?

Hey I think your comment is slightly misleading:

I don't see a reason to suppose that a sufficiently intelligent intelligence would experience valence, or for that matter that it would necessarily even be conscious

I do not make those assumptions.

nor, if conscious, that it would value the happiness of other conscious entities

I don’t suppose that either, I give an argument for that (in the longer post).

Anyway:

I am not convinced by the longer post either

I’m not surprised: I don’t expect that my argument will move masses of people who are convinced of the opposite claim, but that someone who is uncertain and open-minded can read my argument and maybe find something useful in it and/or a reason to update their beliefs. That’s also why I wrote that the practical implications for AI are an important part of that post, and why I made some predictions instead of focusing just on philosophy.

Replying toDoing good... best?

One more reason for AI capable of independent moral reasoning: alignment itself and cause prioritisation

I am not assuming a specific metaethical position, I’m just taking into account that something like moral naturalism could be correct. If you are interested in this kind of stuff, you can have a look at this longer post.

Speaking of this, I am not sure it is always a good idea to map these discussions into specific metaethical positions, because it can make updating one’s beliefs more difficult, in my opinion. To put it simply, if you’ve told yourself that you are e.g. a moral naturalist for the last ten years, it can be very difficult to read some new piece of philosophy arguing for a different position (maybe even opposite), then rationally update and tell yourself something like: “Well, I guess I’ve just been wrong for all this time! Now I’m a ___ (new position)”

6mo

Posted also on the EA Forum.

By “independent” I mean an AI to which an external observer may attribute freedom of thought, or something similar to it. You can think of it as an AI that is not too biased by what its designers or programmers think it’s good or right; an AI that could tell us something new, something we don’t know yet about ethics.

I’ve already given various reasons why having such an AI would be valuable. Here I want to focus on a reason I haven’t talked about yet, which is the importance of AI alignment itself — and cause prioritisation more generally.

It is a not-too-informed opinion of mine that we... (read 847 more words →)

-3

With enough knowledge, any conscious agent acts morally

6mo

Posted also on the EA Forum.

I try to communicate the point of this sequence of posts in an intuitive way.

Everyone wants to do good; however, sometimes people don’t take into consideration that they might be wrong about what is good. This is particularly important if, for example, one’s actions will affect many individuals, or have long-term repercussions. Think about the Crusades: it’s likely that the Christians who initiated them believed they were doing good.

I bet that you are not planning to start a holy war tomorrow while stressing over whether it’s a good idea or not. Still, wouldn’t it be nice if there was a way to improve our understanding of what... (read 595 more words →)

-1

Four visions of Transformative AI success

6mo

Posted also on the EA Forum.

I think that:

For any conscious agent A, there is some knowledge such that A acts morally if A has that knowledge.

Or, less formally:

With enough knowledge, any conscious agent acts morally.

What a bold statement!

This post clarifies what I mean with the terms above, makes an argument supporting the statement, then discusses the implications for machine ethics and AI alignment — in simpler words, how to create artificial agents that do good things instead of bad things.

Warning: this is speculative! The argument I give is far from being the ultimate proof that will settle all discussions around this topic. But the implications for AI are important and they rely on... (read 10551 more words →)

-2

Replying toFree agents

Michele Campolo2y

Free agents

This story is definitely related to the post, thanks!

Replying toFour visions of Transformative AI success

Michele Campolo2y

This was a great read, thanks for writing!

Despite the unpopularity of my research on this forum, I think it's worth saying that I am also working towards Vision 2, with the caveat that autonomy in the real world (e.g. with a robotic body) or on the internet is not necessary: one could aim for an independent-thinker AI that can do what it thinks is best only by communicating via a chat interface. Depending on what this independent thinker says, different outcomes are possible, including the outcome in which most humans simply don't care about what this independent thinker advocates for, at least initially. This would be an instance of vision 2 with... (read more)

Agents that act for reasons: a thought experiment

Posted also on the EA Forum.

In Free agents I’ve given various ideas about how to design an AI that reasons like an independent thinker and reaches moral conclusions by doing so. Here I’d like to add another related idea, in the form of a short story / thought experiment.

Cursed

Somehow, you have been cursed. As a result of this unknown curse that is on you now, you are unable to have any positive or negative feeling. For example, you don’t feel pain from injuries, nothing makes you anxious or excited or sad, you can’t have fun anymore. If it helps you, imagine your visual field without colours, only with dull shades of black and... (read 681 more words →)

Replying toFree agents

Michele Campolo2y

Free agents

[Obviously this experiment could be extremely dangerous, for Free Agents significantly smarter than humans (if they were not properly contained, or managed to escape). Particularly if some of them disagreed over morality and, rather than agreeing to disagree, decided to use high-tech warfare to settle their moral disputes, before moving on to impose their moral opinions on any remaining humans.]

Labelling many different kinds of AI experiments as extremely dangerous seem to be a common trend among rationalists / LessWrongers / possibly some EA circles, but I doubt it's true or helpful. This topic itself could be the subject of a (many?) separate post(s). Here I'll focus on your specific objection:

I haven't claimed

... (read more)

Free agents

On value in humans, other animals, and AI

Posted also on the EA Forum.

2025 comment: the ideas here are still relevant and worth reading in my opinion. I'm a bit unsatisfied with the toy model: although it does a good job of describing what a kind of agent (that I call free) looks like from the outside, it leaves many details underspecified. I'm writing a follow-up post that should be more useful to someone who would like to engineer such a system.

Shameless attempt at getting your attention:

If you’ve heard of AI alignment before, this might change your perspective on it. If you come from the field of machine ethics or philosophy, this is about how to create an independent moral... (read 3900 more words →)

Criticism of the main framework in AI alignment

This will be posted also on the EA Forum, and included in a sequence containing some previous posts and other posts I'll publish this year.

Introduction

Humans think critically about values and, to a certain extent, they also act according to their values. To the average human, the difference between increasing world happiness and increasing world suffering is huge and evident, while goals such as collecting coins and collecting stamps are roughly on the same level.

It would be nice to make these differences obvious to AI as they are to us. Even though exactly copying what happens in the human mind is probably not the best strategy to design an AI that understands ethics,... (read 1436 more words →)

Some alternative AI safety research projects

Originally posted on the EA Forum for the Criticism and Red Teaming Contest.

0. Summary

AI alignment research centred around the control problem works well for futures shaped by out-of-control misaligned AI, but not that well for futures shaped by bad actors using AI. Section 1 contains a step-by-step argument for that claim. In section 2 I propose an alternative which aims at moral progress instead of direct risk reduction, and I reply to some objections. I will give technical details about the alternative at some point in the future, in section 3.

The appendix clarifies some minor ambiguities with terminology and links to other stuff.

1. Criticism of the main framework in AI alignment

1.1 What

... (read 1795 more words →)

From language to ethics by automated reasoning

These are some "alternative" (in the sense of non-mainstream) research projects or questions related to AI safety that seem both relevant and underexplored. If instead you think they aren't, let me know in the comments, and feel free to use the ideas as you want if you find them interesting.

Access-to-the-internet scenario and related topics

A potentially catastrophic scenario that appears somewhat frequently in AI safety discourse involves a smarter-than-human AI which gets unrestricted access to the internet, and then bad things happen. For example, the AI manages to persuade or bribe one or more humans so that they perform actions which have a high impact on the world.

What are the worst (i.e. with... (read 644 more words →)

What is the strongest argument you know for antirealism?

Posted also on the EA Forum.

Introduction

In Naturalism and AI alignment I claimed that we might get an aligned AI by building an agent which derives its own preferences (a partial order over world-histories) from its knowledge of the natural world.

Now I am working on a formal (i.e. mathematical or algorithmic) model of that agent. When AI capabilities will reach human level in natural language understanding, the formal model will help turn a natural language AI system that understands the world at human level of intelligence into an AI that is able to reason about ethics and possibly do EA-style research.

Here, I elaborate on the idea that an AI which reasons like a... (read 1501 more words →)