There was conversation on Facebook over an argument that any sufficiently complex system, whether a human, a society, or an AGI, will be unable to pursue a unified goal due to internal conflict among its parts, and that this should make us less worried about "one paperclipper"-style AI FOOM scenarios. Here's a somewhat edited and expanded version of my response:
1) yes this is a very real issue
2) yet as others pointed out, humans and organizations are still able to largely act as if they had unified goals, even if they often also act contrary to those goals
When there is a train, plane, or bus crash, it's newsworthy: it doesn't happen very often, lots of lives at stake, lots of people are interested in it. Multiple news outlets will send out reporters, and we will hear a lot of details. On the other hand, a car crash does not get this treatment unless there is something unusual about it like a driverless car or an already newsworthy person involved.
The effects are not great: while driving is relatively dangerous, both to the occupants and people outside, our sense of danger and impact is poorly calibrated by the news we read. My guess is that most people's intuitive sense of the danger of cars versus trains, planes, and buses has been distorted by this coverage, where most people, say, do not expect buses to be >16x safer than cars. This also...
If it’s worth saying, but not worth its own post, here's a place to put it.
If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.
If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.
If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.
Each person is special. Amanda is in a class of her own, Fred is in a class of his own. Amanda has many different properties: where she came from, what she looks like, what behaviors she habitually does, what she knows and doesn't know, what her plans are, how she speaks, how she would behave if given power, etc. Fred has his own versions of those properties. They overlap partially but not completely with Amanda's properties.
When conflict arises, sides must be chosen . One side has people with one set of properties, the other side has people with the negation of those properties. Sides can be chosen out loud and explicitly, or in silence and implicitly.
Sometimes one side is more coordinated with itself than the other...
This is a linkpost for https://outsidetheasylum.blog/understanding-subjective-probabilities/. It's intended as an introduction to practical Bayesian probability for those who are skeptical of the notion. I plan to keep the primary link up to date with improvements and corrections, but won't do the same with this LessWrong post, so see there for the most recent version.
Any time a controversial prediction about the future comes up, there's a type of interaction that's pretty much guaranteed to happen.
Alice: "I think this thing is 20% likely to occur."
Bob: "Huh? How could you know that. You just made that number up!".
Or in Twitterese:
That is, any time someone attempts to provide a specific numerical probability on a future event, they'll inundated with claims that that number is meaningless. Is this true? Does it make...
confidence level: I am a physicist, not a biologist, so don’t take this the account of a domain level expert. But this is really basic stuff, and is very easy to verify.
Edit: I have added a few revisions and included a fact check of this post by an organic chemist. You can also read the comments on the EA forum to see Yudkowsky's response.
Recently I encountered a scientific claim about biology, made by Eliezer Yudkowsky. I searched around for the source of the claim, and found that he has been repeating versions of the claim for over a decade and a half, including in “the sequences” and his TED talk. In recent years, this claim has primarily been used as an argument for why an AGI attack...
There were several responses to What I Would Do If I Were Working On AI Governance which focused on the liability section, and had similar criticisms. In particular, I’ll focus on this snippet as a good representative:
Making cars (or ladders or knives or printing presses or...) "robust to misuse", as you put it, is not the manufacturer's job.
The commenter calls manufacturer liability for misuse “an absurd overreach which ignores people's agency in using the products they purchase”. Years ago I would have agreed with that; it’s an intuitive and natural view, especially for those of us with libertarian tendencies. But today I disagree, and claim that that’s basically not the right way to think about product liability, in general.
With that motivation in mind: this post lays out some...