Suppose that 1% of the world’s resources are controlled by unaligned AI, and 99% of the world’s resources are controlled by humans. We might hope that at least 99% of the universe’s resources end up being used for stuff-humans-like (in expectation).

Jessica Taylor argued for this conclusion in Strategies for Coalitions in Unit-Sum Games: if the humans divide into 99 groups each of which acquires influence as effectively as the unaligned AI, then by symmetry each group should end, up with as much influence as the AI, i.e. they should end up with 99% of the influence.

This argument rests on what I... (Read more)

I've been thinking more about partial agency. I want to expand on some issues brought up in the comments to my previous post, and on other complications which I've been thinking about. But for now, a more informal parable. (Mainly because this is easier to write than my more technical thoughts.)

This relates to oracle AI and to inner optimizers, but my focus is a little different.

1

Suppose you are designing a new invention, a predict-o-matic. It is a wonderous machine which will predict everything for us: weather, politics, the newest advances in quantum physics, you name it. The machi... (Read more)

Below I will present a (small but qualitative ) list of those that I think are some of the best sites/blog that a human being can find on the world wide web.

The main criterion I used to draw up the list was to consider how the websites promote the dissemination of knowledge among people and how, over the course of time, they have helped me both with regards to work and in terms of intellectual self-formation. The order in which they are listed is not to be considered restrictive ( except perhaps for the first two ).

Please feel free to criticize the catalog (as long as the criticisms are r... (Read more)

"Gradient hacking" is a term I've been using recently to describe the phenomenon wherein a deceptively aligned mesa-optimizer might be able to purposefully act in ways which cause gradient descent to update it in a particular way. In Risks from Learned Optimization, we included the following footnote to reflect this possibility:

Furthermore, a deceptively aligned mesa-optimizer would be incentivized to cause there to be a systematic bias in the direction of preventing the base optimizer from modifying its mesa-objective. Thus, in the context of a local optimization process, a deceptive mesa-o

Introduction

Epistemic spot checks are a series in which I select claims from the first few chapters of a book and investigate them for accuracy, to determine if a book is worth my time. This month’s subject is The Fall of Rome, by Bryan Ward-Perkins, which advocates for the view that Rome fell, and it was probably a military problem.

Like August’s The Fate of Rome, this spot check was done as part of a collaboration with Parallel Forecasting and Foretold, which means that instead of resolving a claim as true or false, I give a confidence distribution of what I think I would answer if I spent 1... (Read more)

Introduction

Internal Family Systems (IFS) is a psychotherapy school/technique/model which lends itself particularly well for being used alone or with a peer. For years, I had noticed that many of the kinds of people who put in a lot of work into developing their emotional and communication skills, some within the rationalist community and some outside it, kept mentioning IFS.

So I looked at the Wikipedia page about the IFS model, and bounced off, since it sounded like nonsense to me. Then someone brought it up again, and I thought that maybe I should reconsider. So I looked at the WP page again... (Read more)

Let me clarify what I mean when I say that math consists of nouns and verbs. Think about elementary school mathematics like addition and subtraction. What you learn to do is take a bunch of nouns—1, 2, 3, etc.—and a bunch of verbs—addition, subtraction—and make sentences. “1 + 2 = 3.”

When you make a sentence like that, what you're doing is taking an object, 1, and observing how it changes when it interacts—specifically, adds—with another object, 2. You observe that becomes a 3. Just like how you can observe a person (object) bump the... (Read more)

Cross-posted to my personal blog.

For a while now, I've been using "(a)" notation to denote archived versions of linked pages. This is a small effort towards creating Long Content (a) – content that has a lifespan of decades or centuries, rather than months or years.

I think basically anyone whose writing includes links to other work should include archived links alongside the original hyperlinks, if the writing is intended to be long-lived. (And if you're not trying to write long-lived content, what are you doing, even?)

I was happy to see Zuck (a) & Guzey (a) usin... (Read more)

• If it’s worth saying, but not worth its own post, here's a place to put it.
• You can also make a shortform post.
• And, if you are new to LessWrong, here's the place to introduce yourself.
• Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ.

The Open Thread sequence is here.

This is a response to Abram's The Parable of Predict-O-Matic, but you probably don't need to read Abram's post to understand mine. While writing this, I thought of a way in which I think things could wrong with dualist Predict-O-Matic, which I plan to post in about a week. I'm offering a $100 prize to the first commenter who's able to explain how things might go wrong in a sufficiently crisp way before I make my follow-up post.

Dualism

Currently, machine learning algorithms are essentially "Cartesian dualists" when it comes to themselves and their environment. (Not a philosophy major -- let

I’m happy to announce a semi-public beta of Foretold.io for the EA/LessWrong community. I’ve spent much of the last year working on coding & development, with lots of help by Jacob Lagerros on product and scoring design. Special thanks to the Long-Term Future Fund and it’s donors, who’s contribution to the project helped us to hire contractors to do much of the engineering & design.

You can use Foretold.io right away by following this link. Currently public activity is only shown to logged in users, but I expect that to be opened up over the next few weeks. There are currently only a fe

Notes from the Salon

ETA: This is a write-up of some interesting points raised at an in-person rationalist meetup on October 6th, 2019. It is not intended to be a comprehensive overview of the topic. It is conventional for attendees to do all the suggested readings before salon starts, so some parts of the write-up might not make sense without that context.

Assigned readings: Siderea on Class, Thoughts on the "STEM" class

Economic Class vs. Social Class

Economic class and social class are not the same thing. The two are decoupled, but only partly. You can be wealthy but lower class (e.g.... (Read more)

What do balance and alignment mean with respect to the human body?

This is an introduction piece for my hypothesis of human health and movement.

The anatomical information presented here should be easily verifiable.

Alignment and Balance.

A couple of definitions for balanced:

1. Different parts of something exist in equal or correct amounts.

2. A state of equilibrium, being in harmonious arrangement.

Alignment has many definitions, the two I feel most relevant to "body alignment" are:

1. Arrangement in a straight line.

2. Arranged in the correct relative positions.

The Median Pla

[Epistemic status: Sharing current impressions in a quick, simplified way in case others have details to add or have a more illuminating account. Medium-confidence that this is one of the most important parts of the story.]

Here's my current sense of how we ended up in this weird world where:

• I still intermittently run into people who claim that there's no such thing as reality or truth;
• a lot of 20th-century psychologists made a habit of saying things like 'minds don't exist, only behaviors';
• a lot of 20th-century physicists made a habit of saying things like 'quarks

This follows an introduction to the median plane and our midline anatomy as the references for alignment and balance of the human body.

Now consider the moving body and the muscles that you use.

Do you have a full range of natural movement?

A Full Range of Natural Movement.

A full range of natural movement is what the body should be able to do. The body's full potential. Not what you have got used to!

Movement should be smooth and controlled, the body stable as it glides through an almost infinite number of potential positions.

. ​

To have a full range of natural movement, the bo... (Read more)

We used to make land. We built long wharves for docking ships, and then over time filled in the areas between them. Later we built up mudflats wholesale to make even larger areas. Here's a map of Boston showing how much of the land wasn't previously dry:

(Map reproduction courtesy of the Norman B. Leventhal Map & Education Center at the Boston Public Library)

In expensive areas, converting wetlands and shallow water into usable land is a very good thing on balance, and we should start doing it again. To take a specific example, we should make land out of the San Francisco B... (Read more)

Reply to Extreme Rationality: It's Not That Great, Extreme Rationality: It could Be Great, the Craft and the Community and Why Don't Rationalists Win?

I’m going to say something which might be extremely obvious in hindsight:

If LessWrong had originally been targeted at and introduced to an audience of competent business people and self-improvement health buffs instead of an audience of STEM specialists and Harry Potter fans, things would have been drastically different. Rationalists would be winning.

Right now, rationalists aren’t winning. Rationality helps us choose which charities to ... (Read more)

I. The question of confidence

Should one hold strong opinions? Some say yes. Some say that while it’s hard to tell, it tentatively seems pretty bad (probably).

A quick review of purported or plausible pros:

1. Strong opinions lend themselves to revision:
   1. Nothing will surprise you into updating your opinion if you thought that anything could happen. A perfect Bayesian might be able to deal with myriad subtle updates to vast uncertainties, but a human is more likely to notice a red cupcake if they have claimed that cupcakes are never red. (Arguably—some would say having opinions makes you less able

[A draft section from a longer piece I am writing on prediction and forecasting. Epistemic Status: I don't know what I am missing, and I am filled with doubt and uncertainty.]

If the notion of professional forecasters disturbs you in your sleep, and you toss and turn worrying about the blight of experts brooding upon the world, perhaps the golden light of distributed information systems have peaked out from beyond these darkest visions, and you have hope for the wisdom of crowds.

Prediction markets aggregate information by incentivizing predictors to place bets on the outcomes of well-defin... (Read more)