(Disclaimer: Philip Trammell is planning to rewrite this blogpost and make it clearer and more precise, but I think there's something good in this direction. Sharing it to see what the rest of the community thinks.)

Some smart people, including some of my friends, believe that advanced AI poses a serious threat to human civilization in the near future, and that AI safety research is therefore one of the most valuable uses, if not the very most valuable use, of philanthropic talent and money. But most smart people, as far as I can judge their behavior—including some, like Mark Zuckerberg and R

In the last few months, I've gotten increasingly alarmed by leftist politics in the US, and the epistemic conditions that it operates under and is imposing wherever it gains power. (Quite possibly the conditions are just as dire on the right, but they are not as visible or salient to me, because most of the places I can easily see, either directly or through news stories, i.e., local politics in my area, academia, journalism, large corporations, seem to have been taken over by the left.)

I'm worried that my alarmism is itself based on confirmation bias, tribalism, catastrophizing, or any number

Automatically crossposted

Suppose that a kingdom contains a million peasants and a thousand nobles, and:

• Each noble makes as much as 10,000 peasants put together, such that collectively the nobles get 90% of the income.
• Each noble cares about as much about themselves as they do about all peasants put together.
• Each person’s welfare is logarithmic in their income.

Then it’s simultaneously the case that:

1. Nobles prefer to keep money for themselves rather than donate it to peasants—money is worth 10,000x as much to a peasant, but a noble cares 1,000,000 times less about the peasant’s welfare.
2. Nobles pref

Two scenarios:

• I take a vision or language model which was cutting edge in 2000, and run it with a similar amount of compute/data to what's typically used today.
• I take a modern vision or language model, calculate how much money it costs to train, estimate the amount of compute I could have bought for that much money in 2000, then train it with that much compute

In both cases, assume that the number of parameters is scaled to available compute as needed (if possible), and we generally adjust the code to reflect scalability requirements (while keeping the algorithm itself the same).

Which of t... (Read more)

When considering writing a hypothetical apostasy or steelmanning an opinion I disagreed with, I looked around for something worthwhile, both for me to write and others to read. Yvain/Scott has already steelmanned Time Cube, which cannot be beaten as an intellectual challenge, but probably didn't teach us much of general use (except in interesting dinner parties). I wanted something hard, but potentially instructive.

So I decided to steelman one of the anti-sacred cows (sacred anti-cows?) of this community, namely inefficiency. It was interesting to find that it was a little easier tha... (Read more)

There's been a long history of trying to penalise an AI for having a large impact on the world. To do that, you need an impact measure. I've designed some myself, back in the day, but they only worked in narrow circumstances and required tricks to get anything useful at all out from them.

A more promising general method is attainable utility. The idea is that, as an agent accumulates power in the world, they increase their ability to affect a lot of different things, and could therefore achieve a lot of different goals.

So, if an agent starts off unable to achieve many goals, but suddenly it can

Automatically crossposted

Creating really good outcomes for humanity seems hard. We get bored. If we don’t get bored, we still don’t like the idea of joy without variety. And joyful experiences only seems good if they are real and meaningful (in some sense we can’t easily pin down). And so on.

On the flip side, creating really bad outcomes seems much easier, running into none of the symmetric “problems.” So what gives?

I’ll argue that nature is basically out to get us, and it’s not a coincidence that making things good is so much harder than making them bad.

First: some other explanations

Two commo

There are different estimates of the possible severity of the current coronavirus outbreak. One estimation is based on the straight extrapolation of the exponential growth of the infected people number, which doubles every two days. This implies that the whole population of Earth will be ill in March. Another view takes into account that many mild cases are not included in the stat, so lethality is small and probably not everybody will be ill at all. We just don't know yet.

How we should act personally in this situation?

Firstly, we should act in the way, which should be good if everybody ... (Read more)

Claim: problems of agents embedded in their environment mostly reduce to problems of abstraction. Solve abstraction, and solutions to embedded agency problems will probably just drop out naturally.

The goal of this post is to explain the intuition underlying that claim. The point is not to defend the claim socially or to prove it mathematically, but to illustrate why I personally believe that understanding abstraction is the key to understanding embedded agency. Along the way, we’ll also discuss exactly which problems of abstraction need to be solved for a theory of embedded agency.

What do we m

I have been thinking about Stuart Armstrong's preference synthesis research agenda, and have long had the feeling that there's something off about the way that it is currently framed. In the post I try to describe why. I start by describing my current model of human values, how I interpret Stuart's implicit assumptions to conflict with it, and then talk about my confusion with regard to reconciling the two views.

The two-layer/ULM model of human values

In Player vs. Character: A Two-Level Model of Ethics, Sarah Constantin describes a model where the mind is divided, in game terms,... (Read more)

I know a fair number of people who put in a lot of effort into things like emotional healing, digging up and dealing with buried trauma, meditative and therapy practices, and so on. (I count myself in this category.)

And I think that there’s a thing that sometimes happens when other people see all of this, which is that it all seems kinda fake. I say this because even I have this thought sometimes. The core of the thought is something like, “if all of this stuff really worked, shouldn’t you be finished sometime? You claim that practice X was really beneficial, so why have you now been talking a... (Read more)

I notice that when I write for a public audience, I usually present ideas in a modernist, skeptical, academic style; whereas, the way I come up with ideas is usually in part by engaging in epistemic modalities that such a style has difficulty conceptualizing or considers illegitimate, including:

• Advanced introspection and self-therapy (including focusing and meditation)
• Mathematical and/or analogical intuition applied everywhere with only spot checks (rather than rigorous proof) used for confirmation
• Identity hacking, including virtue ethics, shadow-eating, and applied performativity theory
• A

What I mean by the "usual good actions" is notable good deeds you do may be a few times every week. Some examples:

• Help a friend end an abusive relationship.
• Go out of your way to help a stranger.
• Do something extra nice and thoughtful for your partner.
• Show up in person to support your friend's endeavor.
• Procure a Burning Man ticket for your friend.
• Refrain from calling your co-founder's idea "the dumbest thing I've heard this year" and do your best to listen.

What I mean by "altruistic impact" is probably QALYs, but I guess it's harder to use that m... (Read more)

American football has some properties that make it useful for practicing making predictions:

• A game consists of a sequence of discrete plays with well-defined outcomes
• The plays are fast enough that you get quick feedback on each prediction, and there are enough plays in a game that you can make enough predictions to see how well calibrated you are
• You go into each prediction with at least some information about what you're predicting

I've created a forecasting practice exercise that you can do while watching a game, either on paper or using this spreadsheet.

For those with no idea how footba

[This is Part 1. See also Part 2.]

Introduction

The Sleeping Beauty problem has been debated ad nauseum since Elga's original paper [Elga2000], yet no consensus has emerged on its solution. I believe this confusion is due to the following errors of analysis:

• Failure to properly apply probability theory.
• Failure to construct legitimate propositions for analysis.
• Failure to include all relevant information.

The only analysis I have found that avoids all of these errors is in Radford Neal's underappreciated technical report on anthropic reasoning [Neal2007]. In this note I'll discuss ho... (Read more),,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,