(Disclaimer: Philip Trammell is planning to rewrite this blogpost and make it clearer and more precise, but I think there's something good in this direction. Sharing it to see what the rest of the community thinks.)
Some smart people, including some of my friends, believe that advanced AI poses a serious threat to human civilization in the near future, and that AI safety research is therefore one of the most valuable uses, if not the very most valuable use, of philanthropic talent and money. But most smart people, as far as I can judge their behavior—including some, like Mark Zuckerberg and R... (Read more)
In the last few months, I've gotten increasingly alarmed by leftist politics in the US, and the epistemic conditions that it operates under and is imposing wherever it gains power. (Quite possibly the conditions are just as dire on the right, but they are not as visible or salient to me, because most of the places I can easily see, either directly or through news stories, i.e., local politics in my area, academia, journalism, large corporations, seem to have been taken over by the left.)
I'm worried that my alarmism is itself based on confirmation bias, tribalism, catastrophizing, or any number... (Read more)
Suppose that a kingdom contains a million peasants and a thousand nobles, and:
Then it’s simultaneously the case that:
This post was written for Convergence Analysis by Michael Aird, based on ideas from Justin Shovelain and with ongoing guidance from him. Throughout the post, “I” will refer to Michael, while “we” will refer to Michael and Justin or to Convergence as an organisation.
Epistemic status: High confidence in the core ideas on an abstract level. Claims about the usefulness of those ideas, their practical implications, and how best to concretely/mathematically implement them are more speculative; one goal in writing this post is to receive feedback on those things. I’m quite new to many of the concepts... (Read more)
In both cases, assume that the number of parameters is scaled to available compute as needed (if possible), and we generally adjust the code to reflect scalability requirements (while keeping the algorithm itself the same).
Which of t... (Read more)
When considering writing a hypothetical apostasy or steelmanning an opinion I disagreed with, I looked around for something worthwhile, both for me to write and others to read. Yvain/Scott has already steelmanned Time Cube, which cannot be beaten as an intellectual challenge, but probably didn't teach us much of general use (except in interesting dinner parties). I wanted something hard, but potentially instructive.
So I decided to steelman one of the anti-sacred cows (sacred anti-cows?) of this community, namely inefficiency. It was interesting to find that it was a little easier tha... (Read more)
There's been a long history of trying to penalise an AI for having a large impact on the world. To do that, you need an impact measure. I've designed some myself, back in the day, but they only worked in narrow circumstances and required tricks to get anything useful at all out from them.
A more promising general method is attainable utility. The idea is that, as an agent accumulates power in the world, they increase their ability to affect a lot of different things, and could therefore achieve a lot of different goals.
So, if an agent starts off unable to achieve many goals, but suddenly it can... (Read more)
Creating really good outcomes for humanity seems hard. We get bored. If we don’t get bored, we still don’t like the idea of joy without variety. And joyful experiences only seems good if they are real and meaningful (in some sense we can’t easily pin down). And so on.
On the flip side, creating really bad outcomes seems much easier, running into none of the symmetric “problems.” So what gives?
I’ll argue that nature is basically out to get us, and it’s not a coincidence that making things good is so much harder than making them bad.
Two commo... (Read more)
At some point in history a lot of thought was put into obtaining the equation:
R*T = P*V/n
The ideal gas equation we learn in kindergarten, which uses the magic number
in order to make predictions about how
moles of an “ideal gas” will change in pressure, volume or temperature given that we can control two of those factors.
This law approximates the behavior of many gases with a small error and it was certainly useful for many o' medieval soap volcano party tricks and Victorian steam engine designs.
But, as is often the case in scienc... (Read more)
Assortative mating is when similar people marry and have children. Some people worry about assortative mating in Silicon Valley: highly analytical tech workers marry other highly analytical tech workers. If highly analytical tech workers have more autism risk genes than the general population, assortative mating could put their children at very high risk of autism. How concerned should this make us?
Methods / Sample Characteristics
I used the 2020 Slate Star Codex survey to investigate this question. It had 8,043 respondents selected for being interested in a highly analytical blog a... (Read more)
There are different estimates of the possible severity of the current coronavirus outbreak. One estimation is based on the straight extrapolation of the exponential growth of the infected people number, which doubles every two days. This implies that the whole population of Earth will be ill in March. Another view takes into account that many mild cases are not included in the stat, so lethality is small and probably not everybody will be ill at all. We just don't know yet.
How we should act personally in this situation?
Firstly, we should act in the way, which should be good if everybody ... (Read more)
Claim: problems of agents embedded in their environment mostly reduce to problems of abstraction. Solve abstraction, and solutions to embedded agency problems will probably just drop out naturally.
The goal of this post is to explain the intuition underlying that claim. The point is not to defend the claim socially or to prove it mathematically, but to illustrate why I personally believe that understanding abstraction is the key to understanding embedded agency. Along the way, we’ll also discuss exactly which problems of abstraction need to be solved for a theory of embedded agency.
I have been thinking about Stuart Armstrong's preference synthesis research agenda, and have long had the feeling that there's something off about the way that it is currently framed. In the post I try to describe why. I start by describing my current model of human values, how I interpret Stuart's implicit assumptions to conflict with it, and then talk about my confusion with regard to reconciling the two views.
In Player vs. Character: A Two-Level Model of Ethics, Sarah Constantin describes a model where the mind is divided, in game terms,... (Read more)
I know a fair number of people who put in a lot of effort into things like emotional healing, digging up and dealing with buried trauma, meditative and therapy practices, and so on. (I count myself in this category.)
And I think that there’s a thing that sometimes happens when other people see all of this, which is that it all seems kinda fake. I say this because even I have this thought sometimes. The core of the thought is something like, “if all of this stuff really worked, shouldn’t you be finished sometime? You claim that practice X was really beneficial, so why have you now been talking a... (Read more)
I notice that when I write for a public audience, I usually present ideas in a modernist, skeptical, academic style; whereas, the way I come up with ideas is usually in part by engaging in epistemic modalities that such a style has difficulty conceptualizing or considers illegitimate, including:
What I mean by the "usual good actions" is notable good deeds you do may be a few times every week. Some examples:
What I mean by "altruistic impact" is probably QALYs, but I guess it's harder to use that m... (Read more)
American football has some properties that make it useful for practicing making predictions:
I've created a forecasting practice exercise that you can do while watching a game, either on paper or using this spreadsheet.
For those with no idea how footba... (Read more)
[This is Part 1. See also Part 2.]
The Sleeping Beauty problem has been debated ad nauseum since Elga's original paper [Elga2000], yet no consensus has emerged on its solution. I believe this confusion is due to the following errors of analysis:
The only analysis I have found that avoids all of these errors is in Radford Neal's underappreciated technical report on anthropic reasoning [Neal2007]. In this note I'll discuss ho... (Read more),,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,