Or in other words:
If we're lucky the mercury in our thermometer moves up a constant amount each time, and we can avoid mixing things to calibrate our thermometer in the future. If we're unlucky it doesn't and we need to painstakingly calibrate all thermometers. If we're really unlucky a scale that works on one substance doesn't work on another and then we need to rethink this whole temperature thing again.
In fact you get unlucky on both counts.
I mean the actual temperature, which existed before humans had defined it. Maybe another non-circular way to say this is that you will get different results depending on whether you mix 99 parts ice and 1 part boiling water, vs other materials at the same temperature.
They discuss this in the book! The problem is that it might take different amounts of heat to raise the temperature by a fixed amount if you start from different temperature levels, so you can't be sure the increments you get from this method are really equal.
We start off with some early values, and then develop instrumental strategies for achieving them. Those instrumental strategies become crystallized and then give rise to other instrumental strategies for achieving them, and so on. Understood this way, we can describe an organism's goals/strategies purely in terms of which goals "have power over" which other goals, which goals are most easily replaced, etc, without needing to appeal to some kind of essential "terminalism" that some goals have and others don't.
I think it would help me if you explained what you think it would mean for there to be an instrumental/terminal distinction, since to my eyes you've just spelled out the instrumental/terminal split.
Can you describe what you think the important inequivalencies are?
Am I right that this is about the version of the RAISE act that passed the senate, not the one that will be put into law?
Anthropic wants to stay near the front of the pack at AI capabilities so that their empirical research is relevant, but not at the actual front of the pack to avoid accelerating race-dynamics.
— From an Anthropic employee in a private conversation, early 2023
Note that this is not a quote of an Anthropic employee in a private conversation. Instead, it is a quote of Ray Arnold describing his memory/summary of something an Anthropic told him in a private conversation.
I think it's not just that it's slower/deeper: my personal sense (which might be just a thing of not requiring much medical care between the ages of 5 and 30) is that the pace at which awesome new stuff is happening in medicines I can buy got much faster in the last few years. If my perception is right, it seems like that requires some explanation of "bio is slower/deeper and also 40 years ago there was a massive breakthru that took 40 years to percolate", and not just "bio is slower/deeper".
This year, we received a $1.6M matching grant from the Survival and Flourishing Fund, which means that the first $1.6M we receive in donations before December 31st will be matched 1:1. We will only receive the grant funds if it can be matched by donations.
Does anyone know (i.e. actually know, not just have a speculative guess) why SFF does this sort of thing? Naively it seems like a weird way to operate from a perspective of maximizing the good done by their donations. Is the reasoning something like "if other people donate to MIRI, that increases our estimate of how valuable MIRI is, and so we want to donate more to it"?
Yep, see this paper - Bayesian updating is the same as having all the hypotheses in your head be agents in a prediction market using the Kelly criterion, where their inital wealths are your prior probabilities on those hypotheses, the market price of some observation is your marginal probability over that observation, and the wealths after the bets resolve are the posterior distributions.