Dweomite

As I read the OP, I thought to myself: If I were to steelman the people the post is complaining about, I would guess that they are interpreting the complaint about the problem as an implied proposal for how to address the problem, and they are reacting to the perceived-implied-proposal.

It seems like you're thinking along similar lines, but are about 3 assumptions further down that road:

• Not only are you guessing that they might have thought this, you're pretty sure this is what they were thinking
• Not only are you sure they thought this, you're sure they were

Alternate steelman - they're worried that you're intending to misleadingly quote their answer in a different context, and have rigged the question to get the quote you want.

My primary guess is that the 6% who like RSI but not ASI are answering based on vibes rather than coherent models, and ASI currently has worse vibes.

Though I could imagine some people thinking that RSI will stop before "superintelligence", and other people thinking that orthogonality is wrong and RSI would continue beyond some window of "dangerous superintelligence" into "godlike benevolence". I personally consider both of those possibilities to be so staggeringly implausible that they're basically just wishful thinking, but I also think that more than 6% of people are engaging in some amount of wishful thinking about AI.

Could you clarify what you meant by this combination of remarks?

I looked at all of those questions and thought, "two years down the road that's hell. No thank you."

...

Let me say, for myself, I absolutely want such an outcome.

My personal take is that I can imagine there are people who really would be happier in a scenario where they'll literally starve to death if they aren't productive enough, and I guess it would be good if those people could experience the scenario where they thrive, but if there are also people who do fine with a permanent vacation then... (read more)

For some reason, I didn't think of this when I read the results, but immediately thought of it when I read the actual question's wording (even though the question doesn't mention this).

Framing effects are scarily powerful.

How is that different from saying that you do not have an Earth unless you can point to it? If we require that you augment the recipe with some spacetime coordinates that tell you where in the resulting universe to look, why are those coordinates any longer for a BB than for an Earth?

Any given configuration of the universe either eventually will produce a BB, or won't.

Because BBs are much smaller than Earth, the number of configurations that eventually lead to BBs should be much larger than the number that eventually lead to Earths, and therefore the number of variables you need to control to ensure you eventually get a BB should (in expectation) be much smaller than the number of variables you need to control to ensure you eventually get an Earth. BBs are a larger target in possibility space, and therefore (in expectation) easier to hit.

1. Doesn't this argument prove too much? Why doesn't similar reasoning rule out other mentally-impaired states like drunkenness, dreams, and brain injuries? In fact, why doesn't it rule out being a flawed human being, rather than an ideal reasoner?
2. Even if all your memories are false, couldn't you still reason validly about abstract concepts (like math) and hypothetical scenarios? This seems to show that BBs can do at least some valid reasoning.
3. Even if you might be in some state S that would totally prevent you from reasoning validly, if you conjectured you we

I'm confused about why you think BBs should be complexity-penalized for the difficulty of specifying which of all hypothetically-possible BBs we're talking about, but you don't think Earth should be complexity-penalized for the difficulty of specifying which of all hypothetically-possible Earths we're talking about.

I get that you can (we think) specify a recipe that eventually produces Earth, rather than explicitly specifying every detail of the current Earth. But presumably you could also specify a recipe that eventually produces a BB. And since the final... (read more)

What makes you think the explanation for why you won the lottery won't help you make useful predictions about what follow-up actions will fulfill your values? For example, if the explanation were something like "the lottery was rigged, and there's about to be a criminal investigation targeting you", that seems pretty relevant to your follow-up plans.

Explanations of previously-mysterious phenomena are often useful in ways that are hard to foresee before you know the explanation.

If you think that understanding "normal" things is typically useful, why single out this one specific thing to be incurious about?

This seems like a neat idea, but I'd like to flag that this strategy only seems applicable when the fact you are looking for already exists explicitly and at an appropriate explainer level. I'm not sure you can do anything equivalent if you want the LLM to explain, synthesize, summarize, or do original reasoning.

I would describe a critical try as one where the act of trying is likely to prevent further attempts. Launching an ASI is a critical try because the ASI itself could likely stop you from launching more ASIs later on (e.g. by killing you).

If it's possible to send out missions to intercept the asteroid before it arrives, then it seems to me that the asteroid is better understood as a time limit than as a critical try. You could set the parameters of the asteroid scenario in such a way that you have time for exactly one try, but you could also set the paramet... (read more)

Yes, that is the sort of example I meant. Though of course this particular example does not prove that the game of Catan, in particular, has situations like this.

Based on his other reply, I expect James would want to point out that there is an equivalent equilibrium where player A, instead of saying "button N is blue", says "either button N is blue or no button is", which produces the same outcome without technically lying.

I'm coming to think that there should be some other distinction we can draw that rhymes with the truthful/lying distinction but that talks about consequences instead of semantics, and therefore can't be dodged by relabeling the signals. Still thinking about it.

In principle, IF the norms are more important to you than to everyone else combined, then there should be some amount you can pay them that is higher than how much they care about the norms but lower than how much you care about them.

(In practice, finding that amount may be hard, and treating it too much like a transaction may have friendship-corroding effects.)

Your final sentence clarified some things for me:

• In that sentence, you are not arguing that the lie is no better than silence, you are arguing that it is no better than some truthful message. (This is technically still a falsification of my previously-stated interpretation of your claim.)
• This argument is based on the assumption that the other players already know all circumstances under which you would transmit this message, so there's no harm in admitting them.

I now realize that if all players have perfect knowledge of the exact conditions under which you... (read more)

Been thinking more about this claim:

Also, with rational agents silence is just as good as dishonesty.

I don't think this claim particularly matters to the thrust of your post, since I think we agree that you're not playing with perfectly rational agents, but I'm interested in the claim as a matter of game theory.

To be clear, I'm interpreting this as saying something at least as strong as: "In a game of Catan where there is common knowledge that all players are perfectly rational, speaking a falsehood is never more advantageous for the speaker than remaining... (read more)

Have you talked explicitly with them about the norms you'd like to have? I, for one, would not have assumed that "don't try to manipulate other players to your own advantage" would be an expected norm, but would probably be willing to go along with it if the group asked me to.

You also might consider offering to play with a handicap, so that they don't feel that they need to target you to prevent you from winning too often.

As a rule of thumb, I strongly approve of play groups mutually agreeing on whatever rules and norms work best for them. But I also think... (read more)

• Identical twins share the same genome, but are usually considered separate objects for most purposes.
• A genetic chimera has different genomes in different parts of its body, but is usually considered a single object for most purposes. (Often, you can't tell the organism is a chimera on a casual inspection.)

I submit that "same genome" often coincides with the natural object boundary, but isn't usually a good criterion for the boundary. The common genome is not a significant part of what makes a squirrel a good object.

I think the thing we usually care about i... (read more)

Based on this and your other comments in this thread, I suspect you're mixing up questions of

1. What's good?
2. How do we know what's good?
3. How should we treat other people who disagree with us about what's good?

It's possible to think someone is baking bread wrong without thinking that you should use violence to force them to do it differently. It's possible to think that bakers should be allowed to pick their own baking methods without thinking that all methods produce equally tasty bread.

Civilization typically uses many different levels of coercion for different... (read more)

I think you do a good job of arguing (in the earlier part of the article) that it is logically possible to drop the independence axiom without being money-pumped by giving up logical consequentialism but keeping dynamic consistency. However, I think you do a poor job of arguing (in the later parts) that we should give up consequentialism.

You examine 3 in-depth examples to try to show that we'd be fine if we dropped independence: ergodicity economics, the Allais Paradox, and the Ellsburg Paradox. In all 3 cases, it think your argument is missing a critical ... (read more)