Wiki Contributions


Frame Control

(Upvoting for the edit.)

You are way more fallible than you think

Also... it seems like you're assuming this as background:

Most of us are very much uncalibrated. We give 80%+ chance of success/​completion to the next project even after failing a bunch of similar ones before. Even those of us who are well calibrated are still bad at the margins, where the probability is low (or, equivalently, high). Events we give the odds of 1% to can happen with the frequency of 20%.

And the rest of the post riffs off that. (Like, your examples seem like not "here are examples to convince you you're uncalibrated" but "here are examples of how to deal with the fact that you're uncalibrated" or something.)

But, citation needed.

I'll grant the "most of us". I recall the studies mentioned in HPMOR, along the lines of "you ask people when they're 95% likely to finish only like a quarter finish by then. And you ask when they're 50% likely to finish and it's statistically indistinguishable". I think to (reliably, reproducibly, robustly) get results like that, most of the people in those studies need to be poorly calibrated on the questions they're being asked.

But the "even those of us"? Given that the first two words of the post are "yes, you" - that is, you project extreme confidence that this applies to the reader... how do you know that the reader is bad at the margins even if they're well calibrated elsewhere?

Is this also true of superforecasters? Is it true of the sorts of people who say "man, I really don't know. I think I'd be comfortable buying an implied probability of 0.01% and selling an implied probability of 1%, I know that's a crazy big range but that's where I am"?

(This seems like the sort of extreme confidence that you warn about in this very post. I know you admit to being more fallible than you think, but...)

You are way more fallible than you think

It seems like you're framing this in terms of "extreme probabilities are unlikely to be accurate", but...

  • You give an example of 80% probabilities being inaccurate.
  • You use AGI risk as an example, which around here I often see estimates like "50% by this date, 75% by this date" and I get the impression you meant it to apply to that sort of thing too.
  • You can always make an extreme probability less extreme. Silly example: "99% chance of AGI tomorrow" becomes "49.5% chance of AGI tomorrow and I get heads on this coin toss".

I feel like this kind of thing needs to be about inputs, not outputs. "If you find yourself calculating a probability under these circumstances, be suspicious", not "if you find you calculated a probability of this level, be suspicious".

Paxlovid Remains Illegal: 11/24 Update

Do you not need a control group in a safety trial?

Chris Voss negotiation MasterClass: review

Sorry, yeah, that was unclear. I think I prefer the way you wrote it.

Chris Voss negotiation MasterClass: review

I think I prefer it this way, fwiw.

The poetry of progress

Another song from To Touch The Stars, Starfire, has a verse celebrating group endeavors:

Ten thousand hands to build the shining shell
It took a dozen years, and love to build it well
All those who touched its birth, though they be bound on earth
Will be with the astronauts that in her dwell

(Not sure if it's intended as "birth" or "berth" or deliberately ambiguous.)

There's also the song "Somebody Will" by Sassafras.

I'm sure I know other examples, but those are what's coming to mind right now.

Ngo and Yudkowsky on alignment difficulty

Oh, thanks for the pointer. I confess I wish Robin was less terse here.

I'm not sure I even understand the claim, what does it mean to "recover interest rates"? Is Robin claiming any such bet will either

  1. Have payoffs such that [the person receiving money now and paying money later] could just take out a loan at prevailing interest rates to make this bet; or
  2. Have at least one party who is being silly with money?

...oh, I think I get it, and IIUC the idea that fails is different from what I was suggesting.

The idea that fails is that you can make a prediction market from these bets and use it to recover a probability of apocalypse. I agree that won't work, for the reason given: prices of these bets will be about both [probability of apocalypse] and [the value of money-now versus money-later, conditional on no apocalypse], and you can't separate those effects.

I don't think this automatically sinks the simpler idea of: if Alice and Bob disagree about the probability of an apocalypse, they may be able to make a bet that both consider positive-expected-utility. And I don't think that bet would necessarily just be a combination of available market-rate loans? At least it doesn't look like anyone is claiming that.

A Defense of Functional Decision Theory

Here's an attempt to ground this somewhat concretely.

Suppose there's an iterated prisoner's dilemma contest. At any iteration an agent can look at the history of plays that itself and its opponent have made.

Suppose that TitForTatBot looks at the history, and sees that there's been 100 rounds so far, and in every one it has defected and its opponent has cooperated. It proceeds to cooperate, because its opponent cooperated in the previous round. And so the "actual" game history will never be (D,C) x 100. What's happened here is that someone has instantiated a TitForTatBot and lied to it. It's not impossible that TitForTatBot will observe this history, but it's impossible that this history actually happened, in some sense that I claim we care about.

Ngo and Yudkowsky on alignment difficulty

I don't think that holds up, because with traditional uses of prediction markets both parties expect to see the end of the bet. If it's a long term bet then that would increase the expected payoff they require to be willing to bet, but there's some amount of "money later" that's worth giving up "money now" in exchange for. So both are willing to lock money up.

With a bet on the end of the world, all of the upside for one party comes from receiving "money now". There's no potential "money later" payoff for them, the bet isn't structured that way. And putting money in escrow means their "money now" vanishes too.

That is, if I receive a million dollars now and pay out two million if the world doesn't end, then: for now I'm up a million; if the world ends I'm dead; if the world doesn't end I'm down a million. But if the two million has to go in escrow, the only change is that for now I'm down a million too. So I'm not gonna do this.

Load More