This is a good way to think about it although I think your numbers are way too high

And donate 50%

This number in particular is 10x too high IMO. Virtually nobody donates 50%. EA Survey shows that the median self-identified earner-to-give only donates about 5% of their income (IIRC, I can't find the data now)

If they spread that over 20 years, at current interest rates that's about 500 million a year

I expect the giving to be more front-loaded than that because a lot of Anthropic employees have short timelines

Another consideration is that money is disproportionately held by people who are high up in the company, who I would guess are more selfish than average which means lower donations

Your made-up numbers came up with $7.5B donated over 20 years. My guess is the total amount donated will be more like $250M–$1B but heavily front-loaded, so perhaps $100M in the first few years and then much less thereafter

My guess is you're the only person in the world who does this, but also this is better than what everyone else is doing and maybe I should start doing it

Also, sometimes people let you know some important reason why you shouldn't count their datapoint. For example, someone might rate the food 1/10, which sounds terrible, but then they'll clarify that they weren't there during mealtimes and didn't eat the food, and just gave it 1/10 because it was mandatory! This is rarely predictable, but especially with autistic people you occasionally get odd edge-cases like this.

My autistic friend would like to know: if there's a mandatory question to rate the food but he didn't eat it, which number should he pick?

Or maybe I'm the one who's been misunderstanding it! I don't think I have a great understanding of the term tbh so you're probably right.

If that's what it means then instead of "Bayes points", Quinn could call it "credibility" or "predictive accuracy" or something.

Thanks, I didn't see the page on how matching works. I edited that section to fix it.

My first draft actually said something more accurate, but then I edited it to be worse :/ I have edited it back to say something more like the original version.

I think the term means something like "you demonstrated truth-seeking character/virtue".

Example: Someone (I forget who it was, sorry) came up with a novel AI alignment theory, and then they wrote a long post about how their own theory was deeply flawed. That post earned them Bayes points.

In terms of surviving superintelligence, it’s still the scene from The Phantom Menace. As in, that won’t be enough.

Are you talking about the scene near the beginning where the Neimoidians send some battle droids to kill the Jedi and all the droids die in 4.5 seconds flat?

Edit: I looked up the script and I was close but not quite right. "That won't be enough" is a line spoken by one of the Neimoidians after locking the doors to the bridge to keep the Jedi out. Qui-Gon was poised to cut through the door in 4.5 seconds flat until they closed the blast doors.

Good callout. I was glad to hear that Ilya is thinking about all sentient life and not just humans.

I didn't interpret it to mean that he's working on thing 1. The direct quote was

I think in particular, there’s a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone, because the AI itself will be sentient. And if you think about things like mirror neurons and human empathy for animals, which you might argue it’s not big enough, but it exists. I think it’s an emergent property from the fact that we model others with the same circuit that we use to model ourselves, because that’s the most efficient thing to do.

Sounds to me like he expects an aligned AI to care about all sentient beings, but he isn't necessarily working on making that happen. AFAIK Ilya's new venture hasn't published any alignment research yet, so we don't know what exactly he's working on.

The problem of coordinating on AI development isn't the same thing as solving the alignment problem, but it's not the thing I'm pointing at in this post because it's still about avoiding misalignment.

I plan on writing something longer about this in the future but people use "alignment" to refer to two different things, basically thing 1 is "ASI solves ethics and then behaves ethically" and thing 2 is "ASI does what people want it to do". Approximately nobody is working on thing 1, only on thing 2, and thing 2 doesn't get us a solution to non-alignment problems.