I don't do this on purpose but I feel like 90% of what I write about AI is something Eliezer already said at some point.

Yeah I pretty much agree with what you're saying. But I think I misunderstood your comment before mine, and the thing you're talking about was not captured by the model I wrote in my last comment; so I have some more thinking to do.

I didn't mean "can be trusted to take AI risk seriously" as "indeterminate trustworthiness but cares about x-risk", more like "the conjunction of trustworthy + cares about x-risk".

Like, the most important thing to estimate when evaluating a political candidate is their trustworthiness and integrity! It's the thing that would flip the sign on whether supporting someone is good or bad for the world.

I agree that this is an important thing that deserved more consideration in Eric's analysis (I wrote a note about it on Oct 22 but then I forgot to include it in my post yesterday). But I don't think it's too hard to put into a model (although it's hard to find the right numbers to use). The model I wrote down in my note is

• 30% chance Bores would oppose an AI pause / strong AI regulations (b/c it's too "anti-innovation" or something)
• 40% chance Bores would support strong regulations
• 30% chance he would vote for strong regulations but not advocate for them
• 90% chance Bores would support weak/moderate AI regulations

My guess is that 2/3 of the EV comes from strong regulations and 1/3 from weak regulations (which I just came up with a justification for earlier today but it's too complicated to fit in this comment), so these considerations reduce the EV to 37% (i.e., roughly divide EV by 3).

FWIW I wouldn't say "trustworthiness" is the most important thing, more like "can be trusted to take AI risk seriously", and my model is more about the latter. (A trustworthy politician who is honest about the fact that they don't care about AI safety will not be getting any donations from me.)

An important part of my model of college admissions—which unfortunately I didn't acquire until after I was done applying for colleges—is to consider what type of person becomes a college admissions officer. What percentage of admissions officers majored in math? (Is it possibly as high as 1%? I doubt it.) What percentage of admissions officers understand the significance of something like "solved the Mizohata-Takeuchi conjecture"? What percentage have a vague (or even explicit) disdain for anything math-flavored?

On my model, it is not surprising that admissions officers would fail to appreciate a math prodigy.

Administrators overriding an acceptance does seem like a remarkable failure. I can't say I'm surprised, but it's a much worse indictment of those universities, I think.

This means that an extra $300,000 would better position the campaign such that Alex Bores would be able to net an extra 1000 votes in expectation, which (as per my earlier estimate) has a 1.6% chance of counterfactually winning him the election. That would translate to $190,000 for a 1% increase in his chance of winning

Sanity check: in 2023–2024, median new/incumbent House member raised $2.4 million/$2.1 million respectively (source), and median incumbent in a "toss-up" race raised $7.9 million.

The median race raised ~10x more than the amount needed for a 1% move according to the model in OP. Is that reasonable? It sounds basically reasonable to me.

So it seems like Kelly's critique is kinda self-defeating. If Dorothy Martin's little UFO cult isn't really an example of the mechanism Festinger popularized, in the way Kelly describes, then Festinger and his colleagues themselves are an even better example of it.

I think there is a huge difference between the two situations, for several reasons:

1. a large number of people sticking with a doomsday cult that's been falsified, vs. a small number of authors fabricating evidence to advance their careers
2. a trivially-verifiable prediction that's falsified, vs. a prediction that involves a lot of research and data and is hard to check (I'm assuming the debunking is correct but I don't really know because I haven't looked at any of the evidence presented)
3. as I understand, Festinger et al. did actually change their minds in response to evidence, but then they published findings they knew were false because it was in their personal interest to do so

Why does Eliezer dislike the paperclip maximizer thought experiment?

Numerous times I have seen him correct people about it and say it wasn't originally about a totalizing paperclip factory, it was about an AI that wants to make little squiggly lines for inscrutable reasons. Why does the distinction matter? Both scenarios are about an AI that does something very different from what you want and ends up killing you.

My guess, although I'm not sure about this, is that the paperclip factory is an AI that did as instructed, but its instructions were bad and it killed everyone. Whereas the squiggly line thing is about AI not doing what you want. And perhaps the paperclip factory scenario could mislead people into believing that all you have to do is make sure the AI understands what you want.

FWIW I always figured the paperclip maximizer would know that people don't want it to turn the lightcone into paperclips, but it would do it anyway, so I still thought it was a reasonable example of the same principle as the squiggly-lines AI. But I can see how that conclusion requires two steps of reasoning whereas the squiggly-lines scenario only requires one step. Or perhaps the thing that Eliezer thinks is wrong with the paperclip-maximizer scenario is something else entirely.

people being more likely to click on or read shortforms due to less perceived effort of reading (since they're often shorter and less formal)

And because you can read them without loading a new page. I think that's a big factor for me.

maybe they help workshop new analogies that eventually can be refined into If Anyone style books or podcast interviews.

I think it's helpful to write arguments multiple times. And I think it's sensible to write out the argument in a "you-shape" and then refine it and try to make it more appealing to a broader range of people.

This kind of post might also give fuel for someone else to make basically the same argument, but in a different style, which could end up being helpful.

we can never know our own true reason for doing something

I read Sam Harris's book on free will which I think is what you're referring to, but I don't recall him saying anything like that. If he did, I presume he meant something like "you don't know which set of physical inputs to your neurons caused your neurons to fire in a way that caused your behavior", which doesn't mean you can't have a belief about whether someone's motivation is (say) religious or geopolitical.