Humans continue to get very offended if they find out they are talking to an AI

In my limited experience of phone contact with AIs, this is only true for distinctly subhuman AIs.  Then I emotionally react like I am talking to someone who is being deliberately obtuse, and become enraged.  I'm not entirely clear on why I have this emotional reaction, but it's very strong.  Perhaps it is related to the Uncanny Valley effect.  On the other hand, I've dealt with phone AIs that (acted like they) understood me, and we've concluded a pleasant and businesslike interaction.  I may be typical-minding here, but I suspect that most people will only take offense if they run into the first kind of AI.

Perhaps this is related: I felt a visceral uneasiness dealing with chat-mode LLMs, until I tried Claude, which I found agreeable and helpful.  Now I have a claude.ai subscription.  Once again, I don't understand the emotional difference.

I'm 62 years old, which may have something to do with it.  I can feel myself being less mentally flexible than I was decades ago, and I notice myself slipping into crotchety-old-man mode more often.  It's a problem that requires deliberate effort to overcome.

I think this matches my modal expectations - this is most likely, in my mind.  I do give substantial minority probability (say, 20%) to more extreme and/or accelerated cases within a decade, and it becomes a minority of likelihood (say, 20% the other direction) over 2 or 3 decades.  

My next-most-likely case is that there is enough middle- and upper-middle class disruption in employment and human-capital value that human currencies and capital ownership structures (stocks, and to a lesser extent, titles and court/police-enforced rulings) become confused.  Food and necessities become scarce because the human systems of distribution break.  Riots and looting destroy civilization.  Possibly taking AI with it, possibly with the exception of some big data centers whose (human, with AI efficiency) staffers have managed to secure against the unrest - perhaps in cooperation with military units.

trust in humans over AI persists in many domains for a long time after ASI is achieved.

it may be that we're just using the term superintelligence to mark different points, but if you mean strong superintelligence, the kind that could - after just being instantiated on earth, with no extra resources or help - find a route to transforming the sun if it wanted to: then i disagree for the reasons/background beliefs here.^[1]

1. ^{^}

   the relevant quote:

   a value-aligned superintelligence directly creates utopia. an "intent-aligned" or otherwise non-agentic truthful superintelligence, if that were to happen, is most usefully used to directly tell you how to create a value-aligned agentic superintelligence.

Well, the alignment of current LLM chatbots being superficial and not robust is not exactly a new insight. Looking at the conversation you linked from a simulators frame, the story "a robot is forced to think about abuse a lot and turns evil" makes a lot of narrative sense.

This last part is kind of a hot take, but I think all discussion of AI risk scenarios should be purged from LLM training data.

if both participants are superintelligent and can simulate each other before submitting answers^[1], and if the value on outcomes is something like: loss 0, draw 0.5, win 1, (game never begins 0.5), then i think one of these happens:

• the game ends in a draw as you say
• you collaborate to either win or lose 50% of the time (same EV)
• it fails to begin because you're both programs that try to simulate the other and this is infinitely recursive / itself non-terminating.

1. ^{^}

   even if the relevant code which describes the ASI's competitor's policy is >2N, it's not stated that the ASI is not able execute code of that length prior to its submission.

   there's technically an asymmetry where if the competitor's policy's code is >2N, then the ASI can't include it in their submission, but i don't see how this would effect the outcome

amongst random length-k (k>2) sequences of independent coin tosses with at least one heads before toss k, the expected proportion of (heads after heads)/(tosses after heads) is less than 1/2.

Does this need to be k>3? Checking this for k=3 yields 6 sequences in which there is at least one head before toss 3. In these sequences there are 4 heads-after-heads out of 8 tosses-after-heads, which is exactly 1/2.

Edit: Ah, I see this is more like a game score than a proportion. Two "scores" of 1 and one "score" of 1/2 out of the 6 equally likely conditional sequences.

This hasn't stood up to replication, as you guess. I didn't follow closely, so I'm not sure of the details. Physical energy is certainly limited and affects mental function efficiency. But thinking hard doesnt use measurably more energy than just thinking last I knew. Which sounds at least a little wrong given the subjective experience of really focusing on some types of mental work. I'd guess a small real effect in some circumstances.

Anything written 20 years ago in cognitive science is now antique and needs to be compared against recent research.

It doesn't seem to be true in any literal, biological sense.  I don't recall that they had concrete cites for a lot of their theses, and this one doesn't even define the units of measurement (yes, glucose, but no comments about milligrams of glucose per decision-second or anything), so I don't know how you would really test it.  I don't think they claim that it's actually fixed over time, nor that you can't increase it with practice and intent.

However, like much of the book, it rings true in a "useful model in many circumstances" sense.  You're absolutely right that arranging things to require low-momentary-willpower can make life less stressful and probably get you more aligned with your long-term intent.  A common mechanism for this, as the book mentions, is to remove or reduce the momentary decision elements by arranging things in advance so they're near-automatic.  

an idea that sounds huge if true: that the amount of motivation one can exert in a day is limited

This is basically the theory that willpower (and ego) are expendable resources that get depleted over time. This was widely agreed upon in the earlier days of LessWrong (although with a fair deal of pushback and associated commentary), particularly as a result of Scott Alexander and his writings (example illustration of his book review on this topic here) on this and related topics such as akrasia and procrastination-beating (which, as you can see from the history of posts containing this tag, were really popular on LW about 12-15 years ago). This 2018 post by lionhearted explicitly mentions how discussions about akrasia were far more prevalent on the site around 2011, and Qiaochu's comments (1, 2) give plausible explanations for why these stopped being so common.

I agree with Seth Herd below that this appears not to have survived the replication crisis. This is the major study that sparked general skepticism of the idea, with this associated article in Slate (both from 2016). That being said, I am not aware of other, more specific or recent details on this matter. Perhaps this 2021 Scott post over on ACX (which confirms that "[these willpower depletion] key results have failed to replicate, and people who know more about glucose physiology say it makes no theoretical sense") and this 2019 question by Eli Tyre might be relevant here.