I'm not sure how this would work, especially in conflicting-expectation cases (one person watches a die roll 20 times, another person only saw the last 5, they have DIFFERENT expectations in this non-independent-trials world.  What actually happens with what frequency?  Depending on specifics, presumably casinos hire people to differentially watch games and preferentially shuffle or change the dice.  

Other good effects: nobody expects to get cancer, so I guess it doesn't happen?  

But really, it's just confused in conception - this fallacy is based on a reference class - events are no longer independent, but it's unclear in what way they're entangled.  If I roll 12 different dice, does it apply?  If I roll the same die 12 times, but only observe the first 2 and the last, do the 8 unobserved rolls affect the expectation?

There are (at least) two models which could partially explain this:
1) The high-status/high-rank people have that status because they're better at abstract and long-term thinking, and their role is more toward preventing catastrophe rather than nudging toward improvements.  They leave the lesser concerns to the underlings, with the (sometimes correct) belief that it'll come out OK without their involvement.

2) The high-status/high-rank people are rich and powerful enough to be somewahat insulated from most of the prosaic AI risks, while the average member can legitimately be hurt by such things.   So everyone is just focusing on the things most likely to impact themselves.

edit: to clarify, these are two models that do NOT imply the obvious "smarter/more powerful people are correctly worried about the REAL threats, and the average person's concerns are probably unimportant/uninformed".  It's quite possible that this division doesn't tell us much about the relative importance of those different risks.  

I don't know enough about attacker motivations and economics, or what is the elasticity of attempts to extort (nor about elasticity of victims who choose to pay).  It certainly won't solve the problem.  It MAY reduce the incidence, or it may just make it more expensive for victims and slightly less profitable (but sill way positive) for attackers.

I do wonder why it's preferable to tax at punitive rates, but not to just outlaw entirely.  Philosophically, sin taxes are annoying, because they assert a sin without much justification or calculation of "how bad is it".  When framed in a Pigou contex (tax enough to be neutral about the externality), it's a much stronger theory.  In this case, it's very hard to calculate the pigouvian cost (externality value) of paying a given ransomware demand, so taxation seems a weaker tool than just outlawing it (note: I don't know enough to really advocate for this either, I'm just comparing the two).

Take what used to be a property tax and turn it into a sales tax, which is levied on sales of property.

This would alleviate a lot of my concerns.  Sales taxes (on actual sales, as long as it's not imputed or assumed-sale where no money is actually changing hands) have a ton of advantages, not least of which is that the money is ALWAYS there to pay the taxes.  I suspect it won't satisfy the Georgists, though, as it doesn't capture appreciation in value if there's no sale for decades or longer.  Maybe - it does remove the incentive for empty-land speculation.

[I have bad self-control, so my statements that I'm bowing out of this don't seem to have stuck.  Apologies.]

3. Practical: we need to calculate what fraction of the value is provided just by other people.

4. Practical: we tax that fraction of the price, and of the income from the property.

The theoretical calculation problem is in what you call "practical".  There's no actual price signal or ground truth for that portion of the value.  The use of the property combines land and improvement values in a way that's idiosyncratic and inseparable.  That calculation is going to be made up, and wildly inaccurate and unsupportable even in the ideal world where it's not politically adjusted.

I'm not sure how to interpret 

Your suspected answer is how current implementations of the system work, but

I know that.  I don't know how the proposal differs from "run it the same way, just with higher values framed as land-value tax".  If the amounts are low, it's workable.  If the amounts are high, it's not.  I don't know if the proposal is to somehow separate the ownership of land and improvements, or if there's something else that makes it practically different from "much higher normal property taxes".  If it's NOT just using a land-value justification to raise the dollar amounts greatly, please educate me.

Your first alternative hypothesis (there's ALREADY a path to extinction) is clear to me, and it is unclear what sign or magnitude of change in that risk that AI will bring.  Which makes your title a bit suspect - AI doesn't bring a risk of extinction, it "merely" changes the likelihood, and perhaps the severity, of possible extinction paths.

I think there's an underlying failure to define what it is that's logically conceivable.  Those math problems have a formal definition of correctness.  P-zombies do not - even if there is a compelling argument, we have no clue what the results mean, or how we'd verify them.  Which leads to realizing that even if someone says "this is conceivable", you have no reason to believe they're conceiving the same thing you mean.

They can only make decisions that generate enough income to pay the taxes

Out of curiosity, how is this different from current property taxes, or from mortgages for that matter?

Most of my objection (and confusion that it gets handwaved away so often) is NOT that the unimproved theoretical value of land could be taxed.  It seems complex and unnecessary, but that's not a unique problem with tax proposals.

My objection is to the core of the proposal that it's taxed at extremely high levels, based on theoretical calculations rather than actual use value.

I have plenty of other concerns (like how it ACTUALLY works, for improved properties - do we split all deeds in two, one for the land and one for the improvements, and allow people to sell them separately?  How does that work?), but they weren't the crux of THIS discussion, and I suspect the answer is just "no - this is just regular property taxes, just calculated differently (and much higher), we still take the improvements if the tax is unpaid".

people don't like big sudden changes that cost them a lot of money.

People don't like ANY changes that cost them a lot of money. The only saving grace of gradual rollout is that it's easy to continuously delay the next step until it gets fully killed before going too far.

Any proposal that amounts to "nationalize Trillions of dollars worth of land value (or of net present value of future rent streams, same thing), without compensation" is going to face backlash from a lot of people, including me.   REGARDLESS of timeframe or gradualness.  

if a sufficient bribe to landowners would cost less than indefinitely continued rent-seeking. 

Note that if made in public, for legitimately-owned assets, and voluntarily accepted, the term is no longer "bribe" but "purchase".  A whole ton of my objections go away if the government (or even a private entity) is buying land and then figuring out the best use for it, charging optimal rent to the people who own the improvements separately.