I think this approach wouldn't work for rationalists, for two reasons:
Can't speak for Said Achmiz, but I guess for me the main stumbling block is the unreality of the hypothetical, which you acknowledge in the section "This is not a literal description of reality" but don't go into further. How is it possible for me to imagine what "I" would want in a world where by construction "I" don't exist? Created Already in Motion and No Universally Compelling Arguments are gesturing at a similar problem, that there is no "ideal mind of perfect emptiness" whose reasoning can be separated from its contingent properties. Now, I don't go that far - I'll grant at least that logic and mathematics are universally true even if some particular person doesn't accept them. But the veil-of-ignorance scenario is specifically inquiring into subjectivity (preferences and values), and so it doesn't seem coherent to do so while at the same time imagining a world without the contingent properties that constitute that subjectivity.
I think ancient DNA analysis is the space to watch here. We've all heard about Neanderthal intermixing by now, but it's only recently become possible to determine e.g. that two skeletons found in the same grave were 2nd cousins on their father's side, or whatever. It seems like this can tell us a lot about social behavior that would otherwise be obscure.
It took me years of going to bars and clubs and thinking the same thoughts:
before I finally realized - the whole draw of places like this is specifically that you don't talk.
In the end, despite cheaper feed, the daily cost of horse upkeep (the horse’s subsistence wage, if you will) was higher than the horse’s productivity in its transport and agricultural roles.
Presumably the absolute productivity of a horse (the amount of land it can plow or stuff it can haul) has not changed. So this only makes sense if the market value of the horse's labor has declined even faster than the price of feed. Is that the case?
rather than, say, assigning equal probability to all strings of bits we might observe
If the space of possibilities is not arbitrarily capped at a certain length, then such a distribution would have to favor shorter strings over longer ones in much the same way as the Solomonoff prior over programs (because if it doesn't, then its sum will diverge, etc.). But then this yields a prior that is constantly predicting that the universe will end at every moment, and is continually surprised when it keeps on existing. I'm not sure if this is logically inconsistent, but at least it seems useless for any practical purpose.
In the following, I will use the term "my DIT" to refer to the claim that:
In some specific non-trivial contexts, on average more than half of the participants in online debate who pose as distinct human beings are actually bots.
I agree with this version, and I was surprised to see that the Wikipedia definition also includes the bit about it being a deliberate conspiracy, which seems like a strawman, since I have always understood the "Dead Internet Theory" to include only the first part. There's a lot of stuff on the internet that's very obviously AI-generated, and so it's not too far a stretch to suppose that there's also a lot of synthetic content that hides it better. But this can be explained far more simply than by some vast conspiracy - as SEO, marketing, and astroturfing campaigns.
If Dead Internet Theory is correct, when you see something online, the question you should ask yourself is not "Is this true?" but "Why am I seeing this?" This was always the case to some extent of any algorithmically-curated feed (where the algorithm is anything more complex than "show me all of the posts in reverse chronological order"), but is even more significant when the content itself is algorithmically generated.
If I'm searching online for information about e.g. what new car I should buy, there's a very strong incentive for all the algorithms involved (both the search engine itself, and the algorithm that spits out the list of recommended car models) to sell their recommendations to the highest bidder and churn out ex post facto justifications explaining why their car is really the best. These algorithms are almost totally uncorrelated with the underlying fact about which car I'd actually want, so I know to consider it of very little value. On the other hand, I would argue, asking a bunch of random acquaintances for car recommendations is much more useful because, although they might not be experts, they were at least not specifically selected in order to deceive me. Even if I ask a friend and they say "Well, I haven't bought a new car in years, but I heard my coworker's cousin bought the XYZ and never stops complaining about it", then this is much more useful information than anything I could find online, because it's much less likely that my friend's coworker's cousin was specifically being paid to say that.
More broadly, on many questions of public concern there may be parties with a strong interest in using bots to create the impression of a broad consensus one way or another. This means that you have no choice but to go out into the real world and ask people, and hope ideally that they're not simply repeating what they read online, but have some non-AI-mediated connection to the thing.
Isn't this what we experience every day when we go to sleep or wake up? We know it must be a gradual transition, not a sudden on/off switch, because sleep is not experienced as a mere time-skip - when you wake up, you are aware that you were recently asleep, and not confused how it's suddenly the next day. (Or at least, I don't get the time-skip experience unless I'm very tired.)
(When I had my wisdom teeth extracted under laughing gas, it really did feel like all-or-nothing, because once I reawoke I asked if they were going to get started with the surgery soon, and I had to be told "Actually it's finished already". This is not how I normally experience waking up every morning.)