I think that what we know doesn't exclude Doomsday Argument completely but only narrows its applicability. From what we know about x-risks, two theories seems equally plausible: there is 95 per cent probability of human extinction in next millennia, but after that we will live millions years - and there is 100 per cent probability of human extinction in next millennia. Doomsday argument strongly favors the second theory (there is a way to escape its curse e.g. by creating simulations, but first we need to accept it).
In other words, DA sneaks inside small uncertainty that remains - especially the one about our remote future.
I see where you are coming from, but I am not convinced.
As a general rule, I subscribe to the idea that reasoning (or arguments) take in a set of observations and facts, and they output consequences of those observations. Information that we didn't put in at the beginning, isn't accounted for. The Doomsday argument doesn't include any information about x-risks. It doesn't include any knowledge about AI or nuclear weapons. It would give the same conclusion in a world without those things. Therefore, I argue, it is not accounting for them.
Your take on x-risks and 95% chance of human extinction in next millennia is one I can understand. But, to me, the idea that the Doomsday argument, that takes as its input no information at all about AI or x-risk, somehow produces data pertaining to AI or x-risk, seems kind of "magical".
The counterargument to my position would look something like saying "I am not an impartial observer looking at the universe from the outside. I reside in a human mind, specifically this human mind born at this time, and that fact is an extra data point that I am updating on." I still don't buy the counterargument myself. Partly because (as in the post above) I think the information provided is miniscule even if we accept the premise. And partly because I don't really accept the premise. The reasoning for why I don't accept the premise is more hazy but it kind of comes down to the causal connection between the number of previous humans and x-risk seeming super dubious. In a world where the medieval period had lasted an extra few thousand years AIs wouldn't necessarily be any safer. If an ancient person miraculously emerges from a block of ice alive, then I don't think they should have different views on x-risk by virtue of having been born a few thousand years earlier. (I am sure their lived experience would give them different views on x-risk, but not their birth order). I really do think that the answer to the question "If their are going to be trillions of people in the future, isn't it a weird coincidence that I find myself born all the way back here in history" is "You were never a disembodied soul that was inserted into a human selected randomly from all humans in all determinisitic history. If we swapped "you" and Julius Ceaser's souls around, moving no information at all, then that isn't a distinct timeline that we should account for when counting possible worlds, its just the same timeline up to a re-labeling."
First, I should note that I count as real observers only those who are in the same epistemic situation as me, so are anthropically-aware observers. This excludes typical counting of human births and instead we should count only appearing of antropically-aware observers. They appeared only in 1970. (This view is super obvious to me, but most people don't agree.)
Second, there are two types of Doomsday Argument: Gott's version which just predicts probability of disappearing of the reference class (not necessary extinction), and Carters-Leslie's one which compares two hypothesis. What I wrote in the comment above is about comparing two hypotheses, so it Carter-Leslie's DA. This version is aware about x-risks exactly because we put this knowledge into hypotheses.
If we use Gott's version to the anthropically-aware observrs' time of birth, we get 50 per cent chance of extinction in 2082. This surprisingly coincides with predictions based on x-risks. (If we use birth rang of entropically aware observers, not time, we get even less remaining time, likely mid-century).
I think that only including anthropically aware observers is maybe kind of reasonable, if we want to take any version of the Doomsday argument seriously then the reference class is very important, and should almost certainly exclude ants for example, and where the boundary is put is not obvious to me.
I think the idea that Anthropically aware observers are such a recent phenomenon seems off to me. When I was about 12 I got really confused about "Why am I not an ant? Isn't that really weird?". Now, its possible that cartoons with people swapping bodies, and the modern ways of teaching children to be nice ("You wouldn't like it if you were him would you?"), plus sayings like "what a time to be alive" kind of mean that anthropics is "in the water" these days and me thinking about it at 12 was a reaction to that. But like, Buddhists and Hindus have been thinking about reincarnation for millennia. I think in some versions they even have a "spin the wheel" thing to determine what you will incarnate as next. So I think that the anthropic reasoning goes back much further. Its probably the combination of mathematical language and the somewhat spiritual vibe together that is very post 60's.
(By "spiritual vibe" I mean that all these arguments require some kind of cosmology where it makes sense to think that "I" "could" have been someone different. For some definitions of the words "I" and "could" that might be very hard to pin down).
Jokingly, a funny thing about having your limited reference class, is that we can imagine a secret society that forms and destroys all public record of anthropic arguments. They then only induce a small number of people to their secret group and assassinate anyone who re-invents anthropics. By rationing the reference class in this way they can improve the chances for humanity to make it.
I am curious, in the two hypothesis case, how much information do you think you get from the Doomsday argument? Like, is it a major update or a tiny suggestion? Even in the binary case, if we assume two universes play out in parallel, one where trillions of people are born over a long timescale and one where things go extinction moderately quickly. We are inserted into one of those people "at random", and find ourselves here. If we are in the crowded universe its a weird coincidence we are so early in its history. But, if we are in the sparsely populated universe its a weird coincidence that we are not in the crowded one.
I also had such thoughts as a child, so reference class can be larger. But an interesting observation is that changing the reference size class has small effect on Doomsday argument's prediction because of the exponential population growth in recent times. If we take birth rang between all humans, we get 5 per cent of extinction in 21 century and if we take anthropically aware-observers who read LW - we get like 50 per cent. It is a small difference for such vague prediction.
I had similar idea about manipulating the known number of observer in order to escape DA curse. Could be interesting EA topic.
More generally, in my view, it is reasonable to have some logical uncertainty about Doomsday Argument validity. It seems reasonable to give it like 30-40 per cent credence, imho.
The informational update in Carter-Leslie version depends on the size of the long hypothesis. If we expect to survive until the end of the universe 100 billions years from now, the DA-update is like 100 million times. But it is known bug of the Carter-Leslie's DA that it depends on the size of long hypothesis. In Gott's version there is no such bug in open form, but our surprise by the idea that we will not colonize the Galaxy assumes that we had a theory that we should do it, and here is the contrast produces surprise.
Most things that we model as random, are in fact deterministic. The outcome of a dice roll is determined by the initial momentum and angular momentum of the die, computer generated random numbers are chosen in a deterministic way by hard-to-predict programs.
Sometimes, the extent of the non-randomness is very clear. As a child I enjoyed the "Lone Wolf" choose-your-own adventure books. In these books, you sometimes had to resolve fights with monsters, which were determined using random numbers between 0 and 9. To generate these numbers the book provided this table:
You close your eyes, get a pencil, wave it around and stab it down into the book to get a number. This system is frustrating and terrible. You very quickly learn how to aim at the numbers you want. You don't want to cheat, its just that not-aiming is harder than aiming. You come up with some new system, where you rotate the book to some skew angle to throw yourself off, you wave your arm dramatically, as if that would help, and you hit a 3. A 3! That is a bad result, so you must have succeeded at not aiming. This technique is good, lets keep using it. You then hit the same 3, or the 2 next to it, for each of your next 5 goes.
In the, very likely, event you have not played Lone Wolf you may have encountered a similar experience in certain card games, which require you to shuffle a deck of cards that can get quite small (Dominion for example). If you get down to shuffling about 5 cards, then doing a "shuffle" motion doesn't randomise them at all. Not because the physical motion looses its power, but because the number is small enough that you can easily track where the most important card or two is. At this point a different randomisation procedure is needed, usually one where another player helps you randomise the cards works fine.
Sticking to cards, a well-shuffled deck of cards looks, to me, a pathetic human, like its random. To Superman with X-ray vision the top card is clear as day. Perhaps to a very perceptive being without X-ray vision trying a shuffle motion on 52 cards would find it just as futile as I find it on 5 cards. This hypothetical being might also find rolling dice to have all the problems I have with the Lone Wolf stabby number grid. It knows that a little more torque will flip it from a 5 to a 6, it can't pretend to itself that it doesn't know.
So, things that we model as random, because they look random to us, might be better modelled as non-random to "stronger agents", for example ones whose grasp of mechanics means that they cannot actually get unexpected dice rolls even if they try.
This post is about the opposite, weaker agents. Imagined agents who have less information than we typically do. Something we would never model as random, they do.
As an extreme illustrative example, lets imagine that some poor person suffers from a brain injury or stroke. They remember the names of the days of the week, and they remember that these days form a cycle. But, the brain injury means that they completely forget the order of the days. They know that today is Monday. They tell you that it "just like a dice roll", there are six possible days it could be tomorrow: Tuesday, Wednesday etc., and its equally likely to be any of them.
You would, I suspect, find this jarring. Its so obvious to you that tomorrow is Tuesday that describing it as random feels viscerally wrong. Even leaving that aside, the fact that it is cyclical, means that even in ignorance calling it "random" feels off somehow.
Why would we even think about weak agents? Its because they make very good examples for learning or teaching some concept in probability. A weak agent who knows only that an election had 3 candidates, candidates A, B and C, can be imagined to give a 1/3rd chance of each candidate winning, and this example could teach us something very basic about probability. Further, we know that the 1/3rd answer is, given this weak agent's state of knowledge, the right answer. If we go for a more "real world" example where the agent has a much bigger knowledge space its very hard to see what the right answer, given that bigger knowledge base, actually is. (What is the correct probability to give for Count Binface winning the Clackton by-election based on everything I know? Or for your preferred candidate in the next election you care about given everything you know?)[1]
When we do basic probability problems, guessing the top card of a deck etc, we are always implicitly (or explicitly) thinking about clean situations where you have nothing else to go on. You don't know that the deck used to be in state X and was badly shuffled, you can't read the body language of your opponent or whatever. We can imagine an annoying student in a maths class derailing our nice probability examples with questions like "But what if I am really good at reading my opponents microexpressions?".
I think that this is a part of why various anthropic type discussions are so confusing. Whenever an anthropic type argument is advanced, we want to focus on the actual substance of the problem, and not get derailed by the annoying student asking about reading body language or making other Sherlock Holmes inferences. This means that the character in the anthropic situation can never be ourselves, it always has to be a hypothetical, weaker character with a lot less knowledge than we have.
I want to concentrate on one specific weak agent, the opinions of which I think are given far to much weight. I will call this hypothetical character the "Doomsday Weak Agent" (DWA).
First, lets look at a few weak agent who know basically nothing, and are forced to make a wild guess.
[1] Agent [1] is given a randomly selected "thingy". She is told that each thingy is numbered 1 through N where N is the total number of thingys. Her thingy has the number 27 on it. Based only on this miniscule amount of information, how many thingys should she expect there are in total?
The best guess is 54. It would be weird for her to have been given one of the first or last of the thingys, she was probably given one close to the middle. So double the number given for the best guess.
[2] Agent [2] knows that the number of whatsits produced last year was 100. He doesn't know what whatsits are, how they are produced, or whether the economy is growing or shrinking. What is his best guess for how many whatsits will be produced next year?
100 of course. It would be weird for him to have been given an unrepresentative year, so we assume that the year given and the next one will both be about the same. Predict 100.
[3] Agent [3] is given the number of blegs produced every year going back 1000's of years. She is asked to guess the number of blegs that will be produced next year. Looking at her data, it turns out that the rate of bleg production matches an exponential curve.
So, she should probably interpolate that curve out another year.
Now we have our weak agents, all each of them knows is the information in their respective briefs ([1]-[3]), and we have already asked and answered some textbook probability questions about what the right answers they should give out are.
Number [1] is the Weak Doomsday Agent. The "thingys" are human beings. The literal only piece of data this hypothetical agent has about our entire universe is that they are human number 120 billion. They know nothing else about what humans are, how they are produced or how many were made last year. So we are put into situation [1]. Given these constraints on our (imagined) state of ignorance the best answer we could give would be 240 billion. That is in some mathematical sense, the correct answer, for this extremely weak agent to give[2].
But, we could instead put our person who knows almost nothing into a situation like number [2], or number [3], or any number of other cases where the information available to them is incredibly restricted in a different way. The situation number [2] implies stable human production at a fixed rate forever, because they have only one data point and take the zeroth order Taylor series as the best guess. [3] gives a load more data points and they fit an exponential, predicting a constantly growing rate of human production forever.
[2] and [3] lead to very questionable predictions. The carrying capacity of the Earth, the death of the sun in a supernova or even the heat death of the universe all seem to prevent them from being exactly correct. But, none of those things (heat death etc) were given to the hypothetical weak agents looking at situations [2] and [3], so of course they are not accounted for. They are also not accounted for in [1].
So, going back to [1]. Why should we take it any more seriously that [2], [3] or any others? I think we shouldn't. It tells us what we should believe, if we were almost, but not entirely, blind to the world around us. We are not that blind. When trying to predict the future human population, we can actually look at population growth graphs. We can speculate about the probability of nuclear war. We can think about demographics and declining birth rates. We know more about the human population than just a number telling us how many came before.
If I have not yet persuaded you to jettison the Doomsday argument, I would now like to talk about the great civilization of Atlantis.
Atlantis was a small island in the mid-Atlantic whose inhabitants invented advanced technology far before the rest of the world. At about the same time the Pyramids were going up there were already many billions of Atlanteans (they had houses that were bigger on the inside, or something). In ~2000BC Earth was invaded by aliens, but the generous Atlanteans made a deal where all of the Atlanteans would be eaten by the ravenous aliens first, and the aliens, their appetites temporarily sated, would leave the primitive peoples of Earth (IE, us) alone for 4,000 years before eating them all. The hungry aliens are due next year.
Any reasonable person, if they could somehow believe this Atlantis story while remaining reasonable, would downgrade the number of humans expected to exist in the future, given that they just learned that hungry aliens are due to eat us all in a years time.
However, the DWA, well, they heard the bit about how there were "many billions" of Atlanteans. We are not human ~#120 billion after all, but number ~#300 billion once Atlanteans are included, and because the future probably contains as many humans as the past, we can assume even more humans in future than we did previously! The Doomsday thought experiment rules by fiat that their position in the list of all humans thus far is the only fact they are allowed to update on.
Conclusion
I totally accept that the DWA is making the right predictions given their incredibly limited state of knowledge. However, exactly because their state of knowledge is so limited, I don't think we should take their opinions very seriously.
This is related to the idea of inside view vs. outside view reasoning. Where the inside view is us, given what we actually know, and the outside view is a weak agent who only knows a subset of the facts that are available to us (https://www.lesswrong.com/w/inside-outside-view). This, of course, implies that there are many possible outside views, depending on what facts we choose to throw out and which ones to keep (which is discussed already at the linked lesswrong wiki page).
The point of this post is to criticize the Doomsday argument on non-anthropic grounds. However, I can't resist somewhat engaging with the anthropic elements. For the Doomsday argument to work I am supposed to assume that I am a randomly sampled human from all the humans that will ever exist. We are supposed to not worry about questions like: "Could I have been born before my parents? Or them before their parents? Doesn't this make my position in human history strongly nonrandom?".
To take an analogy. Is today a day that was chosen randomly from all the days that you will ever live? If you think so, then a cousin of the Doomsday argument says you are probably about halfway through your life. But, don't worry, if you make it through today then you will still (in expectation) be halfway through you life tomorrow! If you don't think it is valid to consider "today" to be randomly chosen from all the days in your life (after all, it couldn't have come before yesterday), then why do you think it is valid to consider yourself to be a randomly chosen human from all those that will ever exist?