The assumption that we can colonize the stars is core to the Astronomical Waste Argument made in favor of working on existential risk reduction. If this assumption is weakened, so is the case for prioritizing work existential risk reduction.

Most things are impossible. Perhaps our belief that we could possible colonize the stars is based only our ignorance. If we actually tried to colonize the stars (or simply tried to actually look into the possibility), we would find that we shouldn't take it for granted at all that space colonization is a realistic possibility.

Summary of the Astronomical Waste Argument

Nick Bockstrom's 2003 paper, Astronomical Waste: The Opportunity Cost of Delayed Technological Development:

With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore a corresponding opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large.

Bostrom arrives at different estimates of the potential number of human minds depending on whether we are satisfied with running "human" minds on computers or wish to stick with biological instantiation.

Using digital instantiation:

As a rough approximation, let us say the Virgo Supercluster contains 10^13 stars. One estimate of the computing power extractable from a star and with an associated planet-sized computational structure, using advanced molecular nanotechnology, is 10^42 operations per second. A typical estimate of the human brain’s processing power is roughly 10^17 operations per second or less. Not much more seems to be needed to simulate the relevant parts of the environment in sufficient detail to enable the simulated minds to have experiences indistinguishable from typical current human experiences. Given these estimates, it follows that the potential for approximately 10^38 human lives is lost every century that colonization of our local supercluster is delayed; or equivalently, about 10^29 potential human lives per second.

Using biological instantiation:

Suppose that about 10^10 biological humans could be sustained around an average star. Then the Virgo Supercluster could contain 10^23 biological humans. This corresponds to a loss of potential of over 10^13 potential human lives per second of delayed colonization.

Bostrom clarifies that not only utilitarians should care about this immense potential value which might be reached:

Utilitarians are not the only ones who should strongly oppose astronomical waste. There are many views about what has value that would concur with the assessment that the current rate of wastage constitutes an enormous loss of potential value. For example, we can take a thicker conception of human welfare than commonly supposed by utilitarians (whether of a hedonistic, experientialist, or desire-satisfactionist bent), such as a conception that locates value also in human flourishing, meaningful relationships, noble character, individual expression, aesthetic appreciation, and so forth. So long as the evaluation function is aggregative (does not count one person’s welfare for less just because there are many other persons in existence who also enjoy happy lives) and is not relativized to a particular point in time (no time-discounting), the conclusion will hold.

These conditions can be relaxed further. Even if the welfare function is not perfectly aggregative (perhaps because one component of the good is diversity, the marginal rate of production of which might decline with increasing population size), it can still yield a similar bottom line provided only that at least some significant 5 component of the good is sufficiently aggregative. Similarly, some degree of time discounting future goods could be accommodated without changing the conclusion.

Clearly, the extent to which we can actually colonize star systems beyond our own affects how strong an argument there is from astronomical waste (or as I would rather call it, our astronomical potential). If we can in fact be confident that we can colonize the entire reachable universe, that might be 10^17 stars instead of the 10^13 in just the Virgo Supercluster. An even stronger argument than Bostrom states. On the other hand, if we can't even colonize beyond our star system, we're just at 10^0 stars. Then there'd be no astronomical argument at all.