To respond to your thinking (in the linked blog post) that, to a first order approximation, if we find an AI in the alien message we should run it:
The preceding analysis takes a cooperative stance towards aliens. Whether that’s correct or not is a complicated question. For the most part, I think growing the pie by enabling intelligence to access more of the universe, is probably the first order term here. That might be justified by either moral arguments (from behind the veil of ignorance we’re as likely to be them as us) or some weird thing with acausal trade (which I think is actually relatively likely).
The moral argument is not very compelling to me, and I think the acausal trade argument depends on the aliens using UDT, and the aliens thinking there's enough logical correlation between them and us (even though they're superintelligent aliens/AIs and we're barely intelligent primates (or rather, a probabilistic mixture of evolved beings that are barely intelligent enough to have built the beginnings of a technological civilization)). If either of these assumptions fail we'd be in trouble. In general I'd only be comfortable with doing anything like acausal trade if I had superintelligence and had solved metaphilosophy, which confirms that UDT (or something like it) is correct, and I could predict what decision theory the aliens are probably using, and calculate "logical correlation" using some sort of philosophically sound method, etc.
The preceding analysis also implicitly presumes aggregative values—for which “twice as big is twice as good.” With respect to more easily-satiable values, like the desire to live out a happy life, I suspect that SETI is an even better deal. I’d expect the message to be something that allows humanity to keep living a happy life and to benefit from significant technological acceleration, since the benefits of killing us are de minimis and we’d have been willing to pay a lot to prevent it.
A CDT agent wouldn't care that we’d have been willing to pay a lot to prevent it, right? I don't think the benefits of killing us are necessarily de minimis, either. It's only that if the aliens or the message (seed AI) can very confidently predict that we won't ever be a threat to its plans. If we might be a threat (for example if there's a risk that some faction of humans might disagree with letting the seed AI loose, and try to stop it before it becomes too powerful), it could easily be safer to kill us (for example by using a biological weapon) and then proceed without any chance of interference.
Overall I think that a 1% absolute reduction in extinction risk is a not-crazy estimate for the value of successful SETI in a sparsely populated universe.
I'm very confused that you'd even consider taking this gamble, if winning gives 1% reduction in extinction risk but losing gives 100% extinction risk. It seems implausible that we could reduce our uncertainty about the alien AI being unfriendly to <\1%, at least before we become superintelligent, etc. What am I missing here?
If it seems plausible that there are aliens, I think "figure out what to do" would become a high-priority item, and I think there is a very significant chance "definitely don't run it" would be the right answer and that the main resulting intervention would be to push hard against passive SETI (about which people are horrifyingly unconcerned).
A CDT agent wouldn't care that we’d have been willing to pay a lot to prevent it, right?
Unless its predecessor entertained the possibility of being in a simulation run by a civilization l... (read more)
ETA: Contest is closed.
Suppose there was a large alien civilization halfway across the observable universe, using a galaxy's resources to try to get our attention. Would we have noticed? What if they were using 0.1% of a galaxy's resources, or 1000 galaxies' resources?
I've argued recently that such an alien civilization is (a) not that unlikely a priori, even given that there aren't any closer aliens, (b) potentially really important to notice.
I believe the answer to my question is probably "definitely." But I can't tell with any confidence, so while it's probably definitely it might be maybe and could be probably not. I'd like to know the answer, but space isn't my thing.
I'm offering a prize for anyone who answers this question. To be a bit more precise:
A simple example of a strategy is to create a really bright beacon somewhere far away from any galaxy, which looks weird in some way. I expect (based mostly on super informal discussions with Anders Sandberg and Jared Kaplan) that this strategy is good enough, i.e. that 0.1% of a galaxy's power is plenty to make a beacon that would be really obvious to us from halfway across the universe. But I'm definitely not sure. The beacon can have a weird spectrum, or flicker in a strange way, or only be active 1% of the time (but be 100x brighter), or whatever.
Note that an answer needs to make reference to the astronomical observations humanity has actually made, e.g. how long telescopes of a particular strength have spent looking at any particular part of the sky, and what kinds of patterns would have been noticed.
With respect to the capabilities of the alien civilization, I'm an unapologetic techno-optimist. If it's within the energy budget, I'm probably willing to believe they can make it happen unless it sounds super crazy. For 1x and 1000x questions, it's fine if they want to grossly disfigure a galaxy if that would be the best way to be noticed. For the 1/1000 question, grossly disfiguring a galaxy isn't allowed unless we can be pretty confident it doesn't reduce the usefulness of that galaxy by >0.1%.
I'm also basically happy to assume that they know exactly what our civilization is looking for and so can optimize their solution to be noticeable to us. (After all, they've run a billion billion simulations of civilizations like ours, they know the distribution, they can spend 5x as much energy to cover the whole thing.)
I don't care about whether we'd notice "things the aliens would want to do anyway," because I have no idea what aliens would want to do and have limited confidence in our ability to make prediction. In particular, it seems plausible that they would blend in with the background by default (e.g. maybe something like aestivation hypothesis is true). I'm much more interested in analyzing deliberate attempts to be observed, since those allow us to argue "If there exists a cheap way to be noticed, and they want to be noticed, they'll do it."
Prize
Note: prize is no longer available.
I'm offering a prize for a convincing answer to this question.
Initially the prize is $100. It increases by 10%/day, until capping out at $10,000 in 49 days.
Submit by writing a comment on this post.
The prize starts out low because I think this might be a really easy question. Feel free to try to be strategic if you want. If you get scooped because you are waiting for the prize to grow, I have zero sympathy.
The criterion is "Paul is convinced." Citations and clear explanations are probably helpful. In general sources don't have to be super authoritative; if you cite Wikipedia I'd prefer a citation to a historical version of a page before the contest started, just to rule out hijinks.
You are allowed to just link to an existing analysis that covers this question, or link with a small amount of extra work, if that's convincing. Assuming the linked explanation was written before my blog post, you'll get the prize, not the author of the linked post. The purpose of this prize is to buy information, it's not like the alignment prize.
I expect that winning submissions will be relatively short, probably just a few paragraphs with some links and calculations. You can take longer if you want, but I assume no responsibility for the harm thereby done to the world.
I reserve the right to be arbitrary in evaluating submissions. I am not going to feel guilty about it. If your willingness to participate depends on me feeling guilty about people who spent a bunch of time but who I unfairly rejected, then please don't participate.
I may give partial credit if something seems like a useful contribution but doesn't resolve the question completely (even if it's just a short comment with a pointer to a useful resource).
I may give feedback in the comments.
If you think this isn't the best thing for me to do with my time and are worrying about my life decisions---it was either this or spend my own hours looking into the question. Don't worry too much, this shouldn't take long.
Note: prize is no longer available.