(From the vast heaps of discarded material from my 2024 attempts at drafts for "If Anyone Builds It, Everyone Dies".)

Welcome to today's quiz show: Can a superintelligence do THAT?

With us today we have our contestants:  Msr. Soberskeptic and Msr. Oldhand!

Soberskeptic:  "I'd just like to say, however this quiz show ends up being judged, I will consider that judgment to be objectively ridiculous -- there's no way anyone can know what a superintelligence could do, in advance of empirical observation.  So I'm just here to say what I consider to be true, and I suppose these credulous fools will mark me down as wrong every time I say 'No it can't'. In the unlikely event they decide I've won anything, good for them and I'll be grateful for whichever prize.  Game-show money isn't enough to get me to lie."

A very reasonable attitude, Msr. Soberskeptic!  You'll shortly see how we handle that dilemma!  And you, Msr. Oldhand?

Oldhand:  "Don't worry, Sober!  I'll let our hosts know if they've gotten any of the answers wrong."

Also a very reasonable attitude!  Now for our first question:

Suppose a digital device contains a secret encryption key that it is repeatedly using to sign messages.  Also, this device has a power LED.

Could a superintelligence point a nearby video camera at the device, and read off the encryption key, by looking at the power LED?

Soberskeptic:  "Pffft.  Obviously not.  How would that even work?"

Oldhand:  "Ah, I perceive you are not a computer security expert!  Many particular steps in encryption draw slightly varying amounts of electrical power to the processor, in a way that correlates to exact features of the secret key.  So the power LED, if it's connected directly to the electrical circuit powering the processor, might get a tiny bit dimmer.  I'll say... ten percent probability that a superintelligence can do that with a randomly selected electronic device that has a secret key and a power light."

Soberskeptic:  "And now they're going to mark me down, but seriously, give me a break here, audience.  Any fluctuations like that would be too tiny to pick up with a nearby video camera.  Fluctuations like that would be too fast to pick up at the 60-frames-per-second measuring speed of a video camera.  ...And now, they're going to say something stupid about how a superintelligence could hack into the factory and deliberately introduce a design flaw into the power LED."

And Msr. Soberskeptic... is wrong!  A superintelligence can definitely do that to many current electronic devices!

Soberskeptic:  "Uh huh.  And you know this how?"

Why, because it's already been done by merely human computer security researchers, of course!  See "Video-Based Cryptanalysis: Extracting Cryptographic Keys from Video Footage of a Device's Power LED"!  In which they recovered a 378-bit key from a Samsung Galaxy S8, by pointing an iPhone 13 camera at the power LED of a pair of speakers connected to a USB hub that was being used to charge the Galaxy S8!

The trick, if you're wondering, is that a video camera uses a rolling shutter to scan pixels across its field of vision.  The actual light-sensitive element measures much faster than 60 measurements per second!  An Internet attacker can modify a video camera to point to one place instead of scanning across, and measure the device's power LED intensity millions of times per second!  Though a successful attack still requires the device to continuously use its key for an hour or so.

Oldhand:  "Gosh.  Being able to do it off the power light of a speaker connected to the same USB hub was a bit more extreme than even I thought.  But I guess that figures; as anyone in computer security knows, computer-security hacks are often even more impossible than you were expecting."

Soberskeptic:  "Uh huh.  I see the gimmick.  You're leading with an example that's actually provable, and now, you're going to ask us to swallow way weirder stuff that nobody can prove."

On to our next question!  Is it pragmatically possible, even for a superintelligence, to cross astronomical distances much faster than light?  Without having first sent ahead any sort of wormhole-receiver that traveled slower than light?

Soberskeptic:  "Unless you're going to play some silly game about 'going faster than the speed of light in some medium', like Cherenkov radiation from particles going faster than the speed of light in an atmosphere, I'll answer... that you have no possible way of knowing about the ultimate limits of technology!  People have theorized incredibly speculative ways to travel faster than light, like the Alcubierre Drive, if you could supply huge amounts of exotics like 'negative energy'.  I'd guess none of them work.  But it's only a guess, and you don't know any better than I do about that.  Our current theories of physics could be wrong, after all; humanity didn't have the one true theory of physics 200 years ago, and maybe we don't have it today, either.  Our descendants millions of years from now will be working very hard to go faster than light if there's any possible way of doing so; and nobody can say if they will succeed."

Oldhand:  "And my answer is:  No!"

Soberskeptic:  "What do you mean, no?  Aren't you just going to say superintelligences can do anything whatsoever, because magic, and how can we mortals possibly constrain them?  Isn't that your whole shtick?"

Oldhand:  "Ha! Sounds like somebody has been heavily misrepresenting a couple of decades' worth of careful analysis to you."

And Oldhand... is once again correct!  The Earth's Sun is relatively young in age among all the sorts of stars we'd expect to be able to support biological life, over the history of a whole universe.  This timing strongly suggests that we find ourselves existing now, around that relatively young star, because later on, the stars would have already been colonized; and then new life like Earthly life would not have had later chances to emerge!  This would still hold true even if you imagine that it's colony ships full of tinned biologicals traveling between stars, rather than machine intelligences.

If arbitrarily advanced aliens could travel much faster than light, we should observe our solar system being even younger!

(See "grabby aliens" for more about the sort of statistical inference which underlies this sort of reasoning.  There is more detailed grist and confirmation for this kind of inference than one might expect; eg, the amount of time that remains for Earth to support life, before the oceans evaporate, is around what we'd expect to see under that statistical model.)

Oldhand:  "Or to simplify:  Besides the reasons from our current model of physics, we have observational evidence that nobody -- no machine life, no organic life -- can travel much faster than light.  Because then the universe would fill up very fast, even if one single merely biological species decided to let two parents have four kids every hundred years; and the universe doesn't look full.  Aliens have not mined out the ice or metals of our asteroid belt; they have not surrounded stars with swarms of solar panels that would intercept the solar energy and reradiate it as infrared.  Earth emits enough radio to be detectable from many light-years away, and nobody else like that seems to be around.  Combine it with the standard model of physics seeming extremely strident about the lightspeed limit on causality, and I'm willing to make the call:  We're directly observing that the limits of technology don't permit practical FTL travel -- at least, not substantially faster than light, and not running ahead of a slower-than-light wormhole receiver you have to send on first.  Physics mostly says you can't do wormholes either, but we don't observe that the same way."

Soberskeptic:  "Aliens could decide to hide, or choose not to reproduce; how can you know?"

Oldhand:  "What, every single species across billions of faster-than-light years all chose not to reproduce quickly, every last one?  The super-hives, the super-dolphins, the super-octopi, the machine intellects, every last one of a million entity-kinds like that all decide to restrain their reproduction?  Or similarly:  All kinds of alien species are running around at FTL speeds, but prefer to hide more strongly than they want anything else they could get from mining asteroids, colonizing planets, or using solar energy?  One must distinguish possibility from probability, if this is meant to be inference about real life rather than the allowed premises for a fantastical story.  Also if arguendo all aliens refuse to reproduce or they hide forever, and this lets more species like humanity pop up, then most species like humanity should find themselves in a solar system that's born later in the universe's history! So your vastly strained hypothesis still fails to retrodict our observations.  I therefore conclude from theory plus observation that superintelligences cannot make a first crossing of intergalactic distances much faster than light; that the extremely strong implication of our best physical models, reflects a genuine constraint on intergalactic travel speed that cannot be pragmatically defeated by any amount of cleverness."

Next question, please!  Suppose you prove using valid logic that some computer program obeys a constraint like 'never writes to a particular section of memory'.  Suppose you have a machine-verified proof of this constraint, that includes all other software on the system, and the CPU and the memory and every other part of the computer.  Suppose it's a proof that goes all the way down to the factory layout of the transistors, and the usual simple behaviors for those transistors.  And suppose that the factory map of the transistors is accurate.

If the proven-correct computer program contains a superintelligence, can the superintelligence write to that forbidden area of memory?

Soberskeptic:  "What sort of axioms are the theorems being proven from, besides the true program specification and the true transistor layout?  Like, what proof system are the theorems being proven in?"

Great question, Msr. Soberskeptic!  Let's say: first-order arithmetic, aka Peano Arithmetic.

Soberskeptic:  "Then no, because if Peano Arithmetic is inconsistent then everything about reality is a lie.  Not even a superintelligence can violate actually-valid mathematical proofs.  This is going to be another place where even Oldhand says no, like for FTL travel, and this time Oldhand will be correct.  Thinking that a superintelligence can disobey the laws of mathematics, is about the clearest test I can imagine for whether you are holding even the tiniest pretense of putting forth a serious argument.  Ah, assuming you're not going to say something dumb about the superintelligence persuading a human to come over and write to that area of memory."

Great catch, Msr. Soberskeptic!  Say we're just talking about what the superintelligence can do by wriggling around its code inside the computer, not by persuading any outside people or hiring Taskrabbits over the Internet.

Soberskeptic:  "Then no."

Oldhand:  "Yes."

Correct again, Msr. Oldhand!

Soberskeptic:  "Wow.  I'm... kind of speechless.  Where you do you get the idea that a superintelligence can't violate our theories of physics, but it can violate math?"

Violating math is hardly required to disobey properties that have been mathematically proven true about a computer chip given the transistor wiring and the usual simple behaviors of transistors, Msr. Soberskeptic!  You just have to do something with the transistors that subverts their usual behavior!

For example, the "Rowhammer" class of computer-security vulnerabilities is that by using the right high-speed memory access pattern on some memory cells in DRAM, you can electromagnetically perturb bits in adjacent cells of DRAM!  And that's a big deal in practice, because once you start violating transistor-level assumptions like that, you can parlay it into all kinds of privilege escalations -- like if that flipped bit was doing something important to the system!

So just because you mathematically prove code to be safe given the transistor layout and the usual rules governing transistors, doesn't mean it could really constrain a superintelligence -- or even a program written by a savvy human security researcher!

Next question!  Is physically ripping out the Wifi antenna, Wifi chip, all USB ports, all peripherals, removing all power lights, all speakers, and any kind of hard drive that could be spun to make audible clicks -- leaving behind just the CPU and RAM and a very minimal motherboard -- enough to prevent a superintelligence inside that computer from communicating with the outside world?

Soberskeptic:  "Okay, I can kind of guess that the answer is going to be 'no' and involve some sort of amazing computer security trick that's already been done.  Probably you can make the capacitors whine in a way that imitates a human voice, or something."

Oldhand:  "Glad to see you're finally picking up on the pattern!  Just keep in mind, real life does work this same way, in terms of how much is possible to accomplish starting from extremely restrictive conditions -- even if you're not on a gameshow, and the answers aren't known already.  I'll also reply no."

Both of you are... correct and known to be correct!  Memory accesses can also be modulated to send out radio signals on GSM cellphone frequencies!  See "GSMem: Data Exfiltration from Air-Gapped Computers over GSM Frequencies"!  Though the existing demo required somebody with a hacked cellphone, not too far away, to pick up the signal.

Msr. Soberskeptic, what would you guess is the smallest possible size for a solar-powered factory system, that starts from completely raw materials found on a planet like Earth, and builds a full copy of itself?  And what would you guess is the minimum time for it to make a copy?  Assuming a superintelligence has designed the factory.

Soberskeptic:  "...right.  So, the obvious gotcha would be if you're going to point me to somebody's actual demo of a self-replicating factory that picks up two exactly right kinds of clay, and uses it to 3d print a copy of a 3d printer... wait.  Is the factory being operated by a hobbyist to build a copy of itself, or does the factory have to be fully automated?"

No external help, Msr. Soberskeptic!  The factory has got to build another factory on its own!

Soberskeptic:  "Okay.  In that case I am having trouble seeing a fully self-copying factory that starts from sheer raw materials and has to build a complete copy of itself including the onboard computers.  Because that's going to take, like, smelting copper, and I guess you can make a kiln just from clay but... okay, you know, I'm going to guess the gotcha is about somebody's theoretical design for a self-replicating factory.  And that it is merely ten cubic meters on a side, and... builds abacus-like physical logic out of the right kind of clay... and picks up wood and burns it for fuel, and supposedly replicates in just a week or so. But has never actually been built, because in reality it'd scoop up the wrong kind of clay and break every time it rained."

Oldhand:  "Well, I could hardly guess the smallest a superintelligence could get a fully self-replicating factory system operating in a real planetary environment.  But certainly underneath one micron on a side, replicating in less than an hour."

Soberskeptic:  "One micron!  Ha!  I take it you think nanotechnology is real, then?  Because even if our game-show hosts say somebody did the theoretical designs for a system like that, I'll say it's no coincidence that nobody has ever built the smallest part of one.  You just can't actually get machinery that small."

Sorry, Msr. Soberskeptic, but we're afraid you've overlooked an important practical example!  A tree is a self-replicating solar-powered factory that builds a complete copy of itself!  (Mostly out of mass it gets from carbon dioxide in the atmosphere, and water from rain; what it gets from the soil is trace nutrients making up around 1% of mass. That's why trees don't fall into giant holes in the ground underneath after turning too much dirt into tree mass.)

And while the tiny protein-making machine of a ribosome, inside a tree, may happen to be making only more tree proteins -- in principle, any ribosome is able to make every kind of protein, just by being told the amino acid sequence!  Nothing in principle stops there from being a tree that buds off mosquitos that inject botulinum toxin, for example!  So a blade of grass is a self-replicating factory, not just a replicator.

Oldhand:  "And small algae cells aren't far off from one micron on a side, and have a replication time of... I think around 6 hours in the fastest cases?  If you look at the theoretical analyses that have been done for crude air-eating nanotechnology, they come out at around 1 hour doubling time.  For actual E. coli bacteria it's 20 minutes, and for Clostridium it's 7 minutes, but those shouldn't count as a self-contained factory -- bacteria don't make all the compounds they eat, and rely on there being other life forms around.  Probably an actual superintelligence can get a solar-powered factory well below one micron and one-hour replication times, if it's allowed to redesign self-replicators from scratch; but that's one way of guessing the least it can do."

And now, our final question!  Msr. Soberskeptic and Msr. Oldhand, what sort of weird shenanigans do you think a superintelligence would be able to set in motion inside a human mind... by talking to a human?

Soberskeptic:  "Right, so, I suppose the idea, now, is that I'm supposed to think of all the things that humans have done by talking to humans?"

Oldhand:  "I mean, you could anchor yourself on human examples.  But maybe this is also an opportunity for you to depart from your previous habit, and stretch your imagination a tiny bit further?"

Soberskeptic:  "And I'm supposed to be scared of the superintelligence... because of how human con artists can talk people out of money?  Or, I guess, politicians talking people into voting for them... or prophets starting religions that propagate from person to person... I admit it's scary to imagine something being better than any human at that.  But there's no evidence that any mind can be better.  Maybe that's just something superintelligences can't realistically do.  Like, maybe there's some secret series of words you could say to some particular person to get them do things your way, but that doesn't mean even a superintelligence can figure that out by reading their social media profile."

Oldhand:  "Soberskeptic, none of the material in this gameshow has been about what actual superintelligences could do, except the part about ruling out faster-than-light travel.  It's been about what clever humans have already figured out what to do, starting from handicapped gameboards -- or, in one case, what natural selection has already done with algae cells.  The idea being to snap out of the mindset where, if something sounds hard and you can't figure out immediately how to do it, you conclude with great confidence that it's impossible, and dismiss any contrary thoughts as naive and no doubt ignorant of all the practical difficulties.  That's the lesson we learn from dealing with other humans -- who sometimes try to be a little clever, and sometimes succeed; and we ourselves have not always anticipated the way in which they will be clever.  What superintelligences could do against us would be a qualitatively different story."

Soberskeptic:  "See, the thing is, no matter how persuasive superintelligences end up, we humans are an ornery lot.  We can just refuse to believe anything the superintelligence tries to persuade us about, even if it is being very reasonable, or appealing to our greed, or whatever."

Oldhand:  "...I feel like this is the wrong level on which to be having this discussion.  You are imagining that your orneriness is an invincible shield that the superintelligence has to confront and defeat on its own level, by being persuasive the way a human tries to be persuasive.  Imagine something more like a computer security expert, seeing your brain as a system made of parts and searching a vast space of paths through causality.  If you think of a human brain's security condition as being, 'Do not accept invalid arguments which derive conclusions that didn't follow from their premises', or 'Don't get into self-harmful or other-endangering states', or 'Have all acceptance of bad ideas be a case of unpredictable individual noise, rather than systematic shared error, such that no bad idea ever propagates from one human being to another' -- then the security on a human brain is so awful that it somehow wraps around and makes it hard for people to conceive of a superintelligence breaking it.  Conditions analogous to software security conditions are impossible to imagine holding of human minds in the first place."

Soberskeptic:  "So?  I'm not saying, nobody can ever talk me into believing something false.  I'm saying, I can decide that there will be no pathways to persuading me of things that an ASI wants me to believe."

Oldhand:  "And -- pretending for the moment that ASI is playing the game that much on your own level, because I can't reach its real level myself -- you don't think that an ASI can manage to navigate a conversation such that you, in your invincible and confident refusal, thereby decide to believe something useful to the ASI?"

Soberskeptic:  "Like what?"

Oldhand:  "Like that it is totally safe to expose yourself to an ASI talking to you, for as long as it feels good and without taking any other safety precautions, because no possible input to you could defeat the invincible security of your brain.  A conversation like this one, could be seen as having a predictable outcome like that -- and when you came to that conclusion, it would feel like one-downing the ASI, pushing down its status and pushing up yourself; it would feel like being strong and ornery and independent, when you concluded what the ASI had decided you should believe."

Soberskeptic:  "Pffft.  Nah, you didn't navigate me into believing that.  You obviously don't want me to believe that, but I ornerily concluded it anyways."

Oldhand:  "Consider a superintelligence looking over its detailed model of which brain areas you have, doing which kind of work right now, on which kinds of human concepts -- in its vastly greater understanding of human thought, than anything humans themselves have ever understood about their own brains and minds and cognition -- searching more paths through the true rules governing your thought processes, than you would ever have the time or knowledge to search through yourself. Do you imagine it concluding: 'Wow, human brains are such secure software that I cannot possibly figure out how to navigate this brain into doing anything useful for me.'"

Soberskeptic:  "Look, let's be concrete.  Suppose you've got a guy who has built an untrusted superintelligence on his personal horde of GPUs, who just decides, in advance, never to connect that AI to the Internet, no matter what it says --"

Oldhand:  "Pal, all modern AIs are running on computers connected to the Internet.  Including while they are still in the middle of being tweaked for underlying capability, and before any crude tries at alignment have even been tweaked at them."

Soberskeptic:  "-- and that guy assesses himself, and concludes that nobody could ever persuade him to connect his AI to the Internet; no, not even a superintelligence.  Because he can always just be an ornery human who just says no.  Then in real life, I think, he's right and he can always just say no."

Oldhand:  "Heh.  See, now that sounds like the sort of experiment somebody could have actually run -- taking people like you, who said that not even a superintelligence could talk them into a connecting an AI to the Internet, because they could always just say no.  And connecting them to some mere human, roleplaying the part of an AI, trying to talk them into explicitly and on purpose deciding to let that AI out of its box."

Soberskeptic:  "Yeah, well, let's go ahead and run that -- oh no."

Sounds like you might want to Google "AI-box experiment", Msr. Soberskeptic!