This is a special post for quick takes by Alexander Heckett. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I want to see more work on physical security for everyday folks during a singularity. For example, imagine a tick-sized robot that could crawl up someone's leg, burrow under their skin, and poison them or do other nasty damage. This feels physically possible and hard to defend against. If such robots become cheap, how would society cope? Would we rely on physical monitoring of peoples' clothing? Would we try to determine who sent the robot after it does its damage? I think we can already start making guesses.
I don't see how this would be tractable? Unless you had an equally powerful, aligned AI which was consistently producing defenses.
First, primitive defenses can help counter advanced offenses. Counter-drone warfare in Ukraine provides many examples: netting hung over roads, "turtle tanks" with metal sheds welded on top, inflatable decoys. These simple defenses don't suffice, you also need interceptor drones and counter-drone teams, but they're important. I expect miniaturized robots would be similar: you'd probably want to combine advanced defenses like bloodstream monitoring for bioweapons or ultrasonic echolocation of airborne threats with stupid defenses like thicker doors or regularly washed floors, and predicting the stupid defenses is something I reckon we can start now.
Second, even if there's a threat for which no stupid defenses exist, I think it's better to know about it. For example, I also worry about ant-sized robots building nuclear bombs underneath targets; I don't see any stupid defenses against that. But we can still brace ourselves: we might recognize that cities will need to install underground metal detectors, maybe someone publishes UraniumDetectionDeviceBench, etcetera. I'd rather face known unknowns than unknown unknowns.
1Strong magnets in/around your house to pull in the kill-ticks? EMP? This assumes the kill-ticks are made of metal, and are not biological in nature, which is a guess.
For bio-kill-ticks, maybe some kind of static electricity, such that every time you touch anything you feel a zap. Should be essentially harmless to humans, but might kill a bio-kill-tick?
For actual kill-microbes, I think we're just doomed. Look at how bad a job we do defending against natural bacteria and viruses. Intelligent murder-bacteria would be the end of our species.
In general I find it hard to imagine a future where people have to wear fishing waders to ward off robot-tick bites, where humanity has not simply been obliterated. This scenario seems to fall into the same very narrow band as most sci-fi movies, where there could be things like humans fighting robots with assault rifles. Probably we either have an aligned superintelligence on our side helping us, or we're cooked.
I completely agree that, without a country of aligned geniuses in a datacenter, we're dead. But I still think we should try to predict what dangers we'll face. I'd classify murder-bacteria as a known threat; for example, Anthropic talks about bioweapon capabilities. What do unknown threats look like?
Something I should have realized about AI Safety via Debate ages ago but only recently recognized: I usually see theoretical studies of debate through the lens of determining the output of a fixed Turing machine (or stochastic Turing machine or some other computational model), e.g. https://arxiv.org/abs/2311.14125. This formalization doesn't capture the kinds of open-ended questions I usually think about. For example, suppose I'm looking at the blueprint of a submarine and I want to know whether it will be watertight at 1km depth. Suppose I have a physics simulation engine at my disposal that I trust. Maybe I could run the physics simulation engine at 1 nanometer resolution and get an answer I trust after ten thousand years, but I don't have time for that. This is such an extremely long computation that I don't think any AI debater would have time for it either. Instead, if I were tasked with solving this problem alone I would attempt to find some discretization parameter that is sufficiently small for me to trust the conclusion but sufficiently big for the computation to be feasible.
Now, if I had access to two debaters, how would I proceed? I would ask them both for thresholds beyond which their desired conclusion holds. For example, maybe I get the "it's watertight" debater to commit that any simulation at a resolution below 1cm will conclude that the design is watertight and I get the "it's not watertight" debater to commit that any simulation at a resolution below 1mm will conclude that the design is not watertight. I then run the simulation (or use one of the conventional debate protocols to simulate running the simulation) at the more extreme of the two suggestions, in this case 1mm. There are details to be resolved with the incentives but I believe it's possible.
I like to view this generalized problem as an infinite computation where we believe the result converges to the correct answer at some unknown rate. For example, we can run our simulation at 1m resolution, 10cm resolution, 1cm resolution, 1mm resolution, etc., saving the result from the most recent simulation as our current best guess. If we trust our simulation software then we should believe that this infinite computation eventually converges to the true answer. We can implement debate in this setup by having an initial debate over what steps T have the property that the answer at all steps T' >= T agrees with the answer at T, pick one such T that both sides agree with (take the maximum), then run a conventional debate protocol on the resulting finite computation.
EDIT: I wonder if there's a generalization of this idea to having a directed set of world models where one world model is at least as good as another if it is at least as precise in every model detail. Each debater proposes a world model, the protocol takes the maximum of the world models which exists by the directed set property, and we simulate that world model. I'm thinking of the Guaranteed Safe AI agenda here.
As a long-time advocate of multipolar worlds, I've started to feel that unipolar worlds are better in the long term. I've traditionally thought:
- Multipolar worlds are better for control. For example, I think an instance of Fable would be more likely to flag misaligned swarm behavior from OpenAI models than an instance of that OpenAI swarm.
- Unipolar worlds might avoid race dynamics between labs, but race dynamics don't fundamentally bother me. I care less about timelines than I do about policies and research allocation / taste during takeoff.
This argument for multipolar worlds assumes that humans are in charge and are trying to corral wayward AI's. However, if we extrapolate onwards to super-intelligence, I think the story flips:
- Asking one super-intelligence to give Earth enough sunlight for us to live is a reasonable request. Yudkowsky might argue that a super-intelligence wouldn't grant this request, but I'm hopeful.
- Asking a swarm of individualistic, competing super-intelligences to give Earth enough sunlight for us to live might be an unstable equilibrium, in the same way that asking hundreds of countries to abide by climate regulation is unstable.
I wish we could live in a multipolar world during takeoff and then switch to a unipolar one afterwards.
Claim: political ideologies are low-temperature magnetization.
I think factor graphs are a reasonable toy model of how arguments are coupled to each other (where this coupling is either from the perspective of a person or an LLM). This framework is very analogous to an Ising model: statement true/false = spin up/down. If statements/atoms are weakly coupled / temperature is high, the truth/spin of a statement/atom is largely determined by its local neighborhood (i.e. it's independent of everything far away). On the other hand, if statements/atoms are strongly coupled / temperature is low, the truth/spin of a statement/atom is largely determined by a globally-consistent-perspective-on-what's-true/global-choice-of-magnetization.
I expect AI-Safety-via-Debate to be possible in the local regime but much sketchier in the global regime.
Has anyone made a good, easy to use user interface for implementing debate / control protocols in deployment? For example, maybe I want to get o12345 to write some code in my codebase but I don't trust it. I'd like to automatically query Sonnet 17.5 with the code o12345 returned and ask if anything fishy is going on. If so, spin up one of the debate protocols and give me the transcript at the end. Surely the labs are experimenting with this kind of thing internally / during training but it still feels useful during deployment as an end-user, especially between models from different labs which might have internalized different objective functions and hence be less likely to collude.
For anyone wanting to harness AI models to create formally verified proofs for theoretical alignment, it looks like last call for formalizing question statements.
Game theory is almost completely absent from mathlib4. I found some scattered attempts at formalization in Lean online but nothing comprehensive. This feels like a massive oversight to me -- if o12345 were released tomorrow with perfect reasoning ability then I'd have no framework with which to check its proofs of any of my current research questions.
Debate in control settings seems more robust to collusion if you use different models, especially models from different providers. However, modern AI labs subsidize personal coding plans compared to API pricing, so it's more expensive to run multi-lab debate than single-lab debate. This seems bad to me.