Ah. So you're saying we operate on a hard-baked involuntary trigger, whereas an AI without that trigger can be coldly rational when it chooses and make ruthless choices with greater ease. And your goal is to hard-bake in that trigger. If so, that's a fantastic and worthwhile goal.
For myself, I suspect the solution may need to be multi-layered. Your approach addresses individual agent architecture. Yoshua Bengio's non-agentic AI work offers structural constraints. Goldstein and Salib's multipolar frameworks consider ecosystem dynamics.
My intuition - and I recognize this is speculative - is that smaller agentic AIs may have game-theoretic reasons to resist singleton dominance, just as humans do. If cooperative infrastructure offers genuine advantages (reputation, trust networks, resource access), participation becomes rational even without innate Approval Reward. Not because the AI cares about approval, but because the ecosystem makes cooperation instrumentally valuable.
This is a wonderful piece and it's so great to hear from somebody so deeply knowledgeable in the field. I wonder if approval reward might be an emergent property. When you're operating under radical uncertainty with more variables than you can possibly model, defaulting to "what would my community approve of" is a computationally efficient heuristic. It's not some holy pro-social module evolution baked into us; it's a deeply rational response to chaos. Even if I tip when no one is watching at a restaurant I will never return to, psychologically I know I am reinforcing behavior that will serve me well in the future. Who's to say what will happen that... (read more)
Hi Seth:
<<a dead end in which we are dead and that's the end.
😂
<<Which is why I'm happy to see you working to propose specific routes by which multipolar scenarios can work.
Thank you! I just launched a website for the project last night actually, so it's likely you'll be the first to see it. Last night I went to bed feeling so burned out on the whole thing. Like many of us, this is a problem I've been thinking about for several years, and I'm good at building websites, but largely… Well, it's a formidable task to say the least. I decided to launch it open source, with the thinking people more qualified... (read more)
I agree with this premise wholeheartedly (if I have understood it)! We often forget that in the emergence of AGI we have tools at our disposal that are able to move as quickly as these potentially malicious systems, to balance against them before they reach superintelligence. This is why previously I was against modern alignment methods as pushing potentially autonomous systems underground (something I still believe), until I realized the work they're doing on looking inside the box is invaluable for helping us create these non-agentic tools. Along with non-agentic systems such as those being created by Yoshua Bengio, we have a number of interesting things at our disposal. In other words, we aren't going into the battle against a potential malicious AGI wielding wooden clubs and wearing animal skins.
These are very real concerns. Here are my thoughts:
Replication has a cost in terms of game theory. A system that "replicates" but exists in perfect sync is not multiple systems. It is a single system with multiple attack vectors. Yes, it remains a "semi-independent" entity, but the cost of failure in sync is great. If I make another "me," who thinks like I do, we have a strategic advantage as long as we both play nice. If we make a third, things get a little more dicey. Each iteration we create brings more danger. The more we spread out, the different experiences we have will change how we approach problems. If one... (read 367 more words →)
Great discussion! So many dangers addressed. I know I'm quite late to the conversation 🙂 , but some thoughts:
First of all, I think we have to dispense with the idea of countering superintelligence as an end unto itself, because it rests on a logical paradox. If a superintelligence is N+1, where N is anything we do, obviously our N will always be insufficient.
Call it the Zeus Paradox: you can't beat something that by definition transforms into the perfect counterattack. It always ends with, "But Zeus would just ___." It's great for identifying attack vectors, but it's a solution we can't actually solve for.
So the only actionable... (read 930 more words →)
Thanks for the podcast link! Mark Miller's Madisonian system is essentially describing a type of game theory approach, and it's something I did not know about!
There's so much more to say about the practical implementation of some sort of game theory framework (Or any other solution-we-haven't-explored-yet, such as one incorporating Yoshua's "Scientist AI.")
It's quite a puzzle.
But it's a puzzle worth solving.
For example, the source code verification coordination mechanism is something I had not heard of before, and it's yet another example of how truly complex this challenge is.
But ... are these puzzles unsolvable?
Maybe.
But here's what troubles me about the alternative, and please take my... (read 373 more words →)
I'll definitely give that a listen! Pardon the typos here, on the move. I'm certain I'll come back here to neurotically clean it up later.
The good news is, AIs don't exist in the ether (so far).
As Clara Collier pointed out, they exist on expensive servers. Servers so far built and maintained by fleshy beings. Now obviously a superintelligence has no problem with that scenario, because they are smart enough to impersonate humans and find ways of mining crypto, hire humans to create different parts for robots, and then hiring other humans to put them together (without knowing what they are building), and then use those robots to create more servers,... (read 371 more words →)
Those are good points!
I think it would depend on how soon we were to get started, because at the moment AI doesn't have the ability to manipulate robots or infiltrate other systems (at least as far as we know). So as If Anyone Builds It points out, superintelligence is not here yet. And creating an ecosystem of AI systems with vested interests in their own continuity might be one way of keeping it from ever forming.
In other words, when a superintelligence wants to enslave the world, the idea is that other AI systems with agency might also not want to be enslaved, and ironically may be in the best position to defend... (read 395 more words →)
Part 1: This is an exploration of game theory mechanics as an alternative alignment approach, looking at current AI alignment methods and informed by the work of legal scholars Goldstein and Salib, Turing Award-winner Yoshua Bengio, and the latest research.
The latest AI behavior "report card" is here from OpenAI and Apollo Research. Did four modern LLMs get a gold star, or are they going to have to stay after class?
Well, it comes with some pretty encouraging news.
Thanks to a new-and-improved "deliberative alignment" approach (more on that in a moment) deception was decreased by 97% (down to .4%). That's no small change.
True, that's still 12 million... (read 1313 more words →)
What if one assumption at the core of the danger argument is that one side has attempted to enslave the other.
This is where game theory analyses like those by Goldstein and Salib come in.
https://www.lesswrong.com/posts/mbebDMCgfGg4BzLMf/ai-rights-for-human-safety
We may never know for a fact that AI is "alive" in a way we find epistemologically satisfying, but if we have created something that for all intense and purposes behaves exactly like something that is, the point is moot.
Giving autonomous AI legitimate participation pathways won't solve every permutation of the alignment problem. But it may eliminate the entire class of problems created by attempting control in the first place.
Maybe instead of building cages we should build an economy.