This is a special post for quick takes by Mira Kennard. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I'm trying to think through an idea. These words are simply process of solidifying it in my head.
This is based around Lotkas Principle, the use of natural selection as a fourth law of thermodynamics, and Odum's Ecological Analog of Ohm's Law.
It seems to me there exists a mechanism of natural selection for social structures in social systems. I posit this is based on energy/information as in biological and physical systems.
The general idea:
Social structures which are more capacity to utilize energy/information efficiently win out.
I think a good example is that of the theory of the linear progression of history from hunter gatherer through slaves and feudalism to where we are now with liberal capitalism. I have a lot of problems with this theory but it illustrates my point well. At each stage the energy/information efficiency of society increased, the social structures that supported this replacing those that didn't.
The problem:
This depends on two assumptions. First is that social structures inherently compete for some constrained quantity, and that energy/information utilization helps gain more access to this quantity. Biological systems are constrained by the energy emission of the sun and radioactive decay under the Earth's crust, but they are also constrained by physical space, breeding partners, etc.
You could base this all on the innate need for energy inherent to the people who make up the social system but this feels like we return back to the original Lotka's principle. It's an argument from "well technically everything is physics", which is one I don't appreciate. There's nothing novel in this basis.
There are strict constrained quantities that must be divided among social structures at any given time. For instance people can only be a part of a finite number of social groups, and thus social structures must compete for human attention. There are also a finite number of people at any given time, which provides us another constraint.
My starting point:
There is a feedback loop with relationships, success breeds success and you talk to people you like more often than you don't. If we interpret human relationships stochastically we can get fuzzy social groups and can avoid doing math on hypergraphs. It also gives us a good probabilistic basis of interpretation so measuring things like entropy is a lot simpler than taking Von-Neumann entropy on a hypergraphs incidence matrix.
If we can solidfy this with simulations we can move onto measuring the actual amount of information being exchanged in these simulations by assuming each person communicates a fixed amount each conversation.
So lets get an agent based model going, something like:
Non-conservative Kinetic Relationship Exchange
Two people contain opinions
Then let the probability of a person be proportional to the friendship relative to the mass of opinion the person contains.
Wait -> this will cause close friends to also have a higher chance of destroying the friendship depending on
I'll start coding. I'll do in manually for the nostalgia.
Halting AI development is a technophysical and sociophysical impossibility. The very nature of software and the race to superintelligence as a mathematical game excludes efficient implementation of any moratorium on AI research in the name of safety. LLMs are now an open source socio-technological contagion.
We can't stop not because we don't want to, but because the physics of the way our society and AI technology interface wont allow it. Only by leaning into AI research which promotes interpretability can we solve the problem of AI control.
This is plausible in some sense. But it's also a self-reinforcing meme (that might end up destroying the world), so discussing it requires some care. In principle, belief that you'll do X shouldn't compel you to do X, for things you don't endorse. But this isn't well-understood, let alone well-known.
(I suspect this would land better as a hypothesis, or as a conclusion under some assumptions, rather than a categorical claim.)
If it was the case that stopping AI development was impossible, then it would be good to know that it's impossible so we can focus on other solutions. I think the problem with this argument isn't that it's a true antimeme, but that stopping AI development isn't impossible.
Okay I'll layout the theory and demonstrate my conclusion.
Assume we have two parties who are developing AI, Company A and Company B. They both can either develop AGI or not with probabilities
Let
Then for A,
Notice that the more likely AGI appears
So the only thing that can stop this mechanism is an actual technological or physical barrier to development. A moratorium doesn't break the theory, it just forces development underground.
Please correct me if I made a mistake.
[EDIT]
Fix to the math to account for the risk of getting caught breaking the 'No AGI' law. Thank you Brendan Long!
Let
Because of this assumption that "success in AGI development
If I'm following your math correctly, it seems like this is a fully-general argument that it's impossible to prevent any action with non-zero reward and non-zero cost of failure. I'm not really a math person, but it seems like something must be wrong with this argument because people fail to do things with non-zero reward and non-zero cost of failure all the time.
It also seems suspicious that your equation has no term for the cost of getting caught breaking the hypothetical anti-AI law/international norm.
I appreciate your feedback and I'll change the math to account for the cost of getting caught, since I think that's a significant oversight on my part. A few subtle distinctions I want to point out.
First: I said corporations/companies, not people. Sorry if I didn't clarify this. The reason behind the corporate framing instead of the personal framing is to motivate the reward to money relationship, and make the quantifiable argument more sound.
Second: AGI works differently. We're concerned not with a rate of success but with success categorically, i.e. 'who gets there first?'. This changes how we should look at prevention. With regular crime we focus on reducing the rate when we draft laws, but with AGI we focus on absolutely preventing the action from occurring. Thus, it must be not only upheld legally, but intrinsicly. It must be such that no company will ever participate in such a project, as it fundamentally goes against basic properties of corporate reward functions (sign, relative magnitude, dynamics, etc).
An rant against the kind of people I find particularly aggravating:
We have a lot of data on policy outcomes. Furthermore social systems can be simulated to give approximations of policy outcomes. If you hold an opinion on how society ought to function and can't back it up with at least a vibe-coded Monte Carlo simulation or some napkin-based game theory, then your policy opinion is founded on anecdotes and feelings rather than facts.
We live in the age of AI i.e. a talking library with the sum total of all of human knowledge. If you can't validate your opinions then you're not putting enough effort into developing them. If you don't change at least one policy opinion you have every week then you either have converged on some fundamental truth of reality or you're stuck in a political belief system you haven't fully thought through.