When you say "Tegmark IV," I assume you mean the computable version -- right?
Yep.
We have this sort of symmetry-breaker in the version of the argument that postulates, by fiat, a "UP-using dupe" somewhere, for some reason
Correction: on my model, the dupe is also using an approximation of the UP, not the UP itself. I. e., it doesn't need to be uncomputable. The difference between it and the con men is just the naivety of the design. It generates guesses regarding what universes it's most likely to be in (potentially using abstract reasoning), but then doesn't "filter" these universes; doesn't actually "look inside" and determine if it's a good idea to use a specific universe as a model. It doesn't consider the possibility of being manipulated through it; doesn't consider the possibility that it contains daemons.
I. e.: the real difference is that the "dupe" is using causal decision theory, not functional decision theory.
We can just notice that we'd all be better off if no one did the malign thing, and then no one will do it
I think that's plausible: that there aren't actually that many "UP-using dupes" in existence, so the con men don't actually care to stage these acausal attacks.
But: if that is the case, it's because the entities designing/becoming powerful agents considered the possibility of con men manipulating the UP, and so made sure that they're not just naively using the unfiltered (approximation of the) UP.
That is: yes, it seems likely that the equilibrium state of affairs here is "nobody is actually messing with the UP". But it's because everyone knows the UP could be messed with in this manner, so no-one is using it (nor its computationally tractable approximations).
It might also not be the case, however. Maybe there are large swathes of reality populated by powerful yet naive agents, such that whatever process constructs them (some alien evolution analogue?), it doesn't teach them good decision theory at all. So when they figure out Tegmark IV and the possibility of acausal attacks/being simulation-captured, they give in to whatever "demands" are posed them. (I. e., there might be entire "worlds of dupes", somewhere out there among the mathematically possible.)
That said, the "dupe" label actually does apply to a lot of humans, I think. I expect that a lot of people, if they ended up believing that they're in a simulation and that the simulators would do bad things to them unless they do X, would do X. The acausal con men would only care to actually do it, however, if a given person is (1) in the position where they could do something with large-scale consequences, (2) smart enough to consider the possibility of simulation-capture, (3) not smart enough to ignore blackmail.
Consider a different problem: a group of people are posed some technical or mathematical challenge. Each individual person is given a different subset of the information about the problem, and each person knows what type of information every other participant gets.
Trivial example: you're supposed to find the volume of a pyramid, you (participant 1) are given its height and the apex angles for two triangular faces, participant 2 is given the radius of the sphere on which all of the pyramid's vertices lie and all angles of the triangular faces, participant 3 is given the areas of all faces, et cetera.
Given this setup, if you're skilled at geometry, you can likely figure out which of the participants can solve the problem exactly, which can only put upper and lower bounds on the volume, and what those upper/lower bounds are for each participant. You don't need to model your competitors' mental states: all you need to do is reason about the object-level domain, plus take into account what information they have. No infinite recursion happens, because you can abstract out the particulars of how others' minds work.
This works assuming that everyone involved is perfectly skilled at geometry: that you don't need to predict what mistakes the others would make (which would depend on the messy details of their minds).
Speculatively, this would apply to deception as well. You don't necessarily need to model others' brain states directly. If they're all perfectly skilled at deception, you can predict what deceptions they'd try to use and how effective they'd be based on purely objective information: the sociopolitical landscape, their individual skills and comparative advantages, et cetera. You can "skip to the end": predict everyone playing their best-move-in-circumstances-where-everyone-else-plays-their-best-move-too.
Objectively, the distribution of comparative advantages is likely very different, so even if everyone makes their best move, some would hopelessly lose. (E. g., imagine if one of the experts is a close friend of a government official and the other is a controversial figure who'd been previously judged guilty of fraud.)
Speculatively, similar works for the MUP stuff. You don't actually need to model the individual details of other universes. You can just use abstract reasoning to figure out what kinds of universes are dense across Tegmark IV, figure out what (distributions over) entities inhabit them, figure out (distributions over) how they'd reason, and what (distributions over) simulations they'd run, and to what (distribution over the) output this process converges given the objective material constraints involved. Then take actions that skew said distribution-over-the-output in a way you want.
Again, this is speculative: I don't know that there are any math proofs that this is possible. But it seems plausible enough that something-like-this might work, and my understanding is that the MUP argument (and other kinds of acausal-trade setups) indeed uses this as a foundational assumption. (I. e., it assumes that the problem is isomorphic (in a relevant sense) to my pyramid challenge above.)
(IIRC, the Acausal Normalcy post outlines some of the relevant insights, though I think it doesn't precisely focus on the topic at hand.)
Agreed, always a good exercise to do when surprised.
Why?
It was already known the AGI Labs were experimenting with synthetic data and that OpenAI are training GPT-5, and the article is light on new details:
I mean, the state of affairs is by no means not worrying, but I don't really see what's in this article would prompt a meaningful update?
Here's my understanding of the whole thing:
This all seems to check out to me. Admittedly I didn't actually confirm this with any proponents of the argument, though.
(Probably also worth stating that I don't think the MUP is in any way relevant to real life. AI progress doesn't seem to be on the track where it features AGIs that use big dumb "where am I?" modules. E. g., if an AGI is born of anything like an RL-trained LLM, seems unlikely that its "where am I?" reasoning would be naive in the relevant sense. It'd be able to "manually" filter out universes with malign consequentialists, given good decision theory. You know, like we can.
The MUP specifically applies to highly abstract agent-foundations designs where we hand-code each piece, that currently don't seem practically tractable at all.)
... is that why this post has had unusually many downvotes?
Nah, I'd expect it's more broadly because it's making an arrogant-feeling claim that the organization-design market is inefficient and that you think you can beat it. Explicitly suggesting that a world in which the median person were you wouldn't be like this maybe put that in sharper relief, but I don't think this specific framing was a leading cause at all.
There might also be some backlash from people who'd previously interfaced with relatively more benign big organizations, who are weary of these sorts of models and view them as overly cynical/edgy.
So how on Earth (as opposed to Wentworld) are large organizations stable?
My model, prior to reading further:
Well, what do such structures maximize? By what goals are they optimal?
The answer that seems obvious to me is "the perceived power of those at the top of such organizations". The people constituting the-organization-as-a-whole are obviously in a suboptimal configuration for maximizing their collective economic efficiency, or their collective power. But the people in charge find themselves in nominal command of an enormous pile of resources. They are poorer, in an absolute sense, than they would've been if the world were in the equilibrium state in which people are arranged in more efficient economic structures. But they are relatively much more rich and influential than their underlings, which parses as optimal by their power- and status-maximizing heuristics, so it's in their individual interests to preserve this status quo.
At the cultural level, this equilibrium is maintained by such organizations instinctively cooperating with each other to preserve it (as Zvi outlines e. g. here, I think). This is probably also culturally supported by upper managers essentially wanting to feel like "the king of their kingdom", using historical kingdoms/governments as reference points. Said kingdom/governments, in turn, have historically ended up in big-dumb-organization configurations to maximize the power of those at their helm. (Except they spread their influence by more obvious conquest-and-pillaging, not by hiring people.)
Everyone outside mazes isn't well-versed enough in the subject to understand what they're signing up for when joining, or unable to imagine superior alternative configurations, or culturally indoctrinated to view such structures as inherently valuable as well. So they don't demand larger pay, aren't unwilling to work with large organizations, don't vote to dissolve them, et cetera.
… so e.g. large companies or government agencies are basically runaway human monuments of dominance and submission, limited mainly by their budget.
Yeah, that seems to agree with my model.
I'd only add that this setup increases upper managers' status outside the organizations as well, and in a fairly robust manner. It's not quite like with the rhesus monkeys: the CEO of a Fortune 500 company, if they leave their organization and seek to join a new one, wouldn't start at the very bottom at all. They'd often end up at the executive level as well, or even straight-up be re-hired as a CEO.
You know how the org structure is always hierarchical for some reason? And managers end up being massive communication-bottlenecks across the org, because horizontal coordination usually has to route through them? That’s not a bug, that’s a feature of an organization whose managers optimize first for dominance-status, and only secondarily for making money.
Hm, I don't know if that's necessarily a bug from the perspective of efficient organization design either. Inasmuch as organizations are control systems reflecting the domain they control, the hierarchy reflects the hierarchy of natural abstractions, with the managers/subsystems at higher levels "acting as natural latents", inducing conditional independence between lower-level subsystems. This has productivity-boosting results as well: see e. g. Joel's "development abstraction layer" argument (the people within the subsystem can focus their optimization-power solely on the domain they're responsible for, with no distractions).
Obviously this is horribly mismanaged in practice, but it feels like a warped feature of a fundamentally productive structure.
I'd expect that if the natural-abstractions theory gets to the point where it's theoretically applicable to fluid dynamics, then demonstrating said applicability would just be a matter of devoting some amount of raw compute to the task; it wouldn't be bottlenecked on human cognitive resources. You'd be able to do things like setting up a large-scale fluid simulation, pointing the pragmascope at it, and seeing it derive natural abstractions that match the abstractions human scientists and engineers derived for modeling fluids. And in the case of fluids specifically, I expect you wouldn't need that much compute.
(Pure mathematical domains might end up a different matter. Roughly speaking, because of the vast gulf of computational complexity between solving some problems approximately (BPP) vs. exactly. "Deriving approximately-correct abstractions for fluids" maps to the former, "deriving exact mathematical abstractions" to the latter.)
Let's see if I get this right...
Is this about right? I'm getting a vague sense of some disconnect between this formulation and the OP...
Fair point! I agree.