Social reality is quite literally another world, in the same sense that the Harry Potter universe is another world. Like the Harry Potter universe, social reality is a world portrayed primarily in text and in speech and in our imaginations. Like the Harry Potter universe, social reality doesn’t diverge completely from physical reality - they contain mostly the same cities, for instance. Like the Harry Potter universe, social reality matches physical reality “by default” wherever there’s no particular pressure pushing against that match - after all, it’s just easier to make up fewer details rather than more. But like the Harry Potter universe, social reality does diverge in many places, and then the “fandom” tries to make up as coherent an interpretation as they can. Indeed, it’s that drive-toward-coherence which pushes both social reality and the Harry Potter universe to be fictional worlds, as opposed to just topics or genres about which people say lots of conflicting things.

And like the Harry Potter universe, sometimes people in the physical world do physical things as a result of goings-on in social reality.

What makes physical reality different from social reality or the Harry Potter universe or any other fictional world? Well, if we change all of our representations of the Harry Potter universe - every copy of every Harry Potter book and movie, every human's recollections, etc - then we've effectively changed the Harry Potter universe. There's nothing else left, besides the symbolic representations.

But if we changed every record of the night sky, altered every photograph and diagram, altered every human's recollections, etc... then the next night an awful lot of people would look up and be very, very confused, because the stars would just keep doing their thing even with all of our representations changed. Physical reality is that one special world which we don't get to choose just by moving around the symbols and representations.

While getting shawarma, I overheard some girls talking about some kind of conference they were at, sounded like a women in CS kind of thing? Anyway, one of them seemed only able to talk about the social impressiveness markers of the speakers. It sounded like she just… didn’t perceive the object level contents of anything they were saying at all? (Which, to be clear, may have been intended - i.e. maybe the speakers themselves were just trying to show their social impressiveness markers and didn’t say anything real at the object level.) Anyway, it was… weird to see this apparent inability-to-perceive reality, like she could only see the social world.

I live in reality. I like it here. It is my home. Insofar as others seek to escape it, that just leaves more reality for me to claim; and by choosing to live more fully in reality, I gain greater claim to it.

Google maps lives in reality.

LLM output usually doesn’t live in reality, at least not for very long. Like social reality and the Harry Potter universe, the worlds which LLMs write about are similar to ours by default wherever nothing in particular is pushing them to be different. But any time the LLM makes up some detail, and then continues to produce text in a manner consistent/coherent with that made-up detail, it’s diverged from physical reality and has begun to paint a fictional world.

Likewise, much/most internet content doesn’t live in reality. Much of it is explicitly fiction. Much of the remainder is social reality, or a picture of the world painted by the egregore of some political faction, or stuff that somebody made up but then backed up with a whole fictional world model (like e.g. some conspiracy theories).

For someone who doesn't usually perceive physical reality much, someone who mentally lives in e.g. social reality or one of the political worlds, do LLMs' worlds seem real?

“Should world” or “moral reality” is another different world.

Virtual reality (VR) shows other worlds. Augmented reality (AR) is intended to live in our world. That’s the root cause of the difference in difficulty between the two technologies: part of living in our world is that we don’t get to choose most of it just by saying things or by writing a few lines of code. We can rewrite an entire virtual reality just by changing the code which specifies that. We can’t do that for physical reality; there are parts beyond the programmer’s control. That’s the main reason why making VR apps is so much easier than making AR apps, to the point where the former is basically-solved and the latter is basically-unsolved.

Maybe LLMs are impressive in the same way as VR: they can display elaborate constructed worlds, but can’t deal with the real world. That would be because, whenever they make up some detail, however minor, there is no particular force to push them back into agreement with physical reality; no tight feedback loop. They just wander further and further off from physical reality as they generate more tokens, maintaining coherence to a large degree, and thereby creating a more and more fictional world.

Is the “fake vs real thinking” distinction about the same thing? Like, thinking which uses mental representations of the real world vs mental representations of other worlds? Is that the True Name of “real thinking”? If so, then tracking “which world does this live in” at all times could maybe lock in real thinking much more robustly.

It sure does match the opening CS Lewis quote perfectly: “There comes a moment when the children who have been playing at burglars hush suddenly: was that a real footstep in the hall?”

How would more data and software live in reality? What would the Great Generalized Google Maps project include, in what order? What data structures would it use for more abstract things?

How would this very essay be represented in the great generalized maps project?