I must admit I am a little pessimistic, and consider the anarchy side of the equation much more likely than consolidation. Modern society as a bunch of protocols is effective at managing diversity at scale, but it was built in a different age. You could only diverge so far from your neighbours, because who else did you talk to? With the internet and now AI, the production process of shared meaning is shifting in a fundamental way, and I'm not sure where it leads.

That’s an interesting question! I think it’s instructive to consider the information environment of a medieval peasant. Let’s speculate that they interact with a max ~150 people, almost all of which are other peasants (and illiterate). Everyone lives in the same place and undertakes a similar set of tasks throughout their life. What shared meanings are generated by this society? Surely it is a really intense filter bubble, with a thick culture into which little outside influence can penetrate? The Dothraki from Game of Thrones have a great line – ‘It is known’ – which they use to affirm something widely understood. This is a ritual rather than an explanation, because everyone within the culture already knows the meaning of whatever it is that has been said.

Comparing this to modern society, one of the biggest differences is the firehose of information we constantly consume, which is unique for each person. More than this, because it is global, the same information gets consumed in different contexts by different people (e.g. your reference to Gangnam Style). I think this leads to a kind of a hierarchical structure. On the one hand, we also have filter bubbles, which are getting more bespoke and personalised every year, and can lead to neighbours in possession of wildly different world models. On the other, unlike the peasants, we have to regularly coordinate and interact with people outside our bubbles. So you get this high-level set of Schelling points people coalesce around, like fashions and conventions, that allow us to have meaningful interactions with people we don’t know very well, but often only to a shallow depth. This sometimes looks like a drab monoculture. Then there is a lot in-between: little pockets of thicker shared meaning generated by families, sub-cultures, organisations etc, the coherence of which constantly have to be reaffirmed to survive the information firehose.

In sum, I think this cashes out as lots of thin shared meanings underpinning massive diversity, albeit in a highly complex structure. Although, I should caveat this with the fact that I would probably struggle to recognise our more deeply held shared meanings, in the whole fish recognising water sense.

Adding this having read your post. I think I agree that it is impossible to speak of a sematically invalid statement, but that is because I am locating the meaning outside the statement, rather than applying validity or invalidity to the statement itself. 'Colourless green ideas sleep furiously' certainly means something to me, and not just as a first order sign about Chomsky. The example of this, par excellence, is Lewis Carroll's Jabberwocky. There's a great section in Goedel, Escher, Bach about translating the Jabberwocky into different languages -- even though it is a nonsense poem, it clearly means something when you read it.

It is interesting to think about why a statement might be found more or less meaningful by people in general. I asked ChatGPT for a summary of Eco's Theory of Semiotics, and it included this passage:

'Eco emphasises that meaning is not inherent in signs but is produced through codes—systems of rules shared within a culture. Interpretation depends on the interpreter’s familiarity with the relevant codes.'

I think this is important. My post doesn't engage with why there are common patterns in words. As I mention in my other comment, I see this as a cultural, emergent thing that I was treating as out-of-scope, concentrating instead on how individual communication works.

Interestingly, my first draft was closer to your point of view, but I talked myself out of it! I convinced myself the kind of information that is in the words/context is a different type of thing than meaning. I haven’t had time to read your link yet, but will do later.

Here are my quick thoughts:

1. As mentioned, words/context are information, but not meaning. A message written in code is meaningless without knowledge of the code, and in this case everyone has a slightly different decoding method. If my method decodes a message to mean the opposite of yours, that implies the meaning is in the method. This is not a hypothetical – people interpret statements in diametrically opposed ways all the time.

2. Subjectivity is a matter of degree. Because we live in the same world and interact all the time, we will share a lot of meanings, and this will make them look somewhat objective. We can, in this sense, speak of an emergent, global meaning for words (‘words are defined by their use’), but we interact with this indirectly, through other individuals, and I think it is also a different kind of thing to the meaning I am talking about in this post.

Do you have any quick examples of value-shaped interpretations that conflict?

This gives us some wiggle room to imagine different ways of resolving those conflicts, some of which will look more like instruction-following and other that will look more like autonomous action.

So perhaps the level of initiative the AI takes? E.g. a maximally initiative-taking AI might respond to 'fetch me coffee' by reshaping the Principal's life so they get better sleep and no longer want the coffee.

I think my original reference to 'perfect' value understanding is maybe obscuring these tradeoffs (maybe unhelpfully), as in theory that includes knowledge of how the Principal would want interpretative conflicts managed.

One problem I have with the instruction-following frame is that it feels like an attempt to sidestep the difficulties of aligning to a set of values. But I don’t think this works, as perfect instruction-following may actually be equivalent to aligning to the Principal’s values.

What we want from an instruction-following system is one that does what we mean rather than simply what we say. So, rather than ‘Do what the Principal says’, the alignment target is really ‘Do what the Principal’s values imply they mean’. And if an AI understands these values perfectly and is properly motivated to act according to them, that is functionally the same as it having those values itself.

If done correctly, this would solve the corrigibility problem, as all instructions would carry an implicit ‘I mean for you to stop if asked’ clause.

Would it make sense to think of this on a continuum, where on one end you have basic, relatively naive instruction-following that is easy to implement (e.g. helpful LLMs) and on the other you have perfect instruction following that is completely aligned to the Principal’s values?

Thank you for your kind words! I’m glad you liked it. Your instruction-following post is a good fit for one of my examples, so I will edit in a link to it.

I agree that alignment is a somewhat awkwardly-used term. I think the original definition relies on AI having quite cleanly defined goals in a way that is probably unrealistic for sufficiently complex systems, and certainly doesn’t apply to LLMs. As a result, it often ends up being approximated to mean something more like directing a set of behavioural tendencies, like trying to teach the AI to always take the appropriate action in any given context. I tend to lean into this latter interpretation.

I haven’t had time to read your other links yet but will take a look!

I'm glad to see someone talking about pragmatism!

I find it interesting that the goal of a lot of alignment work seems to be to align AI with human values, when humans with human values spend so much of their time in (often lethal) conflict. I'm more inclined to the idea of building AI with a value-set that is complementary to human values in some widely-desirable way, rather than literally having a bunch of AIs that behave like humans.

I wonder if this perspective intersects with some of your points about thick and thin moralities, as well as social technology. Am I in the right ballpark to suggest that what you are after is a global thin morality defined via social technology that allows AI to participate in diverse, thick human cultures without escalating conflict between groups and contexts to above an unacceptable level?

In a sense, at the risk of oversimplifying, you are looking for a pragmatic solution for keeping conflict low in a world of diverse, highly powerful AI?

I see the LLM side of this as a first step, both as a proof of concept and because agents get built on top of LLMs (for the forseeable future at least).

I think that, no, it isn't any easier to align an agent's environment as to align the agent itself. I think for perfect alignment, that will last in all cases and for all time, they amount to the same thing, and this is why the problem is so hard. When an agent or any AI learns new capbilities, it draws the information it needs out of the environment. It's trying to answer the question: "Given the information coming into me from the world, how do I get the right answer?" So the environment's structure basically determines what the agent ends up being.

So the key question is the one you say, and that I try to allude to by talking about an aligned ontology: is there a particular compression, a particular map of the territory, which is good enough to initialise acceptable long-term outcomes?