Shannon’s information theory tells us about how much information is sent from a sender to a receiver via a channel. But the implications of that information might be very different depending on how much the sender trusts the receiver, in a way which seems like it might point to some interesting theoretical concepts.

Imagine you’ve just received a message M. There are two possibilities for who it came from: your best friend, or your worst enemy. The message might contain an equal amount of information either way—however, you’ll react to it very differently. In the former case, you want to propagate the contents of the message directly into your world-model: whatever your friend says, you fully believe.

By contrast, you definitely don’t want to just believe what your enemy tells you. Nor should you actively disbelieve it, though, otherwise they could still predictably mislead you. Instead, the main update you want to make is just “my enemy wants me to see the particular string of characters that constitutes message M”.

Another way of putting this: in the former case you update your world-model by conditioning on the semantics of the message, whereas in the latter case you update your world-model by conditioning on the syntax of the message. Why? Because you don’t trust your enemy enough for communication with them to support semantic meaning.

—

Now let’s push it a step further. Instead of a message from your friend or enemy, the message is either from an omnibenevolent god or an omnimalevolent devil. Here we also have a disparity in how much you trust them—but this time, both possible senders are (we’ll assume) far more intelligent than you, and could design messages which manipulate you in all sorts of ways.

So for the message from the devil, you probably don’t want to update “the devil wants me to see M”, because that still involves representing M in your mind, which could allow them to manipulate you. Instead you want to firewall yourself from their message, ideally by not even reading it at all.

The opposite is true for the message from god. In that case, even injecting the message directly into your world-model isn’t sufficient. Ideally you’d inject it straight into your policy—you’d meditate on the message, letting the words echo like poetry through your mind, until you’d internalized it on such a deep level that it was able to directly shape your instincts and impulses. After that point, you’d trust those instincts much more than by default—because they’d been rewired by someone who knows exactly what the right instincts for you to have are.

(This might be how an animal whose nervous system was finely-honed by evolution feels, by the way. Their evolved instincts are way smarter than they are, and so they have to trust them implicitly. And see also Sahil’s discussion of how cells can call for help, and get it, without any understanding of how or why that help arrived.)

I don’t know where to go with all of this but there’s something that seems very interesting here.