I’m going to assume knowledge of the basics of information theory. I won’t assume any physics knowledge (but you’ll probably find it easier if you know physics).

Thanks to Francis Priestland, Anson Ho, Nico Macé, Dan Hatton and Kenneth Freeman for their comments.

This was the result of me being nerd-sniped by a friend who wanted to talk to me about the connection between cognition and thermodynamics. I thought “yea, I know thermodynamics, it should take me a nice afternoon or two of straightening things out in my head to come up with something interesting to say”. How naive of me. In the process, I fell down a number of holes of confusion. I realised that properly understanding Landauer’s principle would be a good waypoint on my quest. This is the result of my clearing up these confusions.

Landauer’s principle is often summarised as something like:

1 bit of memory in a register is erased, destroying 1 bit of information. Because information is entropy(?!), this means the memory register loses 1 bit of entropy. Total entropy always increases over time, so the environment must increase its entropy by at least 1 bit to compensate, resulting in a corresponding minimum amount of heat dissipation, along with a minimum energy requirement for deleting the bit.

But whoa hey hold your horses there; what does entropy really mean here? Is the Shannon entropy of a logic gate really connected to its thermodynamic entropy, and if so, in what way? I always found Landauer’s principle a bit mysterious because it seems, at first glance, to connect something we tend to consider as a subjective construct (a probability distribution) with a measurable physical outcome (the release of heat). And I’m clearly not alone, there have been a bunch of people coming out and saying that the principle is widely misunderstood or even that the principle itself is wrong (but don’t panic, it’s been pretty well experimentally verified). So fleshing out what’s really going on here was helpful for me to understand more intuitively what computation (and by extension cognition maybe?) means from a physics point of view. The central insight is that, as Landauer put it; “information is physical”.

The relationship between Shannon entropy and thermodynamic entropy

Thermodynamic entropy is a pretty objective property of some physical system. It’s related t...

Derek Bronston

Derek Bronston