If you actually read my writing, I obviously understand the potential of reversible computing, and the thermodynamic efficiency limit I'm discussing is only for conventional non-exotic irreversible computers - the kind that humans know how to build now and for the foreseeable future, and also the kind of computer the brain is. Reversible computers may or may not ever be practical at room temp on earth, but nearly nobody is working on them - essentially all research into exotic computation is going into quantum computing.
Brain Efficiency claims that the energy dissipation required to erase a bit becomes many times larger when we try to erase the bit reliably. ...
This adds a factor of about 50 to the energy cost of erasing a bit, so this would be quite significant if true. To back up this claim, Jacob cites this paper by Michael P. Frank.
It is ingenuous[1] to mention only Frank for that claim, as I cite first Landauer himself[2], then Zhirnov[3], and finally Frank[4] and then Cavin[5]. Let's walk through each one by one.
Landauer analyzes three sources of error (switching time error, Swanson decay error, and Boltzmann error), and his model is much more complex than kTlog2. The most relevant single quote here is:
A third source of error consists of the fact that even if the system is allowed to relax completely during switching there would still be a fraction of the ensemble of the order: exp(−2Δ/2kT) left in the unfavored initial state. (Assuming Δ >> kT ) For purpose of the subsequent discussion let us call this the Boltzmann error.
Compare to equation 3 on page 3 of the Zhirnov link[3:1], and the bit energy in Frank[4:1]. Cavin then derives minimal viable bit energies around 1 eV[5:1].
However, in that entire section, Frank is temporarily assuming that the energy used to represent the bit internally is entirely dissipated when it comes time for the bit to be erased. Dissipating that entire energy is not required by the laws of physics, however.
It is entirely dissipated in any conventional irreversible computer, which me, Frank, Zhirnov, Cavin, etc all understand. Frank is then talking about hypothetical reversible computation in the section you are excited about.
Moving a bit from one place to another does not have a minimum cost according to thermodynamics. Erasing a bit costs energy, but merely moving a bit from one place to another doesn't count as erasing it.
Obviously not - if you are using a reversible communication path like optical signaling, but photons and photonics are too big for the bulk fine on-chip interconnect. Electron based interconnect in conventional computers is fully irreversible and dissipative, as discussed in Cavin[5:2] - as it is in brains.
Digital multiplication of two 8 bit values has a minimum cost of less than 28(0.02 eV) = 0.32 eV.
This is nonsense for a variety of reasons starting with your misunderstanding of minimal viable bit energies, and continuing through your lack of analysis of viable multiplier designs and their interconnect requirements as discussed in Cavin[5:3].
If we want to claim that future inventors (or AIs) could never build a computing device more efficient than the human brain,
Again I never claimed that - I claimed the brain is not so far from thermodynamic efficiency limits for conventional computing. Obviously Exotic (reversible/quantum) computers can eventually be more efficient.
As in an innocent, naive, and unsuspecting mistake. ↩︎
Irreversibility and heat generation in the computing process ↩︎
Limits to binary logic switch scaling-a gedanken model ↩︎ ↩︎
Since this is a top level comment, I guess it should probably have a reply by me. (Even though I'm writing this way after all the discussion has already happened, mostly under other comments and in other posts. I'm mainly just leaving it so future readers can find that stuff more easily.)
See my post here (and the subsequent discussion) on the question of whether the need for reliable erasure adds an extra factor of 50 onto Landauer's limit.
Introduction
This post is written as a response to jacob_cannel's recent post Contra Yudkowsky on AI Doom. He writes:
Most of Jacob's analysis for brain efficiency is contained in this post: Brain Efficiency: Much More than You Wanted to Know. I believe this analysis is flawed with respect to the thermodynamic energy efficiency of the brain. That's the scope of this post: I will respond to Jacob's claims about thermodynamic limits on brain energy efficiency. Other constraints are out of scope, as is a discussion of the rest of the analysis in Brain Efficiency.
The Landauer limit
Just to review quickly, the Landauer limit says that erasing 1 bit of information has an energy cost of kTlog2. This energy must be dissipated as heat into the environment. Here k is Boltzmann's constant, while T is the temperature of the environment. At room temperature, this is about 0.02 eV.
Erasing a bit is something that you have to do quite often in many types of computations, and the more bit erasures your computation needs, the more energy it costs to do that computation. (To give a general sense of how many erasures are needed to do a given amount of computation: If we add n-bit numbers a and b to get a+bmod2n, and then throw away the original values of a and b, that costs n bit erasures. I.e. the energy cost is nkTlog2.)
Extra reliability costs?
Brain Efficiency claims that the energy dissipation required to erase a bit becomes many times larger when we try to erase the bit reliably.
This adds a factor of about 50 to the energy cost of erasing a bit, so this would be quite significant if true. To back up this claim, Jacob cites this paper by Michael P. Frank. The relevant equation is pulled from section 2. However, in that entire section, Frank is temporarily assuming that the energy used to represent the bit internally is entirely dissipated when it comes time for the bit to be erased. Dissipating that entire energy is not required by the laws of physics, however. Frank himself explicitly mentions this in the paper (see section 3): The energy used to represent the bit can be partially recovered when erasing it. Only kTlog2 must actually be dissipated when erasing a bit, even if we ask for very high reliability.
(I originally became suspicious of Jacob's numbers here based on a direct calculation. Details in this comment for those interested.)
Analog signals?
Quoting Brain Efficiency:
Because of this analog representation, Jacob estimates about 6000 eV required to do the equivalent of an 8 bit multiplication. However, the laws of physics don't require us to do our floating point operations in analog. "are implemented" does not imply "have to be implemented". Digital multiplication of two 8 bit values has a minimum cost of less than 2*8*(0.02 eV) = 0.32 eV.
What if there's something special about the inherently noisy nature of analog? Maybe the random fluctuations in signal strength are actually important in the brain's proper functioning? That's fine too: Just as the Landauer limit imposes an energy cost on erasing a bit, absorbing a random bit from the environment's thermal randomness comes with an energy bonus. If we have to add randomness onto our signals, that actually improves the overall energy efficiency.
Dissipative interconnect?
Jacob claims that there are unavoidable energy costs to "interconnect", all the wiring that allows information to travel between different parts of the brain.
Moving a bit from one place to another does not have a minimum cost according to thermodynamics. Erasing a bit costs energy, but merely moving a bit from one place to another doesn't count as erasing it. This should be intuitively obvious if you think about it for a little while. Imagine a hard drive full of data drifting through space at a constant velocity. No matter how far it carries that data, there's no energy cost. Even the energy used to accelerate the drive can be recovered at the end when slowing it down.
To his credit, Jacob does acknowledge that non-dissipative interconnect is conceivable:
Fair enough for optical, but I have no idea why he's dismissing superconductive interconnect as impractical. We already have superconductors that work at liquid nitrogen temperatures, so if you're willing to make a computer design that requires cooling, you don't even need to discover room temperature superconductors.
More generally, the issue here is that we've moved away from thermodynamic limits and into practical engineering constraints. If we want to claim that future inventors (or AIs) could never build a computing device more efficient than the human brain, then an impossibility proof based on thermodynamic limits is a very powerful way to do that because they're based on fundamental physical principles that probably won't be overturned. If we instead claim that it's impossible because to have non-dissipative interconnect they'd have to use optics or superconductors and both seem impractical, then we're relying on future inventors not being able to invent a third kind of non-dissipative interconnect, and also not being able to make either of the known ones practical.
Invoking Thermodynamics
Thermodynamics has a funny way of surviving scientific revolutions that kill other theories. It's applicable to classical mechanics, special relativity, quantum mechanics, and quantum field theory.
Because of this great reputation thermodynamics has amongst people who know some physics, saying that your ideas are backed up by thermodynamics is almost like saying they've been mathematically proved to be true. People won't literally think you're correct with 100% probability, but pretty darn close.
So consider the following Motte and Bailey: The Motte is "the brain is close to the thermodynamic limits on energy efficiency". The Bailey is a bunch of arguments about engineering difficulty and how the only practical thing might be to use analog signals and dissipative interconnect. Now, even granting that those arguments are correct and the brain is the most efficient computing device that we could build: It will not be because the brain was already close to thermodynamic limits. Rather, it will be because the practical engineering limits turned out to be much tighter than the fundamental thermodynamic limits.
If you're invoking "the laws of thermodynamics", your arguments should generally look like a bunch of claims about energy and entropy and the reversibility of physical laws. They should not depend on things like how soon we're going to discover room temperature superconductors. Because of this, headlining with "the brain is close to the thermodynamic limits on energy efficiency" seems misleading at best.