Adam Shai

Another paper you might be interested in that shows reinforcement learning effects even after training has reached asymptotic performance in a perceptual task: https://elifesciences.org/articles/49834

These points are well taken. I agree re: log-score. We were trying to compare to the most straightforward/naive reward maximization setup. Like in a game where you get +1 point for correct answers and -1 point for incorrect. But I take your point that other scores lead to different (in this case better) results.

Re: cheating. Yes! That is correct. It was too much to explain in this post, but ultimately we would like to argue that we can augment Turing Machines with a heat source (ie a source of entropy) such that it can generate random bits. Under that type of setup the "random number generator" becomes much simpler/easier than having to use a deterministic algorithm. In addition, the argument will go, this augmented Turing machine is in better correspondence with natural systems that we want to understand as computing.

Which leads to your last point, which I think is very fundamental. I disagree a little bit, in a specific sense. While it is true that "randomness" comes from a specific type of "laziness," I think it's equally true that this laziness actually confers computational powers of a certain sort. For now, I'll just say that this has to do with abstraction and uncertainty, and leave the explanation of that for another post.

Wow, despite no longer being endorsed, this comment is actually *extremely *relevant to the upcoming posts! I have to admit I never went through the original paper in detail. It looks like Shannon was even more impressive than I realized! Thanks, I'll definitely have to go through this slowly.

Great question. Hopefully soon I'll write a longer post on exactly this topic, but for now you can look at this recent post, Beyond Kolmogorov and Shannon, by me and Alexander Gietelink Oldenziel that tries to explain what is lacking in Turing Machines. This intuition is also found in James Crutchfield's work, e.g. here, and in the article by him in the external resources section in this post.

In short, the extra desirability condition is a mechanism to generate random bits. I think this is fundamental to computation "in the real world" (ie brains and other natural systems) because of the central role uncertainty plays in the functioning of such systems. But a complete explanation for why that is the case will have to wait for a longer post.

Admittedly, I am overloading the word "computation" here, since there is a very well developed mathematical framework for computation in terms of Turing Machines. But I can't think of a better one at the moment, and I do think this more general sense of computation is what many biologists are pointing towards (neuroscientists in particular) when they use the word. Maybe I should call it "natural computation."

Embarrassingly, I've never actually thought of how compilers fit into this set of questions/thoughts. Very interesting, I'll definitely give it some thought now. I like the idea of a compiler as some kind of overseer/organizer of computation.

Thanks! Sounds like I need to have a better understanding of lambda calculus, and as always, category theory :)

These are both great points and are definitely going to be important parts of where the story is going! Probably we could have done a better job with explication, especially with that last point, thanks. Maybe one way to think about it is, what are the most useful ways we can convert data to distributions, and what do they tell us about the data generation process, which is what the next post will be about.

No I haven't! That sounds very interesting, I'll definitely take a look, thanks. Do you have a particular introduction to it?

Taking a t-shirt, folding it over a few times, and tying it around my head works better than any sleep mask, even the expensive ones, in my experience.

Do you have a particular story that shows the types of negative outcomes that could happen? While it's not impossible for me to imagine an overly sensitive academic getting angry or annoyed unreasonably, at a distillation, it hardly seems to me like it would be at all likely. I have fairly high confidence in my understanding of academic mindsets, and a single sentence at the top "this is a summary of XYZ's work on whatever" with a link would in almost all cases be enough. You could even add in another flattering sentence, "I'm very excited about this work because... I find it super exciting so here's my notes/attempt at understanding it more"

Generally, academics like it when people try to understand their work.