rhaps0dy

Predictive Coding has been Unified with Backpropagation

Not particularly relevant, I think, but interesting nonetheless.

A first drawback of this paper is that its conclusion assumes that the NN underneath trains with gradient *flow (GF)*, which is the continuous-time version of gradient descent (GD). This is a good assumption if the learning rate is very small, and the resulting GD dynamics closely track the GF differential equation.

This does not seem to be the case in practice. Larger initial learning rates help get better performance (https://arxiv.org/abs/1907.04595), so people use them in practice. If what people use in practice was well-approximated by GF, then smaller learning rates would give the same result. You can use another differential equation that does seem to approximate GD fairly well (http://ai.stanford.edu/blog/neural-mechanics/), but I don't know if the math from the paper still works out.

Second, as the paper points out, the kernel machine learned by GD is a bit strange in that the coefficients $a_i$ for weighing different $K(x, x_i)$ depend on $x$. Thus, the resulting output function is not in the reproducing kernel Hilbert space of the kernel that is purported to describe the NN. As a result, as kernel machines go, it's pretty weird. I expect that a lot of the analysis about the output of the learning process (learning theory etc) assumes that the $a_i$ do not depend on the test input $x$.

Strong Evidence is Common

I’m guessing they would have happily accepted a bet at 20:1 odds that my driver’s license would say “Mark Xu” on it.

I think they wouldn't have, mostly because you (or someone) offering the bet is fairly strong evidence that the name on your driver's license is not, in fact, "Mark Xu".

AI x-risk reduction: why I chose academia over industry

Also, you can try for a top-20 tenure-track position and, if you don't get it, "fail" gracefully into a research scientist position. The paths for the two of them are very similar (± 2 years of postdoctoral academic work).

Bootstrapped Alignment

Friendly AI, which is that it be designed to be aligned, say in a mathematically provable way, rather than as an engineered process that approaches alignment by approximation.

I think I understand that now, thank you!

this avoids Goodharting because there's no optimization being applied

I'm confused again here. Is this implying that a Friendly AI, per the definition above, is not an optimizer?

I am very pessimitic about being able to align an AI without any sort of feedback loop on the reward (thus, without optimization). The world's overall transition dynamics are likely to be chaotic, so the "initial state" of an AI that is provably aligned without feedback needs to be exactly the right one to obtain the outcome we want. It could be that the chaos does not affect what we care about, but I'm unsure about that, even linear systems can be chaotic.

It is not an endeavour as clearly impossible as "build an open-loop controller for this dynamical system", but I think it's similar.

Bootstrapped Alignment

I'm confused, I don't know what you mean by 'Friendly AI'. If I take my best guess for that term, I fail to see how it does not rely on optimization to stay aligned.

I take 'Friendly AI' to be either:

- An AI that has the right utility function from the start. (In my understanding, that used to be its usage.) As you point out, because of Goodhart's Law, such an AI is an impossible object.
- A mostly-aligned AI, that is designed to be corrigible. Humans can intervene to change its utility function or shut it down as needed to prevent it from taking bad actions. Ideally, it would consult human supervisors before taking a potentially bad action.

In the second case, humans are continuously optimizing the "utility function" to be closer to the true one. Or, modifying the utility function to make "shut down" the preferred action, whenever the explicit utility function presents a 'misaligned' preferred outcome. Thus, it also represents an optimization-based *weak alignment* method.

Would you argue that my second definition is also an impossible object, because it also relies on optimization?

I think part of my confusion comes from the very fuzzy definition of "optimization". How close, and how fast, do you have to get to the maximum possible value of some function U(s) to be said to optimize it? Or is this the entirely wrong framework altogether? There's no need to answer these now, I'm mostly curious about a clarification for 'Friendly AI'.

Luna Lovegood and the Chamber of Secrets - Part 12

Your particular taste? I loved HPMOR but this is too absurd to be fun for me.

[Personal Experiment] One Year without Junk Media

Great initiative! I'm curious what is the reasoning behind this:

Recorded university lectures and recorded conference talks are forbidden.

Do you find that you get no learning benefit from these things? If so this would explain why also you're not allowing didactic youtube channels (Kurzgesagt, 3blue1brown, CGP grey...)

Interpretations of "probability"

I clicked this because it seemed interesting, but reading the Q&A:

In atypical game we consider, one player offers bets, another decides how to bet, and a third decides the outcome of the bet. We often call the first player Forecaster, the second Skeptic, and the third Reality.

How is this *any* different from the classical Dutch Book argument, that unless you maintain beliefs as probabilities you will inevitably lose money?

Interpretations of "probability"

Frequentist statistics were invented in a (failed) attempt to keep subjectivity out of science in a time before humanity really understood the laws of probability theory

I'm a Bayesian, but do you have a source for this claim? It was my understanding that Frequentism was mostly promoted by Ron Fisher in the 20th century, well *after* the work of Bayes.

Synthesised from Wikipedia:

While the first cited frequentist work (the weak law of large numbers, 1713, Jacob Bernoulli, Frequentist probability) predates Bayes' work (edited by Price in 1763, Bayes' Theorem), it's not by much. Further, according to the article on "Frequentist Probability", "[Bernoulli] is also credited with some appreciation for subjective probability (prior to and without Bayes theorem)."

The ones that pushed frequentism in order to achieve objectivity were Fisher, Neyman and Pearson. From "Frequentist probability": "All valued objectivity, so the best interpretation of probability available to them was frequentist". Fisher did other nasty things, such as using the fact that causality is really hard to soundly establish to argue that tobacco was not proven to cause cancer. But nothing indicates that this was done out of not understanding the laws of probability theory.

AI scientists use the Bayesian interpretation

Sometimes yes, sometimes not. Even Bayesian AI scientists use frequentist statistics pretty often.

This post makes it sound like frequentism is useless and that is not true. The concepts of: a stochastic estimator for a quantity, and looking at whether it is biased, and its variance; were developed by frequentists to look at real world data. AI scientists use it to analyse algorithms like gradient descent, or approximate Bayesian inference schemes, but the tools are definitely useful.

I don't, and my best guess is that nobody has done it :) The paper you linked is extremely recent.

You'd have to start by finding an ODE model of predictive coding, which I suppose is possible taking limit of the learning rate to 0.