(This was inspired by the following question by Daniel Murfet: "Can you elaborate on why I should care about Kelly betting? I guess I'm looking for an answer of the form "the market is a dynamical process that computes a probability distribution, perhaps the Bayesian posterior, and because of out of equilibrium effects or time lags or $X$, the information you derive from the market is not the Bayesian posterior and therefore you should bet somehow differently in a way that reflects that"?")

[See also: Kelly bet or update and Superrational agents Kelly bet influence] 

Why care about Kelly betting? 

1. (Kelly betting is asymptotically dominant) Kelly betting is the asymptotically dominant strategy - it dominates (meaning it has more money) all betting strategies with probability approaching 1 as the time horizon goes to infinity. [this is explained in section of 16.3 of Thomas & Cover's Information Theory textbook]. For long enough time horizons we should expect the Kelly bettors to dominate. 

2. (Evolution selects for Kelly Betting) Evolution selects for Kelly bettor - in the evolutionary biology literature people talk about the mean-variance trade-off.

Define the fitness of an organism O as the number of offspring (this is a random variable) it produces in a generation.  Then according to natural selection the organism 'should' maximize not the absolute fitness E[# of offspring of O] but it should maximize the (long-run) relative inclusive fitness or equivalently the inclusive fitness growth rate. 

Remark.  That evolution selects for relative fitness - not absolute fitness could select for more 'spiteful strategies' like big cats killing each other cubs (both inter and intra-species)

3. (Selection Theorems and Formal Darwinism) 

One of the primary pillars of Wentworth's agenda is 'Selection Theorems': mathematically precise theorems that state what kind of agents might be 'selected' for in certain situations. The Kelly optimality theorem (section 16.3 Thomas & Cover) can be seen as a form of selection theorem: it states that over time Kelly bettors will exponentially start to dominate other agents. It would be of interest to see whether this can be elucidated and the relation with natural selection be improved. 

This closely ties in with a stream of work on Formal Darwinism, a research programme to mathematically if, how and in what sense natural selection creates optimizes for 'fitness' see also Okasha's "Agents and Goals in Evolution"

4. (Ergodicity Economics) Ole Peters argues that Kelly betting (or his more general version of maximizing 'time-averaged' growth) 'solves' the St. Petersburg utility paradox and points to a revolutionary new point of view in foundations of economics: "Ergodicity Economics". As you can imagine this is rather controversial. 

5. (Kelly betting and Entropy) Kelly betting is intrinsically tied to the notion of entropy. Indeed, Kelly discovered Kelly betting to explain Shannon's new informational entropy - only later was it used to beat the house at Las Vegas.

6. (Relevance of Information) A criminally-underrated paper by Madsen continues on Kelly's original idea and generalizes to a notion of (Madsen-)Kelly utility. It measures the 'relevance of information'. Madsen investigates a number of cool examples where this type of thinking is quite useful.

7.  (Bayesian Updating) If we consider a population of hypotheses $H_i$ with a prior $\varphi \left(i\right)$ , we can think of an individual hypothesis $H_i$ as a Kelly bettor with wealth $\varphi \left(i\right)$ distributing its bets according to $H_i$. In other words, it bets $H_i\left(x\right)\varphi \left(i\right)$ on each outcome $x$ in the sample space $\Omega$. It can't 'hold money' at the side - it must bet all its money. In this case, Kelly betting recommends betting according to your internal probability distribution (which is just $H_i$ in this case). 

• Remark. What happens in the case that the bettors can hold money on the side? In other words, we would consider a more flexible bettor. That's quite an interesting question I'd like to answer. I suspect it has to do with Renyi entropy and $\beta$ tempered distributions. 

If we consider a collection of realization $\{x_1,...,x_n\}$ the new wealth of the $H_i$ will be $H_i\left(x_1\right)\cdot H_i\left(x_2\right)...H_i\left(x_n\right)\varphi \left(i\right)$. This is if we bet against Nature. In this case, one can only 'lose'. However, real betting is against a counterparty. In this case it will be betting against the average of the whole market $H\left(x\right)={\sum }_i{H}_i\left(x\right)\varphi \left(i\right)$. If an event $E$ happens the new wealth of $H_i$ will be $\frac{H_i\left(x\right)}{H\left(x\right)}\varphi \left(i\right)$. 

This is of course the Bayesian posterior. 

If we sample from a 'true' distribution $q$, the long-term wealth of $H_i$ will be proportional to $\propto \varphi \left(i\right)\exp \left(KL\right(q|H_i\left)\right)$ 

8. (Blackjack) One application of Kelly betting is bankrupting the House and becoming a 1/2-billionaire.  

Imprecise Probability I: the Bid-Ask Spread measures Adversariality

Definition. A credal set or imprecise probability distribution I is a convex closed set of probability distributions $p_i$. 

For a given event $A$ we obtain an upper- and a lower probability/price 

$\overline{P}\left(A\right)=ma{x}_i{p}_i\left(A\right),\underset{–}{P}\left(A\right)=mi{n}_i{p}_i\left(A\right)$

In other words, we have an buy and a sell price for $A$.

Remark. Vanessa's infraDistributions generalize imprecise probability further in a way that I do not fully understand yet.

Let me talk a little about why thinking in terms of imprecise probability may be helpful. Imprecise probability has a bid-ask spread for events; that is the difference between the upper and lower probability. In many ways this measures the difference between 'aleatoric' and 'epistemic' uncertainty. This is particularly relevant in adversarial situations (which gets into the reasons Vanessa is interested in these things). Let me give a couple e

Example. (Earning calls) When the earning call comes in for a company the bid-ask spread of the stock will increase. Intuitively, the market expects new private information to come into the market and by increasing the bid-ask spread it insures itself agaisnt being mugged.

Example. (Resolution Uncertainty) If you know A will resolve you should buy shares on A, if you know not A will happen you should buy shares on not A. If you think A will not resolve you should sell (!) shares on A. The Bid-ask Spread measures bet resolution uncertainty 

Example. (Selective reporting) Suppose an adversary has an interest in showing you $A$ if $A$ happens and for it not to resolve if  $NOTA$, i.e. this is a form selective reporting that is so essential in politics. In this case you should buy $A$ and sell $NOTA$. 

Example (Forecasting) "for some large class of events, if you ask people how many years until a 10%, 50%, 90% chance of event $X$ occurring, you will get an earlier distribution of times than if you ask the probability that $X$ will happen in 10, 20, 50 years. (I’ve only tried this with AI related things, but my guess is that it at least generalizes to other low-probability-seeming things. Also, if you just ask about 10% on its own, it is consistently different from 10% alongside 50% and 90%." 

This is well-known phenomena is typically considered a failure of human rationality but it can be explained quite neatly using imprecise probability & Knightian uncertainty. [I hasten to caveat that this does not prove that this is the real reason for the phenomena, just a possible explanation!] 

An imprecise distribution I is the convex (closed) hull of a collection of probability distributions $p_i,i\in I$: In other words it combines 'Knightian' uncertainty with probabilistic uncertainty.

If you ask people for 10%, 50%, 90% chance of AI happening you are implicitly asking for the worst case: i.e. there in at least one probability distribution $p_k$, such that $p_k\left(AGI\right)$ = 10%,50%,90% On the other hand when you ask for a certain event to happen for certain in 10,20,50 years you are asking for the dual 'best case' scenario, i.e. for ALL probability distributions $p_i$ what probability $p_i$(AGI in 10y), $p_i($AGI in 20y$),p_i$(AGI in 50y$)$ and taking the minimum.

What is the correct notion of Information Flow?

This shortform was inspired by a very intriguing talk about Shannon information theory in understanding complex systems by James Crutchfield. 

There is an idea to define information flow as transfer entropy.  One of the problems is the following example by Crutchfield:

Example we have two stochastic processes $X=X_0,X_1,...,X_{n-1}$$,X_n$ and $Y=Y_0,Y_1,...,Y_{n-1}$, $Y_n$ where $Y_0=$ is a fair coin flip, and all the $X_i$ are independent fair coin flips.$Y_n$ is defined as $X_{n-1}+Y_{n-1}\left(Mod2\right)$, or said differently $Y_n=X_{n-1}XOR{Y}_{n-1}$. If you write it out it seems like the transfer entropy is saying that $X_i$ is sending 1 bit of information to $Y_{i+1}$ each time-step but this is stymied by the fact that we can also understand this information as coming from $Y_0$.

Initially I thought this was a simple case of applying Pearl's causality - but that turned out to be too naive. In a way I don't quite understand completely this is stymied by higher-order dependencies - see this paper by James & Crutchfield. They give a examples of joint probability distributions over variables X,Y,Z where the higher-order dependencies mean that they cannot be understand to have a Pearlian DAG structure. 

Moreover, they point out that very different joint distributions can look identical from the lens of Shannon information theory: conditional mutual information does not distinguish these distributions. The problem seems to be that Shannon information theory does not deal well with XORed variables and higher-order dependencies. 

A natural thing to attack/understand higher-order dependencies would be to look at 'interaction information' which is supposed to measure 3-way dependencies between three variables. It can be famously negative - a fact that often surprises people. On the other hand, because it is defined in terms of conditional mutual information it isn't actually able to distinguish the Crutchfield-James distributions. 

We need new ideas. An obvious idea would be to involve topological & homotopical & knot theoretic notions - this theories are the canonical ways of dealing with higher-order dependencies. In fact, I would say this is so well-understood that the 'problem' of higher-order dependencies has been 'solved' by modern homotopy theory [which includes cohomology theory]. On the other hand, it is not immediately clear how to 'integrate' ideas from homotopy theory directly into Shannon information theory, but see below for one such attempt. 

I see three possible ways to attack this problem:

1. Factored Sets
2. Cyclic Causality
3. Information Cohomology

Factored Sets. The advantage of factored sets is that they are able to deal with situations where some variables are XORs of other variables and pick out the 'more primitive' variables. In a sense it solves the famous philosophical blue-grue problem. A testament to the singular genius of our very own Garrabrant! Could this allow us to define the 'right' variant of Shannon information theory?

EDIT: in a conversation Scott suggested to me that the problem is that they chose an implict factorization of the variables- they should be looking at all possible variables.  If we also look at information measures on variables like X XOR Y we should be able to tell the distributions apart. 

Cyclic Causality. Pearl's DAG Causality framework is rightly hailed as a great revolution. Indeed, it consequences have only begun to be absorbed. There is a similar, in fact more general theory already known in econometrics research known as 'structural equation modelling'. 

SEM is more general in that it allows for cyclic causality. Although this might invoke in the reader back-to-the-future grandfather paradoxes, closed time-like curves and other such exotic beasts cyclic causality is actually a much more down-to-earth notion: think about a pair of balls connected by a string. The positions of the balls are correlated, and in fact this correlation is causal: move one ball and the other will move with it and vice versa. Cyclic causality! 

Of course balls connected by strings are absolutely fundamental in physics: when the string is elastic this becomes a coupled Harmonic oscillator!

a paper that goes into the details of latent and cyclic causality can be found here  

Information Cohomology. There is a obscure theory called Information cohomology based on the observation that Shannon entropy can be seen as a 1-cocycle for a certain boundary operator. Using the abstract machinery of topos & cohomology theory it is then possible to define higher order cohomology groups and therefor higher-order Shannon entropy. I will not define information cohomology here but let me observe:
  -entropy H(*) is a cocycle, but not a coboundary. H gives rise to a cohomology class. It's actually nontrivial.
- Conditional mutual information is neither a cocycle or a coboundary. This might be either a bug or a feature of the theory, I'm not sure yet. 
- Interaction information I(*,*,*) is a coboundary and thus *also* a cocycle. In other words, it is trivial as a cohomology class. This is might be a feature of the theory! As observed above, because Interaction Information is defined by conditional mutual information it cannot actually pick-up on subtle higher-order dependencies and distinguish the James-Crutchfield distributions... but a nontrivial 3-cocycle in the $H^3$ of Information Cohomology might! This would be really awesome. I will need to speak to an expert to see if this works out. 

[Unimportant Rk. That Shannon entropy might have something to do with differential operators is not completely insane perhaps owing to an observations due to John Baez: the Fadeev characterisiation of Shannon entropy is a the derivatives of glomming partition functions . It is unclear to me whether this is related to the Information Cohomology perspective however]

Remark. I have been informed that both in the Cyclic Causality and the Information Cohomology framework it seems natural to have an 'assymetric independence' relation. This is actually very natural from the point of view of Imprecise Probability & InfraBayesianism. In this theory the notion of independence 'concept splinters' into at least three different notion, two of which are assymetric - and better thought of as 'irrelevance' rather than 'independence' but I'm splitting hairs here. 

Multi-Step Fidelity causes Rapid Capability Gain

tl; dr Many examples of Rapid Capability Gain can be explained by a sudden jump in fidelity of a multi-step error-prone process. As the single step error rate is gradually lowered there is a sudden transition from a low fidelity to a high fidelity regime for the corresponding multistep process. Examples abound in cultural transmission, development economics, planning & consciousness in agent,  origin of life and more. 

Consider a factory making a widget in N distinct steps. Each step has a probability of a fatal error and the subsequent step can only occur if the previous step was succesful. For simplicity we assume that the chance of an error for each step is given by a single 'single step error rate' parameter. What is the error rate for the entire process?

Here are some values

Math nerd remark: for an $N$-step process with $1/N$ single step error rate will be approximately $\frac{1}{e}=0.37$. We see that an order of magnitude difference in the single step error rate or an order of magnitude difference in the number of steps can be the difference between a completely unrealistic plan ($0.27\ast {10}^{-4}$) and plan with at least a fair chance of working (0.34). Another order of magnitude worse fidelity or longer multistep process goes from a unrealistic plan (0.27 *10^{-4}) to an astronomically unlikely. 

It's a simple causal mechanism that shows up in many different places whenever we sudden capability jumps.

Why are some countries much richer than others?

(see also Gareth Jones https://www.sup.org/books/title/?id=23082 )
The ultimate cause(s) are a point of contention, but the proximate cause is simple: rich countries produce complex specialized goods that are much more valuable than their raw inputs. These are produced by large highly hierarchical teams of specialists. Making complex specialized goods like uranium refinement, aeroplanes, microchips, industrial tooling requires many processing steps. To efficiently produce these products it is imperative to have single-step fidelity. For whatever reason rich countries have been succesful in lowering this single-step error rate. 

Rk. As an aside, most of the gains are not actually captured by these specialists because of comparative advantage. Similarly but perhaps surprisingly, low skill workers gained with respect to high skill workers during the Industrial Revolution. [Citation!]

Cultural Transmission Fidelity as key indicator in human cultural evolution

Secret of our Success is a recent book on the high fidelity cultural transmission model.   https://press.princeton.edu/books/paperback/9780691178431/the-secret-of-our-success

Cultural Transmission Fidelity as cause of human-ape divergence

We can also understand cultural transmissionAtleast whales and apes have forms of cultural learning too. The difference seems to be the fidelity of human cultural transmissions. Humans are also smarter per individual though so it can't be just cultural transmission (the relevant quantitity is probably cortical neurons - humans are only outclassed by certain whales on this measure - and even compared to whales human brain are probably superior, being more densely packed). However, it is likely 

Proofs & High Fidelity Reasoning.

The remarkable deep structure of modern mathematics probably partially explained by high fidelity reasoning furnished by proofs.  (see also thin versus thick reasoning)

RNA & DNA copying fidelity

For Life to have started it would have been necessary for a high-fidelity self-replicating process to arise. 

There is a fairly well-supported theory on the Origin of Life that holds that initially life was all RNA based - which has an intrinsically high mutation rate. Once DNA came onto the scene the mutation rate became much lower -> complex life like bacteria became possible. 

IQ-divergence on increasingly harder tasks

Why we see more IQ-divergence on harder tasks? High IQ individuals probably have higher fidelity on single step reasoning - multi-step reasoning problems start to favor the shrewd more and more as the number of steps increases. 

 Global workspace theory 

 The key variable of high-level serial conscious reasoning according to global workspace theory is how fast teh different modules can communicate with one another. That is, the latency of communication. This key parameter plausibly underlies much of intelligence (indeed in the theory this is basically working memory which is highly correlated with IQ). 

Why do we need mental breaks? Why do we get mentally tired? Why do we task switch?

 Anecdotally, many people report that they can focus only for limited few-hour time slots for creative focused concious work. 

Naively, one would think that the brain is getting tired like a muscle yet the brain -as - muscle might be a misleading analogy. It does not seem to get tired or overexert itself.  For instance, the amount of energy used does not significantly vary with the task [LINK?]. 

Global Workspace theory suggests that focused conscious reasoning is all about serially integrating summarized computations from many parallel unconcious computing units. After finishing the serial conscious thought the conclusion is backpropagated to unconscious computing units. Subsequently, these unconscious computing units need to spend time to work on the backpropagated conscious thought before there is enough 'fertile ground' for further serial conscious thought. 

Famous scientists often credit dreams and downtime with creative insights. This explanation would fit that.

It could also explain why it seems easier to change conscious activities. Switching tasks can be more computationally efficient. 

A related but different frame is related to how human memory is encoded:

Human memory is a form of associative memory, very different from the adress-based memory of computers. Our best model of human memory are Hopfield Networks/Ising models. Patterns that are correlated are stored less efficiently as they can interfere with one another. There is an hypothesis that part of sleeping is getting rid of spurious correlations in our learned memories such as to encode them better. This takes time - in the process the data becomes more distinct, better learnt, and more compressed!  This means that later on we may compute with the pattern faster on a later timestep. 

An alternative mechanism is a reinforcement learning task with uncertain and delayed rewards. Task switching becomes optimal if there is a latency/uncertainty in the reward signal. Compare the Procrastination equation: https://www.lesswrong.com/posts/RWo4LwFzpHNQCTcYt/how-to-beat-procrastination

(This was inspired by Gabriel's post on Super Hard problems)

Trapdoor Functions and Prime Insights

One intuition is that solving hard problems is like finding the secret key to a trapdoor function. Funnily enough, the existence of trapdoor functions relies on conjectures implying $P\ne NP$ so the existence of barriers in the PvsNP conjecture is possibly no coincidence. I suspect that we will need to understand computational complexity perhaps intelligence & learning theory significantly better to be able to give convincingly quantify why some problems are Super Hard. 

Strictly speaking, we can't prove trapdoor functions exists but we do use functions which we suspect to be trapdoor functions all the time in cryptography. 

Example. One simply example of a suspected trapdoor function is factoring a composite number.

Given a product of two large primes $N=pq$ the problem is to find the prime factorisation. If I give you $p$, it is easy to find $q$ by polynomial time division. In this sense the prime $p$ (or symmetrically $q$) is akin to an 'insight'.

More generally, we may consider a factorization $N=p_1\cdots p_k$ of a product of $k$ primes. Each prime serves as a 'separate' discoverable insight.  

One would like to argue that because of the Unique Factorisation Theorem the only way to find the factorisation of $N$ is to find each of these prime factors step-by-step. In other words, the each prime represents a 'necessary insight'. This argument is not quite sound for subtle arithmetic reasons but it might give a flavor of what we mean when we talk about 'necessary insights'.

Artificial/Natural

Q: Why do we call some things Natural - other things Artificial? Why do we associate 'Natural' with good, 'Artificial' with bad? Why do we react so vehemently to artificial objects/phenomena that are close to 'natural' objects/phenomena?

A: A mundane answer could be: natural is a word describing a thing, situation, person, phenomenon etc that was experienced in the ancestral environment - whatever way you understand with this - I don't necessarily mean people in caves.  Instead of ancestral environment think 'training set for the oldbrain / learned priors in the brain & human body'. Its counterpart is often used to describe things recently made by humans and their egregores like global capitalism.

Q: Why might this be a useful distinction?

A: In some sense humans, and human culture is 'well-adapted' to natural things/phenomena in a way it isn't for 'artificial' phenomena.

It's a Trap

Artificial things/environment/situations potentially contain more 'traps' - an important concept I learned from Vanessa . For example, it very well could be that some chemical we use nowadays will make us all infertile (even if in 99% cases it is overblown scaremongering).
We 'know' that isn't the case with 'natural' substances/ practices because we have the genetic & cultural memory of humans lasting over long time periods. From a learning-theoretic perspective one could say that sometimes correct beliefs can be obtained in a single episode reinforcement learning + model based computation ("rational reasoning"). In some situations, the world can't efficiently be learned this way.

Simulacra

Artificial often has a stronger negative connotation than 'just' a potentially dangerous thing/phenomena not seen in the ancestral environment. Colloquially, being artificial implies being designed often with the goal of 'simulating' an original 'natural thing'.

For reinforcement learning agents encountering artificial objects & phenomena is potentially dangerous: reinforcement learners use proxies . If those "natural" proxies get simulated by "artificial" substitutes this may lead to the reinforcement learner Goodhearting on the artificial subsitute proxy. In other words, the reward machinery gets 'hacked'.

Animals on Drugs

A paradigmatical example is drug addicts. Rather than an ailment of modern society, habitual drug use and abuse is widespread thoughout human history, and even observed in animals. Other examples could be pornography, makeup and parent birds feedings cuckoo chicks.

Uncanny Valley Defense

Artificial can have an even stronger negative connotation: it is not just unnatural, it is not just 'hacking the reward machinery' by accident - since artificial objects are 'designed', they can be also designed adversarially. 

The phenomena of artificial substitutes hacking ancient reward mechanisms is common now. I claim it is was a common enough problem in the past for humans & perhaps animals to have developed defenses against reward hacking.  This might explain the uncanny valley effect in psychology. It might also explain why many humans & animals are actually surprisingly resistant to drug abuse. 

Normative principles in Rationality

Here are a number of proposed normative principle about how to 'reason well' and/or 'decide well'.

• Kelly betting: Given a wager pick the one that maximizes your growth rate. In many-cases this coincides with log-wealth.
  • Maximize growth rates. Ole Peters' Principle. 
  • Maximize Geometric mean. 
  • Maximize log-wealth. 
• Efficient Market hypothesis (/No Arbitrage/Garrabrant criterion) infinite:
  • Efficient Market Hypothesis (finite cut-off): when betting/trading in a market, we cannot multiply our initial capital $C$ with a large fraction $\alpha$ over the risk-less rate $r$.
• Maximum Entropy Principle (MaxEnt): given constraints $R_i$ on an unknown probability probability distribution $p$, your 'best' estimate is $p^{\ast }$ that maximizes the entropy under the constraints $R_i$.
  • Maximum Caliber (MaxCal): given dynamical (time-dependent) constraints $f_i\left(t\right)$ on an unknown stochastic process the 'best' estimate for the underlying distribution on paths or 'law' $\mu$ is the path distribution that maximizes path entropy under constraints. 
  • Maximum Causal Entropy
• Minimum Description Length Principle
  • Solomonoff Induction
  • Minimize Kolmogorov complexity
  • Minimize Bernouli-Kolomogorov complexity (statistical complexity). 
• Bayesian Updating

How might these be related?

Kelly betting or log-wealth betting is generalized by Ole Peters' Principle of Maximizing Growth Rates.

MaximumCaliber is a straightforward generalization of MaxEntropy. MaxCausalEnt

Bayesian updating is a special case of MaxEnt. 

MaxEnt is a special case of Ole Peters.

Note that Maximum Caliber is different in the following sense: