Eigil Rischel — LessWrong

LESSWRONG
LW

Replying toTransportation as a Constraint

Well, the cars are controlled by a centralized system with extremely good security, and the existence of the cars falls almost entirely withing an extended period of global peace and extremely low levels of violence. And when war breaks out in the fourth book they're taken off the board right at the start, more or less.

(The cars run on embedded fusion engines, so their potential as kinetic weapons is the least dangerous thing about them).

Replying toClarifying the Agent-Like Structure Problem

Eigil Rischel3y

Clarifying the Agent-Like Structure Problem

I mean, "is a large part of the state space" is basically what "high entropy" means!

For case 3, I think the right way to rule out this counterexample is the probabilistic criterion discussed by John - the vast majority of initial states for your computer don't include a zero-day exploit and a script to automatically deploy it. The only way to make this likely is to include you programming your computer in the picture, and of course you do have a world model (without which you could not have programmed your computer)

Replying toOpenAI Solves (Some) Formal Math Olympiad Problems

Eigil Rischel4y

OpenAI Solves (Some) Formal Math Olympiad Problems

I think we're basically in agreement here (and I think your summary of the results is fair, I was mostly pushing back on a too-hyped tone coming from OpenAI, not from you)

Replying toOpenAI Solves (Some) Formal Math Olympiad Problems

Eigil Rischel4y

OpenAI Solves (Some) Formal Math Olympiad Problems

Most of the heavy lifting in these proofs seem to be done by the Lean tactics. The comment "arguments to nlinarith are fully invented by our model" above a proof which is literally the single line nlinarith [sq_nonneg (b - a), sq_nonneg (c - b), sq_nonneg (c - a)] makes me feel like they're trying too hard to convince me this is impressive.

The other proof involving multiple steps is more impressive, but this still feels like a testament to the power of "traditional" search methods for proving algebraic inequalities, rather than an impressive AI milestone. People on twitter have claimed that some of the other problems are also one-liners using existing proof... (read more)

Eigil Rischel4y

Yes, that's right. It's the same basic issue that leads to the Anvil Problem

Eigil Rischel4yQuick Take

Compare two views of "the universal prior"

AIXI: The external world is a Turing machine that receives our actions as input and produces our sensory impressions as output. Our prior belief about this Turing machine should be that it's simple, i.e. the Solomonoff prior
"The embedded prior": The "entire" world is a sort of Turing machine, which we happen to be one component of in some sense. Our prior for this Turing machine should be that it's simple (again, the Solomonoff prior), but we have to condition on the observation that it's complicated enough to contain observers ("Descartes' update"). (This is essentially Naturalized induction

I think of the difference between these as "solipsism" - AIXI... (read more)

Eigil Rischel4y

Right, that's essentially what I mean. You're of course right that this doesn't let you get around the existence of nonmeasurepreserving automorphisms. I guess what I'm saying is that, if you're trying to find a prior on $[0, 1]$ , you should try to think about what system of finite measurements this is idealizing, and see if you can apply a symmetry argument to those bits. Which isn't always the case! You can only apply the principle of indifference if you're actually indifferent. But if it's the case that $X \in [0, 1]$ is generated in a way where "there's no reason to suspect that any bit should be $0$ rather than $1$ , or that there should... (read more)

Replying toThe Promise and Peril of Finite Sets

Eigil Rischel4y

The Promise and Peril of Finite Sets

I strongly recommend Escardó's Seemingly impossible functional programs, which constructs a function search : ((Nat -> Bool) -> Bool) -> (Nat -> Bool) which, given a predicate on infinite bitstrings, finds an infinite bitstring satisfying the predicate if one exists. (In other words, if p : (Nat -> Bool) -> Bool and any bitstring at all satisfies p, then p (search p) == True

(Here I'm intending Nat to be the type of natural numbers, of course) .

Eigil Rischel4y

Ha, I was just about to write this post. To add something, I think you can justify the uniform measure on bounded intervals of reals (for illustration purposes, say ) by the following argument: "Measuring a real number $x \in [0, 1]$ " is obviously simply impossible if interpreted literally, containing an infinite amount of data. Instead this is supposed to be some sort of idealization of a situation where you can observe "as many bits as you want" of the binary expansion of the number (choosing another base gives the same measure). If you now apply the principle of indifference to each measured bit, you're left with Lebesgue measure.

It's not clear that there's a "right" way to apply this type of thinking to produce "the correct" prior on $N$ (or $R$ or any other non-compact space.

Replying toI’m no longer sure that I buy dutch book arguments and this makes me skeptical of the "utility function" abstraction

Eigil Rischel5y

I’m no longer sure that I buy dutch book arguments and this makes me skeptical of the "utility function" abstraction

This seems somewhat connected to this previous argument. Basically, coherent agents can be modeled as utility-optimizers, yes, but what this really proves is that almost any behavior fits into the model "utility-optimizer", not that coherent agents must necessarily look like our intuitive picture of a utility-optimizer.

Paraphrasing Rohin's arguments somewhat, the arguments for universal convergence say something like "for "most" "natural" utility functions, optimizing that function will mean acquiring power, killing off adversaries, acquiring resources, etc". We know that all coherent behavior comes from a utility function, but it doesn't follow that most coherent behavior exhibits this sort of power-seeking.

Has anybody used quantification over utility functions to define "how good a model is"?

Eigil Rischel

Let's consider a very simple situation: you're an agent playing a game like this: A random point $x \in X$ is generated, which is shown to you. Then you have to choose an action $a \in A$ . This leads to a world-state $w (a, θ) \in W$ , where $θ \in Θ$ is some "hidden variable", which is randomly generated but which you don't see.

Now, the way to make decisions in a situation like this is to form some "statistical model". In this very simple situation, maybe that just means a set of conditional probabilities $P (w = w_{0} | x, a)$ , giving the probability of each world-state as a function of your action and the $x$ you see.

On the other hand, it could also be $P (w \in O | x, a)$ for various subsets $O \subset W$ ... (read 357 more words →)

Where numbers come from

Eigil Rischel

Crossposted from my personal website

Alternative title: Wolves hate him!! Shepherd compares the size of large sets with this one easy trick!

Previously: Recognizing Numbers

Let's do a thought experiment. I place an empty box in front of you. Then, while you're watching, I put these objects into the box:

2 apples

Then I remove these things from the box:

3 apples

You're surprised! Why? Because what I took out is not a subset of what I put in. A new apple appeared.

You can do this experiment with animals, and small children of various ages, and monitor them carefully to see if they seem surprised. You can also try larger collections of apples, to see how large a collection of apples they... (read 1118 more words →)

Demystifying the Second Law of Thermodynamics

Eigil Rischel

(Crossposted from my personal website)

Thermodynamics is really weird. Most people have probably encountered a bad explanation of the basics at some point in school, but probably don't remember more than

Energy is conserved
Entropy increases
There's something called the ideal gas law/ideal gas equation.

Energy conservation is not very mysterious. Apart from some weirdness around defining energy in general, it's just a thing you can prove from whatever laws of motion you're using.

But entropy is very weird. You've heard that it measures "disorder" in some vague sense. Maybe you've heard that it's connected to the Shannon entropy of a probability distribution $H (p) = \sum_x - p (x) ln p (x)$ . Probably the weirdest thing about it is the law it obeys: It's not conserved, but rather... (read 1542 more words →)

How much is known about the "inference rules" of logical induction?

Eigil Rischel

Context: Logical Induction is a framework that makes sense of intuitively plausible statements like "the probability that the $10^{10^{10^{10}}}$ th digit of $π$ is odd is about $0.5$ ".

People often do this sort of informal reasoning about mathematical conjectures. Like "The Collatz conjecture has been checked up to $2^{68}$ , and held for all those - updating on this, I increase my likelyhood that the conjecture is true in general". Logical induction seems to provide, in principle, a set of rules that such updates should follow. How many of these rules are known?

Some example rules that seem very plausible (here all my variables are implicitly natural numbers):

The observation that $ϕ (n_{0})$ is true does not decrease the likelyhood of $\forall n ϕ (n)$ .
Updating on the observations " $ϕ (n)$ for all $0 \leq n \leq N$ ", the probability of $\forall n ϕ (n)$ goes to $1$ as $N \to \infty$

Do these hold for logical inductors?

A thought about productivity systems/workflow optimization:

One principle of good design is "make the thing you want people to do, the easy thing to do". However, this idea is susceptible to the following form of Goodhart: often a lot of the value in some desirable action comes from the things that make it difficult.

For instance, sometimes I decide to migrate some notes from one note-taking system to another. This is usually extremely useful, because it forces me to review the notes and think about how they relate to each other and to the new system. If I make this easier for myself by writing a script to do the work (as I have sometimes done), this important value is lost.

Or think about spaced repetition cards: You can save a ton of time by reusing cards made by other people covering the same material - but the mental work of breaking the material down into chunks that can go into the spaced-repetition system, which is usually very important, is lost.

Bets and updating

Eigil Rischel

Suppose the US presidential election is tomorrow. You currently assign a probability of 50% to each outcome. (We are ignoring the small possibility that neither of the main party candidates will win).

A man approaches you, and offers you a bet of $10, at 2-1 odds. In other words, if candidate one wins, he pays you $20, if candidate two wins, you pay him $10.

Should you accept this bet? What if the bet was for $10000 instead? Assume that your utility is linear in dollars (or assume that the bet is for utilons instead, whatever). If not, why not? Try to think about this before reading on.

The answer is that it depends on your... (read 429 more words →)

I've noticed a sort of tradeoff in how I use planning/todo systems (having experimented with several such systems recently). This mainly applies to planning things with no immediate deadline, where it's more about how to split a large amount of available time between a large number of tasks, rather than about remembering which things to do when. For instance, think of a personal reading list - there is no hurry to read any particular things on it, but you do want to be spending your reading time effectively.

On one extreme, I make a commitment to myself to do all the things on the list eventually. At first, this has the desired effect

... (read more)

Joy in Discovery: Galois theory

Eigil Rischel

Inspired by: Joy in Discovery, Class Project

Your task: Prove either of the following statements:

There exists a polynomial $p (x)$ of degree five over the rational numbers, with a root $x$ , so that $x$ cannot be obtained from the rationals by field operations and iterated extraction of roots.
There is no set formulas, given in terms of field operations and iterated extraction of roots, expressing the roots of a fifth-degree polynomial over the rationals in terms of the coefficients.

Note that the former clearly implies the latter, but you are free to prove either.

By "field operations", i mean just $+, -, \times, /$ . By "extraction of roots", i mean operations of the form "^n\sqrt{(-)}$. You are allowed to take even roots... (read 618 more words →)

Belief: There is no amount of computing power which would make AlphaGo Zero(AGZ) turn the world into computronium in order to make the best possible Go moves (even if we assume there is some strategy which would let the system achieve this, like manipulating humans with cleverly chosen Go moves).

My reasoning is that AGZ is trained by recursively approximating a Monte Carlo Tree Search guided by its current model (very rough explanation which is probably missing something important). And it seems the "attractor" in this system is "perfect Go play", not "whatever Go play leads to better Go play in the future". There is no way for a system like

... (read more)

Eigil Rischel's Shortform

Eigil Rischel

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.