# All of royf's Comments + Replies

An overall schema for the friendly AI problems: self-referential convergence criteria

It seems that your research is coming around to some concepts that are at the basis of mine. Namely, that noise in an optimization process is a constraint on the process, and that the resulting constrained optimization process avoids the nasty properties you describe.

Feel free to contact me if you'd like to discuss this further.

2Stuart_Armstrong6yI fear I will lack time for many months :-( Send me another message if you want to talk later.
Utility vs Probability: idea synthesis

This is not unlike Neyman-Pearson theory. Surely this will run into the same trouble with more than 2 possible actions.

1Stuart_Armstrong6yNo, no real connection Neyman-Pearson. And its fine with more that 2 actions - notice that each action only uses itself in the definition. And u' doesn't event use any actions in its definition.

Our research group and collaborators, foremost Daniel Polani, have been studying this for many years now. Polani calls an essentially identical concept empowerment. These guys are welcome to the party, and as former outsiders it's understandable (if not totally acceptable) that they wouldn't know about these piles of prior work.

You have a good and correct point, but it has nothing to do with your question.

a machine can never halt after achieving its goal because it cannot know with full certainty whether it has achieved its goal

This is a misunderstanding of how such a machine might work.

To verify that it completed the task, the machine must match the current state to the desired state. The desired state is any state where the machine has "made 32 paperclips". Now what's a paperclip?

For quite some time we've had the technology to identify a paperclip in an image, if ... (read more)

0[anonymous]8ySure, it will go ask the user too. And do various other things. But it remains true that if it wants to be maximally confident that it has achieved its target state, at no time will it decide that maximal confidence has been achieved and shut down, because there will always be something that it can do to increase (if only by an increasingly small epsilon) its confidence.
1TheOtherDave8ySure, it will go ask the user too. And do various other things. But it remains true that if it wants to be maximally confident that it has achieved its target state, at no time will it decide that maximal confidence has been achieved and shut down, because there will always be something that it can do to increase (if only by an increasingly small epsilon) its confidence.
Right for the Wrong Reasons

The "world state" of ASH is in fact an "information state" of p("heads")>SOME_THRESHOLD

Actually, I meant p("heads") = 0.999 or something.

(C), if I'm following you, maps roughly to the English phrase "I know for absolutely certain that the coin is almost surely heads".

No, I meant: "I know for absolutely certain that the coin is heads". We agree that this much you can never know. As for getting close to this, for example having the information state (D) where p("heads") = 0.99999... (read more)

0TheOtherDave8yOK. Thanks for clarifying.
Right for the Wrong Reasons

I probably need to write a top-level post to explain this adequately, but in a nutshell:

I've tossed a coin. Now we can say that the world is in one of two states: "heads" and "tails". This view is consistent with any information state. The information state (A) of maximal ignorance is a uniform distribution over the two states. The information state (B) where heads is twice as likely as tails is the distribution p("heads") = 2/3, p("tails") = 1/3. The information state (C) of knowing for sure that the result is heads... (read more)

0TheOtherDave8ySure. And (C) is unachievable in practice if one is updating one's information state sensibly from sensible priors. I am uncertain what you mean to convey in this example by the difference between a "world state" (e.g., ASH or AST) and an "information state" (e.g. p("ASH")=0.668). The "world state" of ASH is in fact an "information state" of p("heads")>SOME_THRESHOLD, which is fine if you mean those terms to be denotatively synonymous but connotatively different, but problematic if you mean them to be denotatively different. Yes, agreed that this is strictly speaking unachievable, just as "I know for absolutely certain that the coin is heads" was. That said, I'm not sure what it means for a human brain to have "I know for absolutely certain that the coin is almost surely heads" as a distinct state from "I am almost sure the coin is heads," and the latter is achievable. Works for me. And now you've lost me again. Of course there are real physical reasons why certain information states are not possible... e.g., my brain is incapable of representing certain thoughts. But I suspect that's not what you mean here. Can you give me some examples of the kinds of cases you have in mind?
Right for the Wrong Reasons

To clarify further: likelihood is a relative quantity, like speed - it only has meaning relative to a specific frame of reference.

If you're judging my calibration, the proper frame of reference is what I knew at the time of prediction. I didn't know what the result of the fencing match would be, but I had some evidence for who is more likely to win. The (objective) probability distribution given that (subjective) information state is what I should've used for prediction.

If you're judging my diligence as an evidence seeker, the proper frame of reference is ... (read more)

0TheOtherDave8yI thought I was following you, but you lost me there. I certainly agree that if I want to evaluate various aspects of your cognitive abilities based on your predictions, I should look at different aspects of your predictions depending on what abilities I care about, as you describe, and that often the accuracy of your prediction is not the most useful aspect to look at. And of course I agree that expecting perfect knowledge is unreasonable. But what that has to do with Omega, and what the uselessness of Omega as a frame of reference has to do with constraints on reality, I don't follow.
Right for the Wrong Reasons

This is perhaps not the best description of actualism, but I see your point. Actualists would disagree with this part of my comment:

If I believed that "you will win" (no probability qualifier), then in the many universes where you didn't I'm in Bayes Hell.

on the grounds that those other universes don't exist.

But that was just a figure of speech. I don't actually need those other universes to argue against 0 and 1 as probabilities. And if Frequentists disbelieve in that, there's no place in Bayes Heaven for them.

Right for the Wrong Reasons

2CronoDAS8yNot really. Let me elaborate: In a book of his, Daniel Dennett appropriates the word "actualism" to mean "the belief that only things that have actually happened, or will happen, are possible." In other words, all statements that are false are not only false, but also impossible: If the coin flip comes up heads, it was never possible for the coin flip to have come up tails. He considers this rather silly, says there are good reasons for dismissing it that aren't relevant to the current discussion, and proceeds as though the matter is solved. This strikes me as one of those philosophical positions that seem obviously absurd but very difficult to refute in practice. (It also strikes me as splitting hairs over words, so maybe it's just a wrong question in the first place?)
Right for the Wrong Reasons

Predictions are justified not by becoming a reality, but by the likelihood of their becoming a reality [1]. When this likelihood is hard to estimate, we can take their becoming a reality as weak evidence that the likelihood is high. But in the end, after counting all the evidence, it's really only the likelihood itself that matters.

If I predict [...] that I will win [...] and I in fact lose fourteen touches in a row, only to win by forfeit

If I place a bet on you to win and this happens, I'll happily collect my prize, but still feel that I put my money ... (read more)

1AlexSchell8yI notice that I am confused. What you say seems plausible but also in conflict with the (also plausible) Yudkowskian creed that probability is in the map.
0CronoDAS8yThat suggests a question. If I flip a fair coin, and it comes up heads, what is the probability of that coin flip, which I already made, having instead been tails? (Approximately) 0, because we've already seen that the coin didn't come up tails, or (approximately) 50%, because it's a fair coin and we have no way of knowing the outcome in advance?
Update Then Forget

Thanks!

The best book is doubtlessly Elements of Information Theory by Cover and Thomas. It's very clear (to someone with some background in math or theoretical computer science) and lays very strong introductory foundations before giving a good overview of some of the deeper aspects of the theory.

It's fortunate that many concepts of information theory share some of their mathematical meaning with the everyday meaning. This way I can explain the new theory (popularized here for the first time) without defining these concepts.

I'm planning another sequence wh... (read more)

1William_Quixote8yThanks
Update Then Forget

This is a perfect agent, of theoretical interest if not practically realizable.

0timtyler8yAssigning a probability to each possible world state?!? That is incredibly inefficient and wasteful. Any implementation attempt would result in slow stupidity - not anything intelligent.
Update Then Forget

an intelligent agent should update on it and then forget it.

Should being the operative word. This refers to a "perfect" agent (emphasis added in text; thanks!).

People don't do this, as well they shouldn't, because we update poorly and need the original data to compensate.

If you forget the discarded cards, and later realize that you may have an incorrect map of the deck, aren't you SOL?

If I remember the cards in play, I don't care about the discarded ones. If I don't, the discarded cards could help a bit, but that's not the heart of my problem.

0shminux8yWhat's a perfect agent? No one is infallible, except the Pope.
A fungibility theorem

if you really care about the values on that list, then there are linear aggregations

Of course existence doesn't mean that we can actually find these coefficients. Even if you have only 2 well-defined value functions, finding an optimal tradeoff between them is generally computationally hard.

How to Be Oversurprised

Philosiphically, yes.

Practically, it may be useful to distinguish between a coin and a toss. The coin has persisting features which make it either fair or loaded for a long time, with correlation between past and future. The toss is transient, and essentially all information about it is lost when I put the coin away - except through the memory of agents.

So yes, the toss is a feature of the present state of the world. But it has the very special property, that given the bias of the coin, the toss is independent of the past and the future. It's sometimes more useful to treat a feature like that as an observation external to the world, but of course it "really" isn't.

How to Be Oversurprised

I'm trying to balance between introducing terminology to new readers and not boring those who've read my previous posts. Thanks for the criticism, I'll use it (and its upvotes) to correct my balance.

How to Be Oversurprised

Well, thank you!

Yes, I do this more for the math and the algorithms than for advice for humans.

Still, the advice is perhaps not so trivial: study not what you're most uncertain about (highest entropy given what you know) but those things with entropy generated by what you care about. And even this advice is incomplete - there's more to come.

How to Be Oversurprised

When the new memory state $M_t$ is generated by a Bayesian update from the previous one $M_{t-1}$ and the new observation $O_t$, it's a sufficient statistic of these information sources for the world state $W_t$, so that $M_t$ keeps all the information about the world that was remembered or observed:

$I\(W\_t;M\_t\$=I(W_t;(M_{t-1},O_t)))

As this is all the information available, other ways to update can only have less information.

The amount of information gained by a Bayesian update is

$I\(W\_t;\(M\_\{t\-1\},O\_t\$)-I(W_t;M_{t-1}))

$=\\mathrm E\\left\[\\log\\frac\{\\Pr\(W\_t,M\_\{t\-1\},O\_t\$}{\Pr(W_t)\Pr(M_{t-1},O_t)}-\log\frac{\Pr(W_t,M_{t-1})}{\Pr(W_t)\Pr(M_{t-1})}\right])

$=\\mathrm E\\left\[\\log\\frac\{\\Pr\(W\_t,M\_\{t\-1\},O\_t\$}{\Pr(W_t,

How to Disentangle the Past and the Future

I explained this in my non-standard introduction to reinforcement learning.

We can define the world as having the Markov property, i.e. as a Markov process. But when we split the world into an agent and its environment, we lose the Markov property for each of them separately.

I'm using non-standard notation and terminology because they are needed for the theory I'm developing in these posts. In future posts I'll try to link more to the handful of researchers who do publish on this theory. I did publish one post relating the terminology I'm using to more stan... (read more)

Conservation of Expected Evidence

Let's assume a strong version of Bayesianism, which entails the maximum entropy principle. So our belief is the one that has the maximum entropy, among those consistent with our prior information. If we now add the information that some model is true, this generally invalidate our previous belief, making the new maximum-entropy belief one of lower entropy. The reduction in entropy is the amount of information you gain by learning the model. In a way, this is a cost we pay for "narrowing&... (read more)

0Decius9yAt what point does the decision "This is true" diverge from the observation "There is very strong evidence for this", other than in cases where the model is accepted as true despite a lack of strong evidence? I'm not discussing the case where a model goes from unknown to known- how does deciding to believe a model give you more information than knowing what the model is and the reason for the model. To better model an actual agent, one could replace all of the knowledge about why the model is true with the value of the strength of the supporting knowledge. How does deciding that things always fall down give you more information than observing things fall down?
Conservation of Expected Evidence

You're not really wrong. The thing is that "Occam's razor" is a conceptual principle, not one mathematically defined law. A certain (subjectively very appealing) formulation of it does follow from Bayesianism.

P(AB model) \propto P(AB are correct) and P(A model) \propto P(A is correct). Then P(AB model) <= P(A model).

Your math is a bit off, but I understand what you mean. If we have two sets of models, with no prior information to discriminate between their members, then the prior gives less probability to each model in the larger set than ... (read more)

0aspera9yCrystal clear. Sorry to distract from the point.
0Decius9yHow does deciding one model is true give you more information? Did you mean "If a model allows you to make more predictions about future observations, then it is a priori less likely?"
Internal Availability

The ease with which images, events and concepts come to mind is correlated with how frequently they have been observed, which in turn is correlated with how likely they are to happen again.

Yes, and I was trying to make this description one level more concrete.

Things never happen the exact same way twice. The way that past observations are correlated with what may happen again is complicated - in a way, that's exactly what "concepts" capture.

So we don't just recall something that happened and predict that it will happen again. Rather, we compos... (read more)

Internal Availability

Take for example your analysis of the poker hand I partially described. You give 3 possibilities for what the truth of it may be. Are there any other possibilities? Maybe the player is bluffing to gain the reputation of a bluffer? Maybe she mistook a 4 for an ace (it happened to me once...)? Maybe aliens hijacked her brain?

It would be impossible to enumerate or notice all the possibilities, but fortunately we don't have to. We make only the most likely and important ones available.

Internal Availability

I was trying to give a specific reason that the availability heuristic is there: it's coupled with another mechanism that actually generates the availability; and then to say a few things about this other mechanism.

Does anyone have specific advice on how I could convey this better?

0timtyler9yIt seems obvious why the availability heuristic is there. The ease with which images, events and concepts come to mind is correlated with how frequently they have been observed, which in turn is correlated with how likely they are to happen again. So, the heuristic is a reasonably-good one which just happens to have some associated false positives.
0faul_sname9yI'm still unsure of what you're actually saying. Perhaps you're talking about some sort of a "plausibility heuristic", where we look for instances of something in our model of the world, not just our experiences. That seems trivial, but that's not necessarily a bad thing (I would prefer to see more stuff here that seems really obvious to people, because those few times it's not obvious to everyone tend to be very valuable). If you're saying something else, I'm still not getting it.
The Bayesian Agent

Imagine a bowl of jellybeans. [...]

Allow me to suggest a simpler thought experiment, that hopefully captures the essence of yours, and shows why your interpretation (of the correct math) is incorrect.

There are 100 recording studios, each recording each day with probability 0.5. Everybody knows that.

There's a red light outside each studio to signal that a session is taking place that day, except for one rogue studio, where the signal is reversed, being off when there's a session and on when there isn't. Only persons B and C know that.

A, B and C are stand... (read more)

3Kindly9yNo matter what, someone is still updating in the wrong direction, even if we don't know who it is.

To anyone thinking this is not random, with 42 votes in:

• The p-value is 0.895 (this is the probability of seeing at least this much non-randomness, assuming a uniform distribution)

• The entropy is 2.302bits instead of log(5) = 2.322bits, for 0.02bits KL-distance (this is the number of bits you lose for encoding one of these votes as if it was random)

If you think you see a pattern here, you should either see a doctor or a statistician.

0gwern9yWell, it's worth noting people seem to be trainable to choose randomly: http://dl.dropbox.com/u/85192141/1986-neuringer.pdf [http://dl.dropbox.com/u/85192141/1986-neuringer.pdf] Apropos of the PRNG discussion in http://blog.yunwilliamyu.net/2011/08/14/mindhack-mental-math-pseudo-random-number-generators/ [http://blog.yunwilliamyu.net/2011/08/14/mindhack-mental-math-pseudo-random-number-generators/] for which I wrote some flashcards: http://pastebin.com/CKif0fEf [http://pastebin.com/CKif0fEf]
1[anonymous]9yLooks like we're better at randomness than the rest of the population. If I asked random people for a random number from 1 to 10, I wouldn't be surprised to see substantially less than 3.322 bits of entropy per number (e.g., many more than 10% of the people choosing 7).
4DanArmak9yI wish I could see a doctor-statistician. Or at least a doctor who understood statistics.
The Bayesian Agent

It is perfectly legal under the bayes to learn nothing from your observations.

Right, in degenerate cases, when there's nothing to be learned, the two extremes of learning nothing and everything coincide.

Or learn in the wrong direction, or sideways, or whatever.

To the extent that I understand your navigational metaphor, I disagree with this statement. Would you kindly explain?

There is no unique "Bayesian belief".

If you mean to say that there's no unique justifiable prior, I agree. The prior in our setting is basically what you assume yo... (read more)

The Bayesian Agent

Everything you say is essentially true.

As the designer of the agent, will you be explicitly providing it with that information in some future instalment?

Technically, we don't need to provide the agent with p and sigma explicitly. We use these parameters when we build the agent's memory update scheme, but the agent is not necessarily "aware" of the values of the parameters from inside the algorithm.

Let's take for example an autonomous rover on Mars. The gravity on Mars is known at the time of design, so the rover's software, and even hardware,... (read more)

The Bayesian Agent

If you're a devoted Bayesian, you probably know how to update on evidence, and even how to do so repeatedly on a sequence of observations. What you may not know is how to update in a changing world. Here's how:

$B_{t+1}(W_{t+1}\$%3d\Pr(W_{t+1}|O1,\ldots,O{t+1})%3d\frac{\sigma(O{t+1}|W{t+1})\cdot\Pr(W_{t+1}|O_1,\ldots,O_t)}{\sumw\sigma(O{t+1}|w)\cdot\Pr(w|O_1,\ldots,O_t)})

As usual with Bayes' theorem, we only need to calculate the numerator for different values of $W_{t+1}$, and the denominator will normalize them to sum to 1, as probabilities do. We know $\sigma$ as part of the dynamics ... (read more)

5RichardKennaway9yI'm not seeing how this lets the agent update itself. The formula requires knowledge of sigma, pi, and p. (BTW, could someone add to the comment help text instructions for embedding Latex?) pi is part of the agent but sigma and p are not. You say But all the agent knows, as you've described it so far, is the sequence of observations. In fact, it's stretching it to say that we know sigma or p -- we have just given these names to them. sigma is a complete description of how the world state determines what the agent senses, and p is a complete description of how the agent's actions affect the world. As the designer of the agent, will you be explicitly providing it with that information in some future instalment?
Argument Screens Off Authority

p(H|E1,E2) [...] is simply not something you can calculate in probability theory from the information given [i.e. p(H|E1) and p(H|E2)].

Jaynes would disapprove.

You continue to give more information, namely that p(H|E1,E2) = p(H|E1). Thanks, that reduces our uncertainty about p(H|E1,E2).

But we are hardly helpless without it. Whatever happened to the Maximum Entropy Principle? Incidentally, the maximum entropy distribution (given the initial information) does have E1 and E2 independent. If your intuition says this before having more information, it is good... (read more)

Reinforcement, Preference and Utility

Clearly you have some password I'm supposed to guess.

This post is not preliminary. It's supposed to be interesting in itself. If it's not, then I'm doing something wrong, and would appreciate constructive criticism.

6kjmiller9yYou have presented a very clear and very general description of the Reinforcement Learning problem. I am excited to read future posts that are similarly clear and general and describe various solutions to RL. I'm imagining the kinds of things that can be found in the standard introduction [http://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html], and hoping for a nonstandard perspective that might deepen my understanding. Perhaps this is what Richard is waiting for as well?
3RichardKennaway9yOnly the one in the title of these posts: "reinforcement learning". Both words have indeed appeared in this post, but I don't see you talking about reinforcement, learning, or reinforcement learning yet. I can't say any more than the above. I don't see the main act on stage yet.
Reinforcement Learning: A Non-Standard Introduction (Part 2)

That's an excellent point. Of course one cannot introduce RL without talking about the reward signal, and I've never intended to.

To me, however, the defining feature of RL is the structure of the solution space, described in this post. To you, it's the existence of a reward signal. I'm not sure that debating this difference of opinion is the best use of our time at this point. I do hope to share my reasons in future posts, if only because they should be interesting in themselves.

As for your last point: RL is indeed a very general setting, and classical planning can easily be formulated in RL terms.

Reinforcement Learning: A Non-Standard Introduction (Part 2)

I'm not sure why you say this.

Please remember that this introduction is non-standard, so you may need to be an expert on standard RL to see the connection. And while some parts are not in place yet, this post does introduce what I consider to be the most important part of the setting of RL.

So I hope we're not arguing over definitions here. If you expand on your meaning of the term, I may be able to help you see the connection. Or we may possibly find that we use the same term for different things altogether.

I should also explain why I'm giving a non-standa... (read more)

1RichardKennaway9yBut since we are not, we cannot. Well, there you are. The setting. Not actual RL. So that's two purely preliminary posts so far. When does the main act come on -- the R and the L?
1Johnicholas9yAs I understand it, you're dividing the agent from the world; once you introduce a reward signal, you'll be able to call it reinforcement learning. However, until you introduce a reward signal, you're not doing specifically reinforcement learning - everything applies just as well to any other kind of agent, such as a classical planner.
Reinforcement Learning: A Non-Standard Introduction (Part 1)

I internally debated this question myself. Ideally, I'd completely agree with you. But I needed the shorter publishing and feedback cycle for a number of reasons. Sorry, but a longer one may not have happened at all.

Edit: future readers will have the benefit of a link to part 2

Reinforcement Learning: A Non-Standard Introduction (Part 1)

In the model there's the distribution p, which determines how the world is changing. In the chess example this would include: a) how the agent's action changes the state of the game + b) some distribution we assume (but which we may or may not actually know) about the opponent's action and the resulting state of the game. In a physics example, p should include the relevant laws of physics, together with constants which tell the rate (and manner) in which the world is changing. Any changing parameters should be part of the state.

It seems that you're saying ... (read more)

Reinforcement Learning: A Non-Standard Introduction (Part 1)

There's supposed to be some way to do so partially, if anyone knows what it is.

This should work in Markdown, but it seems broken :(

Edit: t̶e̶s̶t̶ Thanks, Vincent, it works!

7VincentYu9yI never found a way to do it using LW's implementation of Markdown, but I have successfully used this Unicode strikethrough tool [http://adamvarga.com/strike/] before (a̶n̶ ̶e̶x̶a̶m̶p̶l̶e̶).
0fubarobfusco9yThat's the "retract" button.
Reinforcement Learning: A Non-Standard Introduction (Part 1)

I'm not sure what you mean. It looks fine to me, and I can't find where to check / change such a setting.

Edit:

Very strange. Fixed, I hope.

Thanks!

[This comment is no longer endorsed by its author]Reply
0fubarobfusco9yIndeed ... looks normal now. When I posted the above, the "computed style" in Chrome's developer tools showed 26px and the lines were widely spaced. Now it shows 17px as with other posts. Funky.
The Perception-Action Cycle

You are expressing a number of misconceptions here. I may address some in future posts, but in short:

By the action of powering the electromagnet you are not increasing your information on the state of the world. You are increasing your information on the state of the coin, but through making it dependent on the state of the electromagnet which you already knew. This point is clearly worth a future post.

There is no "entropy in environment". Entropy is subjective to the viewer.

0private_messaging9yI think it is mostly a matter of definitions. I am not familiar with your terminology. Also, if I have an atom that is in unknown alignment, and I align it using magnetic field, then take away the resulting heat, then the entropy (number of states) of that subsystem decreases, and this is used to attain extremely low temperatures [http://en.wikipedia.org/wiki/Magnetic_refrigeration] . I am more familiar with the physical notion of entropy. edit: Also, after powering electromagnet, I know that the direction of coin and direction of electromagnet relate in a particular way, which I did not know before. At the same time, I have physically restricted the number of states that the environment can be in - the coin can not now be other way around. It's in this sense that entropy of environment (as seen on large scale) decreases . (and it is of course subjective, that the number of states that environment can be in, decreases. It does not decrease according to agent that already knows which way the coin is)
The Perception-Action Cycle

I realize now that an example would be helpful, and yours is a good one.

Any process can be described on different levels. The trick is to find a level of description that is useful. We make an explicit effort to model actions and observation so as to separate the two directions of information flow between the agent and the environment. Actions are purely "active" (no information is received by the agent) while observations are purely "passive" (no information is sent by the agent). We do this because these two aspects of the process hav... (read more)

0OrphanWilde9yThis would be true regardless of whether you engaged in any action at all, however. The passing of time since your last verification of a piece of information is that by which information is lost. I'm assuming this model is AI-related, so my responses are going to be in line with information modeling with that in mind. If this isn't accurate, let me know. I would, indeed, suggest time since last verification as the mechanism in your model for information contraction, rather than action; assigning a prior probability that your information will remain accurate does a good job of completing the model. Imagine memorizing a room, closing your eyes, and firing a canon into the room. Contemporaneous to your action, your information is still valid. Shortly thereafter, it ceases to be in a rather dramatic way. Importantly for your model, I think, this is so regardless of whether you fire the canon, or another agent does. If it's a soundproof room, and you close the door with another agent inside, your information about the state of the room can contract quite violently through no action of your own.
The Perception-Action Cycle

Excellent point. It will be a few posts (if the audience is interested) before I can answer you in a way that is both intuitive and fully convincing.

The technical answer is that the belief update caused by an action is deterministically contractive. It never increases the amount of information.

A more intuitive answer (but perhaps not yet convincing) is that, proximally, your action of asking your friend did not change the location of your laptop, only your friend's mental state. And the effect it had on your friend's mental state is that you are now less s... (read more)

0private_messaging9yWhat is 'amount of information' ? If I do not know if coin is heads or tails, then I have 0 bits of information about state of the coin, if I open my eyes and see it is heads, I have 1 bit. The information is in narrowing of the possibilities. That is conventional meaning. edit: though of course the information is not increased until next perception - perhaps that is what you meant? edit: still, there is a counter example - you can have axially magnetized coin, and electromagnet that can make the coin flip to heads up when its powered. You initially don't know which way the coin is up, but if the action is to magnetize the electromagnet, you will have the coin be heads up. (Of course the overall entropy of world still did go up, but mostly in form of heat). One could say that it doesn't increase knowledge of environment, but decreases the entropy in environment.
-1OrphanWilde9yTo pick a trivial case: A blind person with acute hearing taps a cane on the floor in order to ascertain, from echoes, the relative positions of nearby objects. The issue is that "action" and "observation" can be entangled; your description of observation makes it into a passive process, ignoring the role of activity in observation. "Step one of my plan: Figure out where the table is so I don't run into it." Which is to say, your pattern is overly rigid. You might argue that the tapping of the cane is itself an observation, in which case you'd also have to treat walking into a room to see what's in it as an observation; the former removes no information, but the latter reduces your certainty of the positions of objects in the room you've just left, meaning either actions can generate information, or observations can reduce it. You could preserve the case that actions cannot generate information if you instead treat hearing the echoes as a secondary observation, but this still leaves you with the case that an action did not, in fact, eliminate any information.
3Oscar_Cunningham9yI'm interested.
Mutual Information, and Density in Thingspace

Having a word [...] is a more compact code precisely in those cases where we can infer some of those properties from the other properties. (With the exception perhaps of very primitive words, like "red" [...]).

Remember that mutual information is symmetric. If some things have the property of being red, then "red" has the property of being a property of those things. Saying "blood is red" is really saying "remember that visual experience that you get when you look at certain roses, apples, peppers, lipsticks and English... (read more)

My Wild and Reckless Youth

The Harsanyi paper is very enlightening, but he's not really arguing that people have shared priors. Rather, he's making the following points (section 14):

• It is worthwhile for an agent to analyze the game as if all agents have the same prior, because it simplifies the analysis. In particular, the game (from that agent's point of view) then becomes equivalent to a Bayesian complete-information game with private observations.

• The same-prior assumption is less restrictive than it may seem, because agents can still have private observations.

• A wide family

My Wild and Reckless Youth

I'm aware of this result. It specifically requires the two Beyesians to have the same prior. My point is exactly that this doesn't have to be the case, and in reality is sometimes not the case.

EDIT: The original paper by Aumann references a paper by Harsanyi which supposedly addresses my point. Aumann himself is careful in interpreting his result as supporting my point (since evidently there are people who disagree despite trusting each other). I'll report here my understanding of the Harsanyi paper once I get past the paywall.

4royf9yThe Harsanyi paper is very enlightening, but he's not really arguing that people have shared priors. Rather, he's making the following points (section 14): * It is worthwhile for an agent to analyze the game as if all agents have the same prior, because it simplifies the analysis. In particular, the game (from that agent's point of view) then becomes equivalent to a Bayesian complete-information game with private observations. * The same-prior assumption is less restrictive than it may seem, because agents can still have private observations. * A wide family of hypothetical scenarios can be analyzed as if all agents have the same prior. Other scenarios can be easily approximated by a member of this family (though the quality of the approximation is not studied). All of this is mathematically very pleasing, but it doesn't change my point. That's mainly because in the context of the Harsanyi paper "prior" means before any observation, and in the context of this post "prior" means before the shared observation (but possibly after private observations).
My Wild and Reckless Youth

Traditional Rationalists can agree to disagree. Traditional Rationality doesn't have the ideal that thinking is an exact art in which there is only one correct probability estimate given the evidence.

This is also true of Bayesians. The probability estimate given the evidence is a property of the map, not the territory (hence "estimate"). One correct posterior implies one correct prior. What is this "Ultimate Prior"? There isn't one.

Possibly, you meant that there's one correct posterior given the evidence and the prior. That's correc... (read more)

0beoShaffer9yhttp://wiki.lesswrong.com/wiki/Aumann%27s_agreement_theorem [http://wiki.lesswrong.com/wiki/Aumann%27s_agreement_theorem]
Fake Causality

A GAI with the utility of burning itself? I don't think that's viable, no.

What do you mean by "viable"?

Intelligence is expensive. More intelligence costs more to obtain and maintain. But the sentiment around here (and this time I agree) seems to be that intelligence "scales", i.e. that it doesn't suffer from diminishing returns in the "middle world" like most other things; hence the singularity.

For that to be true, more intelligence also has to be more rewarding. But not just in the sense of asymptotically approaching op... (read more)

Fake Causality

Not at all. If you insist, let's take it from the top:

• I wanted to convey my reasoning, let's call it R.

• I quoted a claim of the form "because P is true, Q is true", where R is essentially "if P then Q". This was a rhetorical device, to help me convey what R is.

• I indicated clearly that I don't know whether P or Q are true. Later I said that I suspect P is false.

• Note that my reasoning is, in principle, falsifiable: if P is true and Q is false, then R must be false.

• While Q may be relatively easy to check, I think P is not.

• I expe

0Ronny9yI don't want to revise my objection, because it's not really a material implication that you're using. You're using probabilistic reasoning in your argument,i.e., pointing out certain pressures that exist, which rule out certain ways that people could be getting smarter, and therefor increases our probability that people are not getting smarter. But if people are in fact getting smarter, this reasoning is either too confident in the pressures, or is using far from bayesian updating. Either way, I feel like we took up too much space already. If you would like to continue, I would love to do so in a private message.
Fake Causality

I'll try to remember that, if only for the reason that some people don't seem to understand contexts in which the truth value of a statement is unimportant.

0Ronny9yand You see no problem here?
Fake Causality

a GAI with [overwriting its own code with an arbitrary value] as its only goal, for example, why would that be impossible? An AI doesn't need to value survival.

A GAI with the utility of burning itself? I don't think that's viable, no.

I'd be interested in the conclusions derived about "typical" intelligences and the "forbidden actions", but I don't see how you have derived them.

At the moment it's little more than professional intuition. We also lack some necessary shared terminology. Let's leave it at that until and unless someon... (read more)

0CuSithBell9yWhat do you mean by "viable"? You think it is impossible due to Godelian concerns for there to be an intelligence that wishes to die? As a curiosity, this sort of intelligence came up in a discussion I was having on LW recently. Someone said "why would an AI try to maximize its original utility function, instead of switching to a different / easier function?", to which I responded "why is that the precise level at which the AI would operate, rather than either actually maximizing its utility function or deciding to hell with the whole utility thing and valuing suicide rather than maximizing functions (because it's easy)". But anyway it can't be that Godelian reasons prevent intelligences from wanting to burn themselves, because people have burned themselves. Fair enough, though for what it's worth I have a fair background in mathematics, theoretical CS, and the like. I meant that this was a broad definition of the qualitative restrictions to human self-modification, to the extent that it would be basically impossible for something to have qualitatively different restrictions. Why not? Though of course it may turn out that AI is best programmed on something unlike our current computer technology.