In my experience teachers tend to only give examples of typical members of a category. I wish they'd also give examples along the category border, both positive and negative. Something like: "this seems to have nothing to do with quadratic equations, but it actually does, this is why" and "this problem looks like it can be solved using quadratic equations but this is misleading because XYZ". This is obvious in subjects like geography, (when you want to describe where China is, don't give a bunch of points around Beijing as examples, but instead draw the border and maybe tell about ongoing territorial conflicts) but for some reason less obvious in concept-heavy subjects like mathematics.
Another point on my wishlist: create sufficient room for ambition. Give bonus points for optional but hard exercises. Tell about some problems that even world's top experts don't know how to solve.
Thank you very much for posting this! I've been thinking about this topic for a while now and feel like this is criminally overlooked. There are so many resources on how to teach other people effectively, but virtually none on how to learn things effectively from other people (not just from textbooks). Yet we are often surrounded by people who know something that we currently don't and who might not know much about teaching or how to explain things well. Knowing what questions to ask and how to ask them makes these people into great teachers - while you reap the benefits! - this feels like a superpower.
While I agree that the algorithm might output 5, I don't share the intuition that it's something that wasn't 'supposed' to happen, so I'm not sure what problem it was meant to demonstrate. I thought of a few ways to interpret it, but I'm not sure which one, if any, was the intended interpretation:
a) The algorithm is defined to compute argmax, but it doesn't output argmax because of false antecedents.
- but I would say that it's not actually defined to compute argmax, therefore the fact that it doesn't output argmax is not a problem.
b) Regardless of the output, the algorithm uses reasoning from false antecedents, which seems nonsensical from the perspective of someone who uses intuitive conditionals, which impedes its reasoning.
- it may indeed seem nonsensical, but if 'seeming nonsensical' doesn't actually impede its ability to select actions wich highest utility (when it's actually defined to compute argmax), then I would say that it's also not a problem. Furthermore, wouldn't MUDT be perfectly satisfied with the tuple ? It also uses 'nonsensical' reasoning 'A()=5 => U()=0' but still outputs action with highest utility.
c) Even when the use of false antecedents doesn't impede its reasoning, the way it arrives at its conclusions is counterintuitive to humans, which means that we're more likely to make a catastrophic mistake when reasoning about how the agent reasons.
- Maybe? I don't have access to other people's intuitions, but when I read the example, I didn't have any intuitive feeling of what the algorithm would do, so instead I just calculated all assignments , eliminated all inconsistent ones and proceeded from there. And this issue wouldn't be unique to false antecedents, there are other perfectly valid pieces of logic that might nonetheless seem counterintuitive to humans, for example the puzzle with islanders and blue eyes.
Yet, we reason informally from false antecedents all the time, EG thinking about what would happen if
When I try to examine my own reasoning, I find that when I do so, I'm just selectively blind to certain details and so don't notice any problems. For example: suppose the environment calculates "U=10 if action = A; U=0 if action = B" and I, being a utility maximizer, am deciding between actions A and B. Then I might imagine something like "I chose A and got 10 utils", and "I chose B and got 0 utils" - ergo, I should choose A.
But actually, if I had thought deeper about the second case, I would also think "hm, because I'm determined to choose the action with highest reward I would not choose B. And yet I chose B. This is logically impossible! OH NO THIS TIMELINE IS INCONSISTENT!" - so I couldn't actually coherently reason about what could happen if I chose B. And yet, I would still be left with the only consistent timeline where I choose A, which I would promptly follow, and get my maximum of 10 utils.
The problem is also "solved" if the agent thinks only about the environment, ignoring its knowledge about its own source code.
The idea with reversing the outputs and taking the assignment that is valid for both versions of the algorithm seemed to me to be closer to the notion "but what would actually happen if you actually acted differently", i.e. avoiding seemingly nonsensical reasoning while preserving self-reflection. But I'm not sure when, if ever, this principle can be generalized.
I don't quite follow why 5/10 example presents a problem.
Conditionals with false antecedents seem nonsensical from the perspective of natural language, but why is this a problem for the formal agent? Since the algorithm as presented doesn't actually try to maximize utility, everything seems to be alright. In particular, there are 4 valid assignments: , , ,
The algorithm doesn't try to select an assignment with largest , but rather just outputs if there's a valid assignment with , and otherwise. Only fulfills the condition, so it outputs . and also seem nonsensical because of false antecedents but with attached utility - would that be a problem too?
For this particular problem, you could get rid of assignments with nonsensical values by also considering an algorithm with reversed outputs and then taking the intersection of valid assignments, since only satisfies both algorithms.
Could someone explain why this doesn't degenerate into an entirely circular concept when we postulate a stronger compiler; or why it doesn't become entirely dependent on the choice of the compiler?
Now we have a set of sequences that we'd like to encode: S = {, 0, 1, 00, 01, ... }, a set of sequences that are interpreted by the compiler as programs: P = {, 0, 1, 00, 01, ... } and the compiler which is a bijection from P to S. It better not turn out to be the identity function.. And that's with the best possible compiler. If we postulate a reasonable but much weaker compiler then the programs that encode the sequences become on average longer than the sequences themselves!
The only way out of this that I see is to weight elements of S by their frequencies in our universe and/or by how much we care about them, and then let the compiler be a function that minimizes this frequency-importance score. In fact, this compiler starts looking more and more like an encoder (?!). The difficult part then seems to me to be the choice of the optimal encoder, and not the Solomonoff induction itself.
Edit: Of course, when there's a 1 to 1 mapping, then selecting the shortest program is trivial. So in a way, if we make the Solomonoff induction trivial then the only thing that's left is the choice of the compiler. But why isn't this still a problem with weaker, traditional compilers?
I thought of a slightly different exception for the use of "rational": when we talk about conclusions that someone else would draw from their experiences, which are different from ours. "It's rational for Truman Burbank to believe that he has a normal life."
Or if I had an extraordinary experience which I couldn't communicate with enough fidelity to you, then it might be rational for you not to believe me. Conversely, if you had the experience and tried to tell me, I might answer with "Based only on the information that I received from you, which is possibly different from what you meant to communicate, it's rational for me not to believe the conclusion." There I might want to highlight the issue with fidelity of communication as a possible explanation for the discrepancy (the alternative being, for example, that the conclusion is unwarranted even if the account of the event is true and compete).
Richard Feynman once said that if you really understand something in physics you should be able to explain it to your grandmother. I believed him.
Curiously enough, there is a recording of an interview with him where he argues almost exactly the opposite, namely that he can't explain something in sufficient detail to laypeople because of the long inferential distance.
It seems that the mistake that people commit is imagining the the second scenario is a choice between 0.34*24000 = 8160 and 0.33*27000 = 8910. Yes, if that was the case, then you could imagine a utility function that is approximately linear in the region 8160 to 8910, but sufficiently concave in the region 24000 to 27000 s.t. the difference between 8160 and 8910 feels greater than between 24000 and 27000... But that's not the actual scenario with which we are presented. We don't actually get to see 8160 or 8910. The slopes of the utility function in the first and second scenarios are identical.
"Oh, these silly economists are back at it again, asserting that my utility function ought to be linear, lest I'm irrational. Ugh, how annoying! I have to explain again, for the n-th time, that my function actually changes the slope in such a way that my intuitions make sense. So there!" <- No, that's not what they're saying! If you actually think this through carefully enough, you'll realize that there is no monotonically increasing utility function, no matter the shape, that justifies 1A > 1B and 2A < 2B simultaneously.
But is the Occam's Razor really circular? The hypothesis "there is no pattern" is strictly simpler than "there is this particular pattern", for any value of 'this particular'.. Occam's Razor may expect simplicity in the world, but it is not the simplest strategy itself.
Edit: I'm talking about the hypothesis itself, as a logic sequence of some kind, not that, which the hypothesis asserts. It asserts maxentropy - the most complex world.
>>> 20.05751