After reading this, I don't feel like I have a better understanding of how much rent you believe we should be able to get from our beliefs. I feel like what you've written is a really, really good introduction, with no argument or conclusion to go along with it.
hmmm. Thanks for feedback.
Maybe I've fallen prey to the illusion of transparency. The point I'm trying to articulate is just how far it may be possible for a human to take their beliefs towards practicality. I havn't actually developed this idea very far, I posted it to help me articulate where we should try to explore and get some discussion on that.
You're right I could have taken it further. I was thinking I should get into how we act almost exclusively through causal models and are quite good at it, and then relate that to the subject, but it would have gotten long. Maybe another post about that?
I'll reread a few times and see if I can see where improvements should go.
If you have any specific ideas, say em.
Well I think you said it yourself:
I havn't actually developed this idea very far
Maybe do so, and then you can make a post on it. ;-)
Edit: Also, welcome to LW!
The post is in the discussion section, yes? In that context, not having a clear idea of how much rent the author thinks we should pay seems good. This is definitely a question worth discussing and bringing up. In that context, I've voted it up. It also raises interesting issues about algorithmic and processing restrictions on how beliefs pay rent.
So I think this is a very interesting post which may not be beng appreciated as much because the main example is a fictonal one (albeit from a very excellent pair of books). I'd like to give a different example stolen from research that my former college roomate is now doing: We can predict the individual behavior of a water molecule to a very rough approximation from first principles. But as soon as one has more than one molecule predicting very basic questions like "what should the boiling and cooling temperatures be?" "what should the index of refraction be?" "what sort of crystals should I expect to form when I cool it?" are computationally infeasible. So a lot of physicists are working on questions like this but essentially trying to simplify the computations and figure out which approximatons you can get away with and which don't quite work.
In this context, this is an example where to use the sort of analogy in the post, knowing the name of the substance really doesn't pay rent very directly since the computations are just too arduous. Quantum mechanics can pay rent in other ways, but using it to pay rent for this purpose seems to be difficult.
Also note that being able to pay rent to be able to predict something is still not the same as being able to control it. Kvothe might be able to know where every molecule of air is and be able to compute where they are going (ignoring for a moment issues of fundamental uncertainty due to quantum mechanical issues), but that doesn't mean one can figure out an action that will make the air do what one wants. To do that requires not just computing a single path of events but likely requires computing many paths and figuring out which one one wants. Similarly, the evil oracle has a much tougher job computationally than a regular oracle.
Also note that being able to pay rent to be able to predict something is still not the same as being able to control it. Kvothe might be able to know where every molecule of air is and be able to compute where they are going (ignoring for a moment issues of fundamental uncertainty due to quantum mechanical issues), but that doesn't mean one can figure out an action that will make the air do what one wants. To do that requires not just computing a single path of events but likely requires computing many paths and figuring out which one one wants. Similarly, the evil oracle has a much tougher job computationally than a regular oracle.
Conversely, in many cases, e.g., simple chaotic systems, it is easier to control something then to predict what will happen if you don't intervene.
good point on the control. I did try to emphasize the prediction aspect with the sword tree example instead of like the felurian or ambrose scenes, which are impossible.
But some belief algorithms can pay rent in "superpowers" I think. There are things much easier to control than awful nonlinear fluid dynamics.
Edit: also, do you think in possible future versions of this concept I should avoid the fictional examples? The point of bringing up fictional examples was that it is actually a really good example of what I'm talking about and some people are familiar with it.
Care to name a few that you think can be "superpowers"?
I have no objection to using fictional examples, especially when it illustrates a point perfectly. Just be sure to buffer it with real-world examples. You did a good job mentioning occupations that can viscerally feel physics. I did find it rather surprising that you didn't mention basketball players' understanding of parabolas. That was the first example I thought of right after reading the word "machinist".
superpowers is a bit of an exaggeration. I don't know many we can get gains in, but having a more intuitive than mathematical understanding of everyday physics should lead to some interesting abilities. For example, derivable from solid mechanics equations is the fact that a shallow cut on the back of a wood beam will allow you to break it with maybe 1/3 of the force. More brittle materials work even better. I'm sure there are others. I'll keep thinking.
The basketball example is a really good one, thanks.
For example, derivable from solid mechanics equations is the fact that a shallow cut on the back of a wood beam will allow you to break it with maybe 1/3 of the force. More brittle materials work even better.
Holy crap! That's awesome!
I've taken a metal smithing class for several semesters, and noticed that an understanding of the physics involved makes one much better at producing the results desired. The teacher has an excellent balance of knowledge and feeling-about-the-knowledge. It is an admirable trait, feeling what you know.
Edit: also, do you think in possible future versions of this concept I should avoid the fictional examples? The point of bringing up fictional examples was that it is actually a really good example of what I'm talking about and some people are familiar with it.
Can you make the point with non-fictional examples? If not, then it seems like generalizing from fictional evidence. A lot of what Kvothe can do is simply intractable, and so using him as an example seems like magical thinking rather than someone familiar with real-world optimization models.
It wasn't supposed to be evidence. It was an "alice and bob"-type illustration story that happens to exist in fiction.
I'll use better examples in future tho.
Thanks a lot.
Is there a way to get markdown (other than generating and posting the html)?
As an aside, I don't believe in malicious superintelligences. They are even more unlikely than automatically Friendly ones. An unFriendlyAI is dangerous as a side effect of its abilities and goals, not because it is, or is even likely to be, malicious.
I would assign a malicious superintelligence a higher probability than would pure entropy over the space of superintelligences due to the chance of something broken coming out of military research. I would assign this a relatively low likelihood. I am not certain whether I would assign it a higher or lower likelihood than "automatically Friendly ones" - it depends on what you mean by that. I would assign it a higher probability than an AI built without any thought to friendliness being friendly, given that it was built with thought to maliciousness, and there are perhaps a broader range of behaviors we might label "malicious".
By an "automatically Friendly AI" I simply meant one that was Friendly without explicit programming for friendliness. I think that would be more likely than a malicious AI because there are good, rational reasons to be "friendly" (benefits from trade and so on) in the absence of reasons not to be. I can see no rational reason to be malicious - humans that are malicious are usually so for reasons (sadism, revenge, and so on) that I can't see someone programming into an AI.
good, rational reasons to be "friendly" (benefits from trade and so on)
Is that why humans have been so friendly to the non-human inhabitants of lands we want to develop? Humans are likely to have almost nothing to offer an advanced super-intelligence, just as an ant hill has almost nothing to offer me (except as an opportunity to destroy it and plant more grass).
There are good, rational reasons to be friendly in the short term.
The rational reason to be unfriendly in the long term is that sufficiently advanced optimizing processes are powerful, and outcomes that maximize the utility of one agent are not likely to also maximize the utility of other agents with different goals.
there are good, rational reasons to be "friendly" (benefits from trade and so on)
that is a very dangerous statement. A superintelligent AI doesn't care about you one bit. If it is (unlikely) in the situation where it needs something from you that it cannot take with violence, it may offer to trade, but I would give high confidence of it shooting you in the back and taking the goods the moment you let your guard down.
In that case the word arguably can't be applied to people either, as Eliezer pointed out in this post. The only time people actively "desire to harm another" is when (they believe that) they are punishing the other according to what we would call TDT/UDT. Of course, the same motive applies to an AI even one whose terminal goals are indifferent to humans.
In that case the word arguably can't be applied to people either, as Eliezer pointed out in this post.
Yes, it can. People really are malicious sometimes. That we are biased to attribute malice to enemies even when none is present does not rule out malice existing in people.
The only time people actively "desire to harm another" is when (they believe that) they are punishing the other according to what we would call TDT/UDT.
That just isn't true. And even if it were the fact that a TDT or UDT agent may do a similar action does not mean that a person is not feeling malice.
Of course, the same motive applies to an AI even one whose terminal goals are indifferent to humans.
Yes, not that they would need to apply that reasoning with humans. You don't need to punish people when you can just consume them as resources.
Whether they are real or not, malicious things are common in fantasy. I find breaking the laws of thermodynamics or anti-reductionism to be much more immersion-busting.
I disagree.
Both friendly and explicitly malicious AIs need to understand what sentience is.
In addition, a malicious AI needs to know some means of torturing them.
In addition, a friendly AI needs to know how to identify and preserve existing sentient beings, and all human values (highly nontrivial).
That'd be a good argument that explicitly malicious AI is technically simpler than Friendly AI, but technical complexity isn't the only constraint on the likelihood of AI of a particular type arising. I'd consider it extremely unlikely that any development team would choose to inculcate a generally malicious value system in their charges; the AI research community is, fortunately, not made up of Bond villains. It doesn't even work as a mutually-assured-destruction ploy, since the threat isn't widely recognized.
Situational malice seems more plausible (in military applications, for example), but I'd call that a special case of ordinary unFriendliness.
I could easily see military application + bug in the safeguards => malicious AI.
Not as likely as ordinary unfriendliness, I think, but certainly plausible.
I think my reading of this article was impaired because you didn't mention which book. Instead of thinking about what you were writing, I was at least partially trying to pattern-match it to books I've read, or figuring out how to find it, though admittedly googling "kvothe" works. I don't think hiding the name buys you anything.
oh ok. I didn't wan't it to be too spoilery, but then I went and mentioned kvothe. Sorry.
There are limits even with omniscience - instabilities resolved by quantum mechanics will go both ways, predictably - but you aren't going to observe both ways. I wouldn't be surprised if our brains had sufficient instability that quantum mechanics forbids a strong assertion of macro-state after an interval as short as seconds. I'd be surprised if the prediction horizon were milliseconds or hours.
So… we can't demand that much rent, anyway.
And of course, making beliefs pay rent doesn't really necessarily make sense in this context. How much use can I get out of knowing that when I was 11 I doodled a pulley system for a platformer game? I can't remember much else about it, and even if I had it there's no chance I'd build the game or even offer it to someone who was building a game. Yet, I believe that to be true, because I remember it. I take it you're not telling me I shouldn't believe it just because it's not useful, yet that is exactly the subject of making beliefs pay rent (edited to clarify - whether or not you should believe it, not whether it's useful).
Being useful is a different issue - it feeds into the question of which beliefs you take an effort to discriminate the truth of. If I came to doubt whether I had in fact designed that pulley system, I would spend approximately 0.00 J of effort attempting to determine whether it's true or not, even though it's possible I have the notebook around somewhere.
You're right about omniscience. Maybe I shouldn't have mentioned that?
Your pulley system belief isn't really paying rent by any definition. I don't see how demanding algorithms from your beliefs changes that.
The point wasn't that we should not believe things that don't pay rent (that's quite hard). It was that we should not stop at simply having a correct formulation of the behavior of something. We should take it as far into the realm of useful algorithms as possible before we say "ok, this belief is paying rent".
The pulley system belief does pay its rent quite easily - it explains my having a memory of doing so. Any other theory requires further intervention which contains a lot more information than the theory that I simply did it and happened to remember doing it.
That's what rent is about. It's not a rent of utility, but of credence.
It seems to me like the question "How much Rent?" is very easily answered. That's what Bayes Rule does - it tells us exactly how much weight we should give evidence. I'm not sure the question can be answered any more specifically than the generalized formula of Bayes Rule without going into a specific example, for which you would have priors, likelihood ratios, etc.
The point of the post was that simply having the math is not enough. Knowing bayes rule plus say NS equations is mathematically equivalent to kvothe-style knowing the name of the wind, but thinking that's the whole story totally misses the point that you don't have an intuitive system 1 algorithm for predicting or manipulating the thing. The math is the core compressed truth of the matter, but you need to implement it in actual fast algorithms to actually get any use out of it.
Of course we can only talk about specific examples once we get to the level of algorithms, but I don't see that being a problem because you still have specific beliefs even without good algorithms.
The point is that if we focus more on getting our algorithms fast and intuitive, we can get a lot more mileage out of the same belief. Theres a difference between spending 2 hours calculating the utility of applying different forces to a skateboard and actually being able to do a kick-flip.
That doesn't make the NS equations wrong. True and fiction-level-utility-generating are not and should not be the same thing. Once you start asking them as separate questions, I think much of the problem of this post disappears.
The NS equations are not themselves wrong, naturally. However, a memorization of NS in a human brain is wrong in nearly the same sense that a copy of Microsoft Office compiled for x86 on an iPhone is wrong. Utility generation from correct beliefs relies on format as much as correctness.
By "wrong," I meant "false," "not true." It's interesting that the different meanings of the word "wrong" are similar to the categories interchanged in the post, though.
The main beef I have is that what you call "format" is not, in fact, mere format. It's realio trulio a different set of information. It's merely a repackaging of the statement "having different information matters," obscured by using new definitions. Rather than thinking the edgy repackaging is a new thing, it's more accurate and simpler to just appreciate the complexities of the original statement.
You're right that I meant more than "mere format". Let's taboo the word. And for that matter "wrong" too.
Usually when we think of true information or true belief we consider only the belief itself (the NS equations). Beliefs only exist in contexts (human brains, software implementations, even ink and paper), and some of those contexts are not adequate in themselves to take advantage of the belief. It's not a question of having different information. It's a question of having inadequate computational resources. Or in another way of looking at it, it's a question of having a capable, powerful computational context whose structure is not optimized for the problem at hand.
Right, but the NS equations are probably "wrong" enough to disqualify them as the "name of the wind". That is, they're continuum approximations to what are, at tiny scales, actually quantized substances governed by probabilistic laws. (MWI-QM fans, make the appropriate substitution for "probabilistic"; the point stands.) And it's pretty easy to get chaotic turbulence in air flowing over a tree.
NS is very accurate. The point at which it might break down is on extremely tiny scales that don't actually matter.
It also breaks down at sonic shockwakes, but not in a fatal way. NS divides by zero at sonic shockwaves and real air has a very short (but present) gradient. The macro predictions are identical beyond the precision of most measurements.
Also, NS is newtonian, so relativistic stuff breaks it.
I would not disqualify the NS based on a few innacuracies at extremes. It is the name of the wind for all practical purposes. You will not encounter the problems with it.
True. If we disregard the fact that humans take time to do computations, and disregard the information contained in restricting our solutions to e.g. fluid flow - reasonable since those things don't seem much like "names" - our best "name of the X" is the standard model and general relativity, where X is pretty much anything.
Right, from this we get odds at which we should take bets on our beliefs. OP is asking something a little more like: How can we find more wagers to accept?
Not quite wagers. Is tensing the muscles in just the right way to swing a machete to cut something a wager? In some sense yes, but we don't really think of it that way, and I don't think we will become more effective from thinking of it that way. We think of it using words like 'skill' and 'tool'. I think it is more effective to talk about how beliefs can be turned into skills than to discuss how skills relate to wagers.
Make beliefs pay rent. How much rent? Is it enough that they have some theoretical use in designing a GPS or predicting the cosmos? How much rent can actually be extracted from a belief?
In a certain fantasy series, there is a special knowledge of a thing, called the name of the thing, that gives one predictive and manipulative power over it. For example, the protagonist, a young
rationalistarcanist named Kvothe, learns the name of the wind and uses it to predict the movements of the leaves of a razor-sharp 'sword tree' well enough to walk through without getting cut.Another character, which we would recognize as a boxed malicious superintelligence, has the ability to predict everything. Simply talking to it allows it to manipulate your future to its twisted ends.
At first these seem like the usual impossible fantasy magic, but why impossible? If a path exists, a good predictive model should find it.
There's nothing that says the map can't match the territory to arbitrary precision. There's nothing that says beliefs have to just sit passively until they are brought up at a dinner party. But how much rent can we extract?
We are not omniscient superintelligences, so the second power is closed to us for now. The first also seems off-limits, but consider that we do know the name of the wind. Our name of the wind and Kvothe's name of the wind are mathematically equivalent (in that the motion of the sword tree could be predicted by simulation of the wind using the NS equation). So why is it that Kvothe can walk through the leaves of the sword tree, but you, even knowing the NS equations as facts in your map, can not?
Optimization. Algorithmization. Kvothe's name of the wind is optimised and algorithmised for practical use. Your name of the wind is sitting in your cache as a dumb fact ready to guess the password for "how does wind work". Kvothe is reeling in rent utilons while you congradulate yourself for having correct beliefs.
So to collect rent from your beliefs, it is not enough to simply know some fact about the world. It has to be implemented by a good algorthim on the intuitive level. You have to be able to act and see through the wind the way a machinist can act through a lathe and a woodsman can see through footprints in the dirt. The way a surfer or skater can act through his board and see through the subtle vibrations and accelerations.
I don't know if we can reach the level of intuitive causal modeling of the wind that Kvothe has. Maybe it's too hard to integrate such abstract models into system 1. Fluid dynamics is notoriously difficult even for computers. I do know that it's not enough to memorise the differential equations. You can get a lot further than that.
So how much rent can you get from your beliefs? A good rent-paying belief should feel like an extension of your body; You should be able to see and act through your belief like it's another eye or arm. When thinking about how much rent can be extracted from a belief about something, think about what Kvothe would be able to do if he knew its true name.