All of lsusr's Comments + Replies

There's a lot of stuff that scares me about that post.

Resolution Criteria

  • Suppose your counterparty bets on 200:1 odds. Suppose the odds of a LW poll getting trolling results are >0.5%. Then your counterparty loses all of their alpha on that alone (because an incorrect result costs them 200× more than it costs you).
  • "I reserve the right to appeal to the LW community to adjudicate resolution if I believe I am being stiffed." is too vague. If you don't specify exactly how you plan for the LW community to adjudicate resolution, then that's just asking fo
... (read more)
1RatsWrongAboutUAP5d
I will update the post tomorrow and add more detail to address the other concerns
7RatsWrongAboutUAP5d
I know I have no post history, and thus these are just words, but I claim to be a reasonable, rational person who (tries) to operates exclusively in good faith. I've been a lurker of LW and LW adjacent people for a few years now. I learned about lesswrong because I stumbled across eilizers work on decision theories and then subsequently got agi-safety-pilled. I considered myself a "standard materialist atheist" my entire adult life and most of my childhood. Most of your concerns seemed to ignore that the explanations have to ultimately trace back to explaining ufo cases or are otherwise very pedantic. Yes ancient civ's have ruins, yes dinosaurs might have been agrarian, but do either of those address uap cases today? I only win the bet if my counterparty thinks so (or LW does). I tried in good faith to try and cut at the seems of the two world models (all prosaic, not all prosaic) as best I could. I gave multiple lexical tests to make clear what kinds of things I have in mind. I have no intention of getting into nasty disagreements over resolution. But I agree it would be good to have the adjudication method be explicit, though I'm not sure how best to do that. In a world in which I win the bet, I figured I would be making a big post anyways laying out a bit more about me and some of this stuff. If prominent voices contest my win then I would stand down from collecting. I expect a world in which I win is also a world in which LW is pretty unanimous that I won. You're right, I forgot to mention the nontrivial post history in my post, an oversight on my part. That said I was only ever going to engage with people with established reputations because obviously. I reserved the right to choose who to bet with.

The financial terms aren't good enough to entice me. Besides that…

Pretty much all of your weird explanations are too vague. In particular "[s]ome other explanation that's of this level of 'very weird'" is voids the whole thing. It'd be fine for a blog post, but not as a prediction resolution criteria.

"I reserve the right to appeal to the LW community. [I will not abuse this right]" is too vague too. The LW community is not a monolithic entity. I think you need to specify exactly how you plan to appeal to the LW community.

Nope. That's a separate bet. I'd happily bet against that (given good enough terms to overcome friction), but that's still originating from Earth.

Yes, I am implying that it's the only recourse allowed. Doing otherwise exposes me to asymmetric litigation risk, due to the extreme asymmetry of the bet amounts. I believe reputation is a sufficient motivator, given how much effort I've spend accumulating reputation on this website.

I respect your offer, but I'd need much better terms from you than from lc. Lc is someone I've interacted with before, and with whom I have established a level of trust.

Some thoughts:

  • $50 to remember something for five years (and to assume a theoretical $10k liability) is too low a price. I value my attention higher than that.
  • I think your list of hypotheses needs some nitpicking, but it's fine as a rough draft.
  • If you have the right to appeal to the LW community then determination of the resolution is not up to me. You can't have it both ways, especially if you don't have an established reputation.
1RatsWrongAboutUAP5d
I have created a post for my bet https://www.lesswrong.com/posts/t5W87hQF5gKyTofQB/ufo-betting-put-up-or-shut-up
3RatsWrongAboutUAP5d
Fair enough. What kinds of changes would you suggest for the list? My goal is to find at least one person on LW to make a bet with me, so I welcome feedback

Thank you for the offer. I think your offer is reasonable. The problem is that $10 is too low a price for "something I have to remember for a year". In theory, this could be fixed by increasing the wager amount, but $100k is above my risk limit for a bet (even something as simple as "the sun will rise tomorrow").

I think we've both established a market spread…which is kind of the point of this exercise. You get skin-in-the-game points for maxing out the market's available liquidity at a 0.1% price point.

There's a few other details I though of since my last ... (read more)

I will accept under the following conditions:

  • Make it $10 (you) vs $10k (me).
  • I get sole discretion to decide what constitutes "definitively". Official statements aren't enough. I want transhuman tech, alien biological material, or something at that level. (Any one of them is sufficient. Not all are required.) Photographs and sworn testimony aren't good enough.
  • The bet is valid for a three-month time horizon, after which the bet resolves in my favor.
  • "Nonhuman" means aliens in the classic sense. Proper space aliens. No weaseling out with something like "h
... (read more)

My Offer:

  • I Send you $50 Immediately, You pay out $10,000 if I win
  • I want a 5 year time horizon.
  • I am betting that one of the "very weird" hypotheses turns out true. Bet resolves in my favor if within 5 years its determined that >0 UAP are explained by:
    • Literal aliens
    • Magic/spiritual/paranormal/psychic phenomenon
    • Time travel
    • Leftovers of an ancient civilization / Some other unknown non-human advanced civilization on earth
    • Some other explanation that's of this level of "very weird"
      • Explicitly excluding merely hyper-advanced human tech
      • I forfeit any and all potenti
... (read more)
2lc6d
I would much rather have a 12 month time horizon. Evidence could take a while to filter out. Other than that I'd accept.
1jam_brand7d
Serious question: would something originating adjacently from a separate Everett branch count? (sillier-though-hopefully-not-counterproductive question: since your final statement especially would, I think, often seem to go without saying, its "needless" inclusion actually strikes me as probably-not-but-still-hypothetically-maybe worrisome -- surely you're not intending to imply that's the only recourse allowed for being denied one's winning lottery ticket? [or perhaps my own asking is needless since someone deciding to be a jerk and not wanting to pay could simply use such agreed-upon discretion to "fairly" declare themselves the winner anyways, in which case: sorry for the derail!])

Here's the short version: Suppose you think a prediction is mispriced but it's distant in the future. Instead of buying credits that pay out on resolution, you buy futures instead. You don't have to tie up capital, since payment is due on resolution instead of upfront. Your asset equals your liability. There is no beta.

Financial derivatives solve the shorttermism problem in traditional securities markets. If you use them in prediction markets, then they will (theoretically) do the exact same thing, by (theoretically) operating the exact same way. In practi... (read more)

"Predictive market derivatives" is on my list of things I should write about. What, precisely, do you mean by shorttermism? Do you mean how placing long-term bets in prediction markets ties up capital that could otherwise be put to use? (I think I understand you, but I want to be certain first.)

Financial derivatives work the same way in prediction markets as they do on existing securities markets. I already wrote a little bit about derivatives in existing securities markets, but am not sure that post fully answers your question.

2mako yass8d
Yes, so canny traders have an incentive to ignore long-term questions even though that makes the market somewhat useless. There's an adverse selection effect where the more someone has going on the more they'll focus on short closing dates. (But I realize that long-term bets are still worth something to them, this doesn't necessarily prevent them from weighing in at all. If a market consisted solely of very good forecasters, I'd expect the predictions of long-term bets to be more volatile (less liquidity) but still roughly well informed, however, it doesn't consist solely of very good traders, and also, sometimes the skills you need to do well on small timescales are very different from the skills you need on longer timescales, so in some worlds we might not expect prediction markets to work well at all. One of the very anticapitalisms that I've noticed is that expertise doesn't always generalize out of domain, while money can stray into whatever domain it wants to. Reflecting, though, subquestion: If it's not for the expert to decide for themselves (and with the aid of consultants) where their skills generalize to, why should we think we can appoint someone else who's better at making that decision? It seems unlikely that we could.)

My personal confidence of "no aliens" is so high it rounds to 100%. Placing a bet is basically just a loan with a weird "if aliens are real I don't get paid back" tacked on. The real question then is "at what rate am I willing to lend $X0,000 for 10 years to a stranger? If you can guarantee that I'll beat the stock market by 5% (conditional on no aliens) then I'm good to go, but the "guarantee" is very important. It needs to take into account things like bankruptcy on your end. I don't think our spread is wide enough to make that feasible. It'd take a lot of paperwork.

This is why we need real, formal, legal prediction markets, with derivatives. They would solve all of these problems.

4lc8d
I would be willing to send you 100$ in advance, on a promise that you'll pay me 100,000 if it turns out definitively that these UFOs are built by nonhumans.
2mako yass8d
Derivatives solve prediction market shorttermism? I didn't realize that. Has that been written up anywhere?

Most bets I see are on the order of $10-$1000 which, according to the Kelly Criterion, implies negligible confidence. I'm willing to bet substantially more than that.

If we had a real prediction market with proper derivatives, low fees, high liquidity, reputable oracles, etcetera, then I'd just use the standard exchange, but we don't. Consequently, market friction vastly outweighs actual probabilities in importance.

Perhaps you mean that the other person should come up with the odds, and then you'll determine your bet amount using the Kelly criterion, assu

... (read more)

That is an honorable offer (I appreciate it, really), but it has negative expected value for me due to counterparty risk, friction, variance, etcetera. (See bayesed's comment.) I'd need substantially better odds for the expected profit to exceed the friction.

1Taleuntum9d
Thanks and that's fair. I would have liked to bet mostly as a hedge to allow myself to not think about aliens in the next 10 years (I semi-regularly get roped into investigating some claims, but the UFO community's epistemics is the worst I've seen and it is always an awful experience), but the bet wasn't really positive EV for me either, so I don't think I will bet at worse odds, but you can probably find someone on the r/UFOs subreddit if you want, some of them seem to be celebrating the new world order now. 

I'm willing to bet five figures, in theory, but there's a ton of factors that need to be accounted for like capital tie-up, counterparty risk, the value of my time, etc. So if your odds aren't lower than 90%, then it's probably not even worthwhile to bet. Too much friction.

1Aiyen11d
What odds are you willing to give taking friction into account?

Bayesed is correct. There's a whole bunch of factors which affect whether a bet is worthwhile.

I am willing to publicly bet you at 99% odds that, within the next 10 years, there will be no conclusive proof that we have been visited by craft of intelligent, nonhuman origin. I am willing to bet according to the Kelly Criterion, which means I am willing to bet a significant fraction of my total net worth.

[Edit: This gets really complicated really fast. I mean that I'm willing to publicly bet at 99% implied odds on my side, after various costs and risks are factored in. The various costs and risks far outweigh my <1% chance of losing the bet for mundane reasons. A counterparty to this bet would need confidence in a UFO existence far lower than 99% for a bet to make sense.]

6bayesed12d
What do you mean when you say you're "willing to bet according to the Kelly Criterion"? If you're proposing a bet with 99% odds and your actual belief that you'll win the bet is also 99%, then the Kelly Criterion would advise against betting (since the EV of such a bet would be zero, i.e. merely neutral). Perhaps you mean that the other person should come up with the odds, and then you'll determine your bet amount using the Kelly criterion, assuming a 99% probability of winning for yourself.
1Aiyen12d
How much are you willing to bet? I’ll take you up on that up to fairly high stakes. I agree that alien visits are fairly unlikely, but not 99% unlikely.
3Taleuntum12d
If your offer isn't just to lc, then I accept: My 20 usd against your 20*99=1980 usd, both sides adjusted for inflation and the time value of money using US Treasury Bills, paid either at 2033.06.06. or when you admit a conclusive proof was found. Are these terms acceptable?

"Perhaps you might point to some examples of how it’s best applied?" ⇒ "I'd be curious to read some examples of how it’s best applied."

By changing from a question to a statement, the request for information is transferred from a single person [me] to anyone reading the comment thread. This results in a diffusion of responsibility, which reduces the implicit imposition placed on the original parent.

Another advantage of using statements instead of questions is that they tend to direct me toward positive claims, instead of just making demands for rigor. This avoids some of the more annoyingly asymmetric aspects of Socratic dialogue.

7Said Achmiz2mo
The request can be fulfilled by anyone either way, though. There doesn’t seem to me to be any difference, in that regard. Hmm. I’m afraid I find the linked essay somewhat hard to make sense of. But, in any case, I’ll give your comments some thought, thanks.

I also tend to write concisely. A trick I often use is writing statements instead of questions. I feel statements are less imposing, since they lack the same level of implicit demand that they be responded to.

5Said Achmiz2mo
Hmm, it’s an interesting tactic, certainly. I’m not sure that it’s applicable in all cases, but it’s interesting. Perhaps you might point to some examples of how it’s best applied?

One solution is to limit the number of banned users to a small fraction of overall commentors. I've written 297 posts so far and have banned only 3 users from commenting on them. (I did not ban Duncan or Said.)

My highest-quality criticism comes from users who I have never even considered banning. Their comments are consistently well-reasoned and factually correct.

Do you mean the free energy principle?

3bargo4mo
Sure, I mean that it is an implementation of what you mentioned in the third-to-last paragraph.

I think the most virtuous solution to your hypothetical is to say "I don't know anything about existential risk, but I'd bet at 75% confidence that a mathematician will prove that 2+2≠5" (or something along those lines).

Your comment is contingent on several binary possibilities about my intentions. I appreciate your attempt to address all leaves of the decision tree. Here I will help limit the work you have to do by pinning things down.

To clarify,

My post serves one purpose: to register a public prediction. I am betting reputation. But it makes no sense to bet reputation on something everyone agrees on. It only makes sense to bet on things people disagree on. I'm hoping people will make counter-predictions because that can help verify, in the future, that the claims I made... (read more)

2gjm4mo
Thanks for the clarification. I'm not sure I quite agree with you about strategic ambiguity, though. Again, imagine that you'd said "I am 80% confident that the human race will still be here in 100 years, because 2+2=5". If someone says "I don't know anything about existential risk, but I know that 2+2 isn't 5 and that aside from ex falso quodlibet basic arithmetic like this obviously can't tell us anything about it", then I am perfectly happy for them to claim that they knew you were wrong even though they didn't stand to lose anything if your overall prediction turns out right. (My own position, not that anyone should care: my gut agrees with lsusr's overall position "but I try not to think with my gut"; I don't think I understand all the possible ways AI progress could go well enough for any prediction I'd make by explicitly reasoning it out to be worth much; accordingly I decline to make a concrete prediction; I mostly agree that making such predictions is a virtuous activity because it disincentivizes overconfident-looking bullshitting, but I think admitting one's ignorance is about equally virtuous; the arguments mentioned in the OP seem to me unlikely to be correct but I could well be missing important insights that would make them more plausible. And I do agree that the comments lsusr replied to in the way I'm gently objecting to would have been improved by adding "and therefore I think our chance of survival is below 20%" or "but I do agree that we will probably still be here in 100 years" or "and I have no idea about the actual prediction lsusr is making" or whatever.)

Maybe? It depends a lot on how I interpret your question. I'm trying to keep these posts contained and so I'd rather not answer that question in this thread.

Would you like to register a counter prediction?

4abramdemski4mo
Sure. It seems to me like humans are in a bad spot, with a significant chance of not surviving the next hundred years, depending mainly on whether very weak alignment methods are enough to land us in anything like a corrigibility basin.

Would you like to publicly register a counterprediction?

That's a perfectly reasonable position to take.

Philosophical zombie takeover seems like a real possibility to me.

Huh? Anyway, I don't believe in p-zombies, and for example think that human emotions expressed by sufficiently coherent LLM characters are as real as human emotions, because emotions exist as thingy simulacra even when there are no concrete physical or cognitive-architectural correlates.

Thanks. These seem like good definitions. They actually set the bar high for your prediction, which is respectable. I appreciate you taking this seriously.

If you'll permit just a little bit more pedantic nitpicking, do you mind if I request a precise definition of nanotech? I assume you mean self-replicating nanobots (grey goo) because, technically, we already have nanotech. However, putting the bar at grey goo (potential, of course—the system doesn't have to actually make it for real) might be setting it above what you intended.

5DaemonicSigil4mo
Self replicating nanotech is what I'm referring to, yes. Doesn't have to be a bacteria-like grey goo sea of nanobots, though. I'd generally expect nanotech to look more like a bunch of nanofactories, computers, energy collectors, and nanomachines to do various other jobs, and some of the nanofactories have the job of producing other nanofactories so that the whole system replicates itself. There wouldn't be the constraint that there is with bacteria where each cell is in competition with all the others.

What are your definitions for "Strong AGI", "the alignment problem succeed[s]" and "humanity survives"?

3DaemonicSigil4mo
Strong AGI: Artificial intelligence strong enough to build nanotech, while being at least as general as humans (probably more general). This definition doesn't imply anything about the goals or values of such an AI, but being at least as general as humans does imply that it is an agent that can select actions, and also implies that it is at least as data-efficient as humans. Humanity survives: At least one person who was alive before the AI was built is still alive 50 years later. Includes both humanity remaining biological and uploading, doesn't include everyone dying. Alignment problem: The problem of picking out an AI design that won't kill everyone from the space of possible designs for Strong AGI. Aligned by default: Maybe most of the space of possible designs for Strong AGI does in fact consist of AIs that won't kill everyone. If so, then "pick a design at random" is a sufficient strategy for solving the alignment problem. An attempt at solving the alignment problem: Some group of people who believe that the alignment problem is hard (i.e. picking a design at random has a very low chance of working) try to solve it. The group doesn't have to be rationalists, or to call it the "alignment problem" though. A successful attempt at solving the alignment problem: One of the groups in the above definition do in fact solve the alignment problem, i.e. they find a design for Strong AGI that won't kill everyone. Important note: If Strong AGI is aligned by default, then no attempts are considered successful, since the problem wasn't really a problem in the first place.

Would you like to publicly register a counterprediction?

6lc4mo
Sure, P(doom)>=50%, and that's subject to change. But of course I'll only ever be proven wrong.

It seems to me like that benefit is very marginal compared to the simple expected value loss if the world survives.

Edit: Caplan wins alpha but Yudkowsky wins mere liquidity.

7Brendan Long4mo
If he's confident enough it seems like a very good expected value: If Yudkowsky is right, he's getting an infinity% return on investment (since Caplan gave him $100 that he'll never have to pay back).
1simon4mo
In the limit of high expectation of the world ending, the expected value loss if the world survives goes to zero, so we've now established that the bet can, in fact, work. With respect to the actual values, given that the bet is for an amount of money small relative to expected income, I'd expect fairly linear effects where the expected utility cost* of doing extra work/spending less to accumulate $200 in the future is pretty close to twice the utility cost of having $100 more to spend now, such that if the chance of the world ending is greater than 50% it makes sense to make the bet. (*ignoring discount rates, wage rate changes etc., which aren't the point) Further edit: if the world is likely to end, yes Yudkowsky wins alpha.

You're right. I have edited the post to remove the bit about tying up capital. That part was wrong.

$100 is only useful if you spend it or invest it. If Yudkowsky spends it, that means he burdned through all his other investments and bankrupted himself. And it makes no sense to invest it because he only keeps the money if the world ends and investments become worthless.

7simon4mo
No, imagine he has some utility function with one term that depends on spending money (e.g. consumer goods) and another term that depends on having money (preserving optionality/avoiding bankruptcy). In equilibrium, Eliezer will divide his money between saving and spending money to maximize the total utility of those terms. If he has an extra $100, he has an extra $100. He will divide it in some manner between those terms and be, in fact, better off. It isn't important how precisely he divides it.  If he expects the world to end, very little resources should be allocated to savings for expenses expected to occur after the end of the world. If the world doesn't end, then he will suddenly have to pay $200. As it approaches the deadline and he expects the world not to end, he will now be worse off as he has to reallocate resources to paying off the debt. But in expectation, if he initially expects the world to end with high probability, he is still better off having the $100.

Not directly, but that's not quite what I'm getting at. What I really mean is that Eliezer is honor-bound to maintain enough collateral to repay Caplan in case the world survives. [Edit: lc pointed out a flaw in that logic.]

One way Eliezer could benefit from this bet (which I left out in my post) is by spending literally all of his money and then suddenly earning back the $200 right before he has to repay Caplan. But that would require him to literally bankrupt himself, which outweights any benefit he could accrue from this bet. (Except in the world I alre... (read more)

8simon4mo
He's unlikely to bankrupt himself, precisely because it would have bad effects. But, that doesn't mean he doesn't benefit from the extra $100! Instead, it means he has an additional $100 cushion to avoid bankruptcy, and is in fact $100 better off than if he didn't have that $100, conditional on the world ending before the deadline.  ETA: and planning to earn it back right before the deadline is perfectly rational on the assumption that the likelihood of the world ending is very high. It only doesn't make sense if he doesn't actually believe in what he's betting on!

I guess the basic answer to my question is that you're quite motivated by biological plausibility. There are many reasons why this might be, so I shouldn't guess at the specific motives.

You're right. I want to know how my own brain works.

But if you're more interested in a broader mathematical understanding of how intelligence, in general, works, then that could explain some of our motivational disconnect.

2abramdemski4mo
An important question is whether PP contains some secrets of intelligence which are critical for AI alignment. I think some intelligent people think the answer is yes. But the biological motivation doesn't especially point to this (I think). If you have any arguments for such a conclusion I would be curious to hear it.

Here's a quote from Ziz's post My Journey to the Dark Side.

Reject morality. Never do the right thing because it’s the right thing. Never even think that concept or ask that question unless it’s to model what others will think. And then, always in quotes. Always in quotes and treated as radioactive. Make the source of sentiment inside you that made you learn to care about what was the right thing express itself some other way.

Here's a quote from Ziz's post Neutral and Evil.

If you’re reading this and this is you, I recommend aiming for lawful evil. K

... (read more)
-1Gerald Monroe4mo
So this morality would be one of "optimized selfishness"? "Do whatever is best for me, knowing that means it will be on balance evil"? The morality of Bernie Madoff? Interesting.

Ziz's blog is openly, obviously evil. And not in a fun, trolly way—or even inadvertent mundane evil. Boringly explicit evil, with literal endorsement of the Star Wars Sith religion.

7Gerald Monroe4mo
Would it be evil from the perspective of someone with priors who believes animal lives are equal in value or greater than humans lives? Presumably Zizians don't believe they are evil.

Thank you for these. They used to be my best source of COVID information. Technically they still are, but I have stopped reading them since the information is no longer important enough. I look forward to reading the other stuff you write.

What's your source for this definition?

1Erich_Grunewald4mo
See for example Bostrom's original paper (pdf [https://nickbostrom.com/superintelligentwill.pdf]): It makes no claim about how likely intelligence and final goals are to diverge, it only claims that it's in principle possible to combine any intelligence with any set of goals. Later on in the paper he discusses ways of actually predicting the behavior of a superintelligence, but that's beyond the scope of the Thesis.

I appreciate your epistemic honesty regarding the historical record.

As for the theory of wireheading, I think it's drifting away from the original topic of my post here. I created a new post Self-Reference Breaks the Orthogonality Thesis which I think provides a cleaner version of what I'm trying to say, without the biological spandrels. If you want to continue this discussion, I think it'd be better to do so there.

Thank you for the explanations. They were crystal-clear.

[W]hat's so interesting about the PP analysis of the Pong experiment, then, if you agree that the random-noise-RL thing doesn't work very well compared to alternatives? IE, why derive excitement about a particular deep theory about intelligence (PP) based on a really dumb learning algorithm (noise-based RL)?

What alternatives? Do you mean like flooding the network with neurotransmitters? Or do you mean like the stuff we use in ML? There's lots of better-performing algorithms that you can implement ... (read more)

2abramdemski4mo
Well, I guess another alternative (also very simple on the hardware side) would be the generalization of the Pavlov strategy which I mentioned earlier. This also has the nice feature that lots of little pieces with their own simple 'goals' can coalesce into one agent-like strategy, and it furthermore works with as much or little communication as you give it (so there's not automatically a  communication overhead to achieve the coordination). However, I won't try to argue that it's plausible that biological brains are using something like that.  I guess the basic answer to my question is that you're quite motivated by biological plausibility. There are many reasons why this might be, so I shouldn't guess at the specific motives. For myself, I tend to be disinterested in biologically plausible algorithms if it's easy to point at other algorithms which do better with similar efficiency on computers. (Although results like the equivalence between predictive coding and gradient descent can be interesting for other reasons.) I think bounded computation has to do with important secrets of intelligence, but for example, I find logical induction to be a deeper theory of bounded rationality than (my understanding of) predictive processing - predictive processing seems closer to "getting excited about some specific approximation methods" whereas logical induction seems closer to "principled understanding of what good bounded reasoning even means" (and in particular, obsoletes the idea that bounded rationality is about approximating [https://www.lesswrong.com/posts/wgdfBtLmByaKYovYe/what-does-it-mean-to-apply-decision-theory], in my mind).

If you want to get into that level of technical granularity then there are major things that need to change before applying the PP methodology in the paper to real biological neurons. Two of the big ones are brainwave oscillations and existing in the flow of time.

Mostly what I find interesting is the theory that the bulk of animal brain processing goes into creating a real-time internal simulation of the world, that this is mathematically plausible via forward-propagating signals, and that error and entropy are fused together.

When I say "free energy minimization" I mean the idea that error and surprise are fused together (possibly with an entropy minimizer thrown in).

Good point. I've recently been talking with someone whose native language isn't English and we've been using "pretty" as the imprecise translation of a non-gender-specific adjective. I have removed the word "sexy" entirely.

The world model is actually an integral, but it can be approximated by a search by searching for several good hypothesis instead of integrating over all hypothesis.

Can you tell me what you mean by this statement? When you say "integral" I think "mathematical integral (inverse of derivative)" but I don't think that's what you intend to communicate.

2Donald Hobson4mo
Yes integral is exactly what I intended to communicate.  Think of hypothesis space. A vast abstract space of all possibilities.  Each hypothesis has a P(x) the probability of being true, and a Ua(x) the utility of action a if it is true.  To really evaluate an action, you need to calculate ∫P(x)Ua(x)dx an integral over all hypothesis.  If you don't want to behave with maximum intelligence, just pretty good intelligence, then you can run gradient descent to find a point X by trying to maximize P(x). Then you can calculate Ua(X) to compare actions. More sophisticated methods would sum several points.  This is partly using the known structure of the problem. If you have good evidence, then the function P(x) is basically 0 almost everywhere. So if Ua(X) is changing fairly slowly over the region that is significantly nonzero, looking at any nonzero point of P(x) is a good estimate of the integral.  
Load More