https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem

https://www.lesswrong.com/tag/vnm-theorem 

Hi I am working on Rob Miles' Stampy project (https://aisafety.info/), which is creating a centralized resource for answering questions about AI safety and alignment. Would we be able to incorporate your list of frequently asked questions and answers into our system (perhaps with some modification)? I think they are really nice answers to some of the basic questions and would be useful for people curious about the topic to see.

@drocta @Cookiecarver We started writing up an answer to this question for Stampy. If you have any suggestions to make it better I would really appreciate it. Are there important factors we are leaving out? Something that sounds off? We would be happy for any feedback you have either here or on the document itself https://docs.google.com/document/d/1tbubYvI0CJ1M8ude-tEouI4mzEI5NOVrGvFlMboRUaw/edit#

I really liked this post since it took something I did intuitively and haphazardly and gave it a handle by providing the terms to start practicing it intentionally. This had at least two benefits:

First it allowed me to use this technique in a much wider set of circumstances, and to improve the voices that I already have. Identifying the phenomenon allowed it to move from a knack which showed up by luck, to a skill.

Second, it allowed me to communicate the experience more easily to others, and open the possibility for them to use it as well. Unlike many lesswrong posts, I found that the technique in this post spoke to a bunch of people outside of the lesswrong community. For example, one friend who liked this idea. tried applying it to developing an Elijah the Prophet figure that he could interact with.

Cool. So in principle we could just as well use the rationals from the standpoint of scientific inference. But we use the reals because it makes the math easier. Thank you.

Thank you.

I am a little confused. I was working with a definition of continuity mentioned here https://mathworld.wolfram.com/RationalNumber.html : "It is always possible to find another rational number between any two members of the set of rationals. Therefore, rather counterintuitively, the rational numbers are a continuous set, but at the same time countable." 

I understand that Rationals aren't complete, and my question is why this is important for scientific inference. In other words, are we using the reals only because it makes the math easier, or is there a concrete example of inference  which completeness helps with? 

Specifically, since the context Jaynes is interested in is designing a (hypothetical) robot's brain, and in order to achieve that we need to associate degrees of plausibility with a physical state, I don't see why that entails the property of completeness which you mentioned? In fact, we mostly use digital and not analog computers, which use rational approximations for the reals. What does this system of reasoning lack?

You might find the book Is Water H2O? by Hasok Chang, 2012 useful. It was mentioned by Adam Shimi in this post https://www.lesswrong.com/posts/wi3upQibefMcFs5to/levels-of-pluralism

It also reminds me of Richard Feynman not wanting a position at the institute for advance study. 

"I don't believe I can really do without teaching. The reason is, I have to have something so that when I don't have any ideas and I'm not getting anywhere I can say to myself, "At least I'm living; at least I'm doing something; I am making some contribution" -- it's just psychological.

When I was at Princeton in the 1940s I could see what happened to those great minds at the Institute for Advanced Study, who had been specially selected for their tremendous brains and were now given this opportunity to sit in this lovely house by the woods there, with no classes to teach, with no obligations whatsoever. These poor bastards could now sit and think clearly all by themselves, OK? So they don't get any ideas for a while: They have every opportunity to do something, and they are not getting any ideas. I believe that in a situation like this a kind of guilt or depression worms inside of you, and you begin to worry about not getting any ideas. And nothing happens. Still no ideas come.

Nothing happens because there's not enough real activity and challenge: You're not in contact with the experimental guys. You don't have to think how to answer questions from the students. Nothing!

In any thinking process there are moments when everything is going good and you've got wonderful ideas. Teaching is an interruption, and so it's the greatest pain in the neck in the world. And then there are the longer period of time when not much is coming to you. You're not getting any ideas, and if you're doing nothing at all, it drives you nuts! You can't even say "I'm teaching my class."

If you're teaching a class, you can think about the elementary things that you know very well. These things are kind of fun and delightful. It doesn't do any harm to think them over again. Is there a better way to present them? The elementary things are easy to think about; if you can't think of a new thought, no harm done; what you thought about it before is good enough for the class. If you do think of something new, you're rather pleased that you have a new way of looking at it.

The questions of the students are often the source of new research. They often ask profound questions that I've thought about at times and then given up on, so to speak, for a while. It wouldn't do me any harm to think about them again and see if I can go any further now. The students may not be able to see the thing I want to answer, or the subtleties I want to think about, but they remind me of a problem by asking questions in the neighborhood of that problem. It's not so easy to remind yourself of these things.

So I find that teaching and the students keep life going, and I would never accept any position in which somebody has invented a happy situation for me where I don't have to teach. Never."

— Richard Feynman, Surely You're Joking, Mr. Feynman!