R:A-Z Glossary

Anki deck. 

• Backward chaining. Backward chaining is an inference method described colloquially as working backward from the goal.
• Base rate. In probability and statistics, the base rate (also known as prior probabilities) is the class of probabilities unconditional on "featural evidence" (likelihoods).
• Bayes's Theorem. The equation stating how to update a hypothesis H in light of new evidence E. In its simplest form, Bayes's Theorem says that a hypothesis' probability given the evidence, written P(H|E), equals the likelihood of the evidence given that hypothesis, multiplied by your prior probability P(H) that the hypothesis was true, divided by the prior probability P(E) that you would see that evidence regardless. I.e.:
  
  P(H|E) = P(E|H) P(H) / P(E).
  
  Also known as Bayes's Rule. See "odds ratio" for a simpler way to calculate a Bayesian update.
• Bayesian. (a) Optimally reasoned; reasoned in accordance with the laws of probability. (b) An optimal reasoner, or a reasoner that approximates optimal inference unusually well. (c) Someone who treats beliefs as probabilistic and treats probability theory as a relevant ideal for evaluating reasoners. (d) Related to probabilistic belief. (e) Related to Bayesian statistical methods.
• Bayesian updating. Revising your beliefs in a way that's fully consistent with the information available to you. Perfect Bayesian updating is wildly intractable in realistic environments, so real-world agents have to rely on imperfect heuristics to get by. As an optimality condition, however, Bayesian updating helps make sense of the idea that some ways of changing one's mind work better than others for learning about the world.
• beisutsukai. Japanese for "Bayes user." A fictional order of high-level rationalists, also known as the Bayesian Conspiracy.
• Bell's Theorem. Bell's theorem is a term encompassing a number of closely related results in physics, all of which determine that quantum mechanics is incompatible with noncontextual local hidden-variable theories, given some basic assumptions about the nature of measurement.
• Berkeleian idealism. The belief, espoused by George Berkeley, that things only exist in various minds (including the mind of God).
• bias. (a) A cognitive bias. In Rationality: From AI to Zombies, this will be the default meaning. (b) A statistical bias. (c) An inductive bias. (d) Colloquially: prejudice or unfairness.
• bit. (a) A binary digit, taking the value 0 or 1. (b) The logarithm (base 1/2) of a probability—the maximum information that can be communicated using a binary digit, averaged over the digit's states. Rationality: From AI to Zombies usually uses "bit" in the latter sense.
• black box. Any process whose inner workings are mysterious or poorly understood.
• Black Swan. The black swan theory or theory of black swan events is a metaphor that describes an event that comes as a surprise, has a major effect, and is often inappropriately rationalized after the fact with the benefit of hindsight.
• blind god. One of Yudkowsky's pet names for

• a priori. Before considering the evidence. Similarly, "a posteriori" means "after considering the evidence"; compare prior and posterior probabilities.
  
  In philosophy, "a priori" often refers to the stronger idea of something knowable in the absence of any experiential evidence (outside of the evidence needed to understand the claim).
• affect heuristic. People's general tendency to reason based on things' felt goodness or badness.
• affective death spiral. Yudkowsky's term for a halo effect that perpetuates and exacerbates itself over time.
• AGI. See “artificial general intelligence.”
• AI-Box Experiment. A demonstration by Yudkowsky that people tend to overestimate how hard it is to manipulate people, and therefore underestimate the risk of building an Unfriendly AI that can only interact with its environment by verbally communicating with its programmers. One participant role-plays an AI, while another role-plays a human whose job it is interact with the AI without voluntarily releasing the AI from its “box”. Yudkowsky and a few other people who have role-played the AI have succeeded in getting the human supervisor to agree to release them, which suggests that a superhuman intelligence would have an even easier time escaping.
• akrasia. Akrasia is a lack of self-control or acting against one's better judgment.
• alien god. One of Yudkowsky's pet names for natural selection.
• ambiguity aversion. Preferring small certain gains over much larger uncertain gains.
• amplitude. A quantity in a configuration space, represented by a complex number. Many sources misleadingly refer to quantum amplitudes as "probability amplitudes", even though they aren't probabilities. Amplitudes are physical, not abstract or formal. The complex number’s modulus squared (i.e., its absolute value multiplied by itself) yields the Born probabilities, but the reason for this is unknown.
• amplitude distribution. See “wavefunction.”
• anchoring. The cognitive bias of relying excessively on initial information after receiving relevant new information.
• anthropics. Problems related to reasoning well about how many observers like you there are.
• artificial general intelligence. Artificial intelligence that is "general-purpose" in the same sense that human reasoning is general-purpose. It's hard to crisply state what this kind of reasoning consists in—if we knew how to fully formalize it, we would already know how to build artificial general intelligence. However, we can gesture at (e.g.) humans' ability to excel in many different scientific fields, even though we did not evolve in an ancestral environment containing particle accelerators.
• Aumann's Agreement Theorem.
• availability heuristic. The tendency to base judgments on how easily relevant examples come to mind.
• average utilitarianism. Average utilitarianism values the maximization of the average utility among a group's members. So a group of 100 people each with 100 hedons (or "happiness points") is judged as preferable to a group of 1,000 people with 99 hedons each.

• Backward chaining. Backward chaining is an inference method described colloquially as working backward from the goal.
• Base rate.
• Bayes's Theorem. The equation stating how to

• Unfriendly AI. A hypothetical smarter-than-human artificial intelligence that causes a global catastrophe by pursuing a goal without regard for humanity’s well-being. Yudkowsky predicts that superintelligent AI will be “Unfriendly” by default, unless a special effort goes into researching how to give AI stable, known, humane goals. Unfriendliness doesn’t imply malice, anger, or other human characteristics; a completely impersonal optimization process can be “Unfriendly” even if its only goal is to make paperclips. This is because even a goal as innocent as ‘maximize the expected number of paperclips’ could motivate an AI to treat humans as competitors for physical resources, or as threats to the AI’s aspirations.
• uniform probability distribution. A distribution in which all events have equal probability; a maximum-entropy probability distribution.
• universal Turing machine. A Turing machine that can compute all Turing-computable functions. If something can be done by any Turing machine, then it can be done by every universal Turing machine. A system that can in principle do anything a Turing machine could is called “Turing-complete."
• updating. Revising one’s beliefs. See also "Bayesian updating."
• utilitarianism. An ethical theory asserting that one should act in whichever manner causes the most benefit to people, minus how much harm results. Standard utilitarianism argues that acts can be justified even if they are morally counter-intuitive and harmful, provided that the benefit outweighs the harm.
• utility function. A function that ranks outcomes by "utility," i.e., by how well they satisfy some set of goals or constraints. Humans are limited and imperfect reasoners, and don't consistently optimize any endorsed utility function; but the idea of optimizing a utility function helps us give formal content to "what it means to pursue a goal well," just as Bayesian updating helps formalize "what it means to learn well."
• utilon. Yudkowsky’s name for a unit of utility, i.e., something that satisfies a goal. The term is deliberately vague, to permit discussion of desired and desirable things without relying on imperfect proxies such as monetary value and self-reported happiness.

__TOC__A

A

• Peano arithmetic.

• superposition.

• information. (a) Colloquially, anythingany fact or data that helps someone better understand something. (b) In information theory, how surprising, improbable, or complex something is. E.g., there is more information in seeing a fair eight-sided die come up "4" than in seeing a fair six-sided die come up "4," because the former event has probability 1/8 while the latter has probability 1/6.
  
  We can also speak of the average information in a fair six- or eight-sided die roll in general, before seeing the actual number; this a die roll's expected amount of information, which is called its "entropy." If a fair die has more sides, then rolling such a die will have more entropy because on average, the outcome of the roll has more information (i.e., lower probability).
  
  When knowing about one variable can help narrow down the value of another variable, the two variables are said to have mutual information.

• intuition pump.

• intuitionistic logic. An approach to logic that rejects the law of the excluded middle, "Every statement is true or false."

• Laplace's Law of Succession.

• lookup table.

• marginal utility.

• marginal variable.

• economies of scale.

• fungible.

• game theory.

• hyper-real number.

• information. (a) Colloquially, anything that helps someone better understand something. (b) In information theory, how surprising, improbable, or complex something is. E.g., there is more information in seeing a fair eight-sided die come up "4" than in seeing a fair six-sided die come up "4," because the former event has probability 1/8 while the latter has probability 1/6.
  
  We can also speak of the average information in a fair six- or eight-sided die roll in general, before seeing the actual number; this a die roll's expected amount of information, which is called its "entropy." If a fair die has more sides, then rolling such a die will have more entropy because on average, the outcome of the roll has more information (i.e., lower probability).
  
  When knowing about one variable can help narrow down the value of another variable, the two variables are said to have mutual information.

V