R:A-Z Glossary

tomcatfish (+7) Add missing word in "Utilitarianism"
Ruby (+1/-8)
Rob Bensinger (+31)
Rob Bensinger (+16/-8)
Rob Bensinger (+214)
Rob Bensinger (+950)
Rob Bensinger (-1)
Rob Bensinger (+184/-9357) moving entries to the talk page that are some combination of 'pretty obvious' and/or 'pretty unimportant'
Rob Bensinger (+1486)
Rob Bensinger (+5720/-2251) added entries from HACYM print-edition glossary
  • Unfriendly AI. A hypothetical smarter-than-human artificial intelligence that causes a global catastrophe by pursuing a goal without regard for humanity’s well-being. Yudkowsky predicts that superintelligent AI will be “Unfriendly” by default, unless a special effort goes into researching how to give AI stable, known, humane goals. Unfriendliness doesn’t imply malice, anger, or other human characteristics; a completely impersonal optimization process can be “Unfriendly” even if its only goal is to make paperclips. This is because even a goal as innocent as ‘maximize the expected number of paperclips’ could motivate an AI to treat humans as competitors for physical resources, or as threats to the AI’s aspirations.
  • uniform probability distribution. A distribution in which all events have equal probability; a maximum-entropy probability distribution.
  • universal Turing machine. A Turing machine that can compute all Turing-computable functions. If something can be done by any Turing machine, then it can be done by every universal Turing machine. A system that can in principle do anything a Turing machine could is called “Turing-complete."
  • updating. Revising one’s beliefs. See also "Bayesian updating."
  • utilitarianism. An ethical theory asserting that one should act in whichever manner causes the most benefit to people, minus how much harm results. Standard utilitarianism argues that acts can be justified even if they are morally counter-intuitive and harmful, provided that the benefit outweighs the harm.
  • utility function. A function that ranks outcomes by "utility," i.e., by how well they satisfy some set of goals or constraints. Humans are limited and imperfect reasoners, and don't consistently optimize any endorsed utility function; but the idea of optimizing a utility function helps us give formal content to "what it means to pursue a goal well," just as Bayesian updating helps formalize "what it means to learn well."
  • utilon. Yudkowsky’s name for a unit of utility, i.e., something that satisfies a goal. The term is deliberately vague, to permit discussion of desired and desirable things without relying on imperfect proxies such as monetary value and self-reported happiness.
  • information. (a) Colloquially, anythingany fact or data that helps someone better understand something. (b) In information theory, how surprising, improbable, or complex something is. E.g., there is more information in seeing a fair eight-sided die come up "4" than in seeing a fair six-sided die come up "4," because the former event has probability 1/8 while the latter has probability 1/6.

    We can also speak of the average information in a fair six- or eight-sided die roll in general, before seeing the actual number; this a die roll's expected amount of information, which is called its "entropy." If a fair die has more sides, then rolling such a die will have more entropy because on average, the outcome of the roll has more information (i.e., lower probability).

    When knowing about one variable can help narrow down the value of another variable, the two variables are said to have mutual information.
  • intuition pump.
  • intuitionistic logic. An approach to logic that rejects the law of the excluded middle, "Every statement is true or false."
  • Laplace's Law of Succession.
  • lookup table.
  • marginal utility.
  • marginal variable.
  • economies of scale.
  • fungible.
  • game theory.
  • hyper-real number.
  • information. (a) Colloquially, anything that helps someone better understand something. (b) In information theory, how surprising, improbable, or complex something is. E.g., there is more information in seeing a fair eight-sided die come up "4" than in seeing a fair six-sided die come up "4," because the former event has probability 1/8 while the latter has probability 1/6.

    We can also speak of the average information in a fair six- or eight-sided die roll in general, before seeing the actual number; this a die roll's
    expected amount of information, which is called its "entropy." If a fair die has more sides, then rolling such a die will have more entropy because on average, the outcome of the roll has more information (i.e., lower probability).

    When knowing about one variable can help narrow down the value of another variable, the two variables are said to have
    mutual information.
  • ad hominem. A verbal attack on the person making an argument, where a direct criticism of the argument is possible and would be more relevant. The term is reserved for cases where talking about the person amounts to changing the topic. If your character is the topic from the outset (e.g., during a job interview), then it isn't an ad hominem fallacy to cite evidence showing that you're a lousy worker.
  • algorithm. A specific procedure for computing some function. A mathematical object consisting of a finite, well-defined sequence of steps that concludes with some output determined by its initial input. Multiple physical systems can simultaneously instantiate the same algorithm.
  • anthropomorphism. The tendency to assign human qualities to non-human phenomena.
  • ASCIIAumann's Agreement Theorem. The American Standard Code for Information Exchange. A very simple system for encoding 128 ordinary English letters, numbers, and punctuation.
  • Backward chaining.
  • Base rate.
  • Black Swan.
  • bucket. See “pebble and bucket.”
  • conditional independence.
  • conditional probability. The probability that a statement is true on the assumption that some other statement is true. E.g., the conditional probability P(A|B) means "the probability of A given that B."
  • consequentialism. (a) The ethical theory that the moral rightness of actions depends only on those actions' consequences.what outcomes result. Consequentialism is normally contrasted with ideas like deontology, which says that morality is about following certain rules (e.g., "don't lie") regardless of the consequences. (b) Yudkowsky's term for any reasoning process that selects actions based on their consequences.
  • Cox's Theorem.
  • de novo. Entirely new; produced from scratch.
  • decibel.
  • econblog. Economics blog.
  • evolution. (a) In biology, change in a population’s heritable features. (b) In other fields, change of any sort.
  • formalism. A specific way of logically or mathematically representing something.
  • function. A relation between inputs and outputs such that every input has exactly one output. A mapping between two sets in which every element in the first set is assigned a single specific element from the second.
  • hat tip. A grateful acknowledgment of someone who brought information to one's attention.
  • idiot god. One of Yudkowsky's pet names for natural selection.
  • iff. If, and only if.
  • Lamarckism. The 19th-century pre-Darwinian hypothesis that populations evolve via the hereditary transmission of the traits practiced and cultivated by the previous generation.
  • Machine Intelligence Research Institute. A small non-profit organization that works on mathematical research related to Friendly AI. Yudkowsky co-founded MIRI in 2000, and is the senior researcher there.
  • Maxwell’s equations. In classical physics, a set of differential equations that model the behavior of electromagnetic fields.
  • meme. Richard Dawkins’ term for a thought that can be spread through social networks.
  • minimax. A decision rule for turn-based zero-sum two-player games, where one picks moves that minimize one's opponent’s chance of winning when their moves maximize their chance of winning. This rule is intended to perform well even in worst-case scenarios where one’s opponent makes excellent decisions.
  • MIRI. See “Machine Intelligence Research Institute.”
  • money pump. A person who is irrationally willing to accept sequences of trades that add up to an expected loss.
  • natural selection. The process by which heritable biological traits change in frequency due to their effect on how much their bearers reproduce.
  • Neutral Point of View. A policy used by the online encyclopedia Wikipedia to instruct users on how they should edit the site’s contents. Following this policy means reporting on the different positions in controversies, while refraining from weighing in on which position is correct.
  • normality. (a) What’s commonplace. (b) What’s expected, prosaic, and unsurprising. Categorizing things as “normal” or weird” can cause one to conflate these two definitions, as though something must be inherently extraordinary or unusual just because one finds it surprising or difficult to predict. This is an example of confusing a feature of mental maps with a feature of the territory.
  • objective. (a) Remaining real or true regardless of what one’s opinions or other mental states are. (b) Conforming to generally applicable moral or epistemic norms (e.g., fairness or truth) rather than to one’s biases or idiosyncrasies. (c) Perceived or acted on by an agent. (d) A goal.
  • Objectivism. A philosophy and social movement invented by Ayn Rand, known for promoting self-interest and laissez-faire capitalism as “rational.”
  • OLPC. See “One Laptop Per Child.”
  • One Laptop Per Child. A program to distribute cheap laptops to poor children.
  • OpenCog. An open-source AGI project based in large part on work by Ben Goertzel. MIRI provided seed funding to OpenCog in 2008, but subsequently redirected its research efforts elsewhere.
  • oracle. See “halting oracle.”
  • Overcoming Bias. The blog where Yudkowsky originally wrote most of the content of Rationality: From AI to Zombies. It can be found at [www.overcomingbias.com], where it now functions as the personal blog of Yudkowsky’s co-blogger, Robin Hanson. Most of Yudkowsky’s writing is now hosted on the community blog Less Wrong.
  • pebble and bucket. An example of a system for mapping reality, analogous to memory or belief. One picks some variable in the world, and places pebbles in the bucket when the variable’s value (or one’s evidence for its value) changes. The point of this illustrative example is that the mechanism is very simple, yet achieves many of the same goals as properties that see heated philosophical debate, such as perception, truth, knowledge, meaning, and reference.
  • photon. An elementary particle of light.
  • proposition. Something that is either true or false. Commands, requests, questions, cheers, and excessively vague or ambiguous assertions are not propositions in this strict sense. Some philosophers identify propositions with sets of possible worlds -- that is, they think of propositions like “snow is white” not as particular patterns of ink in books, but rather as the thing held in common by all logically consistent scenarios featuring white snow. This is one way of abstracting away from how sentences are worded, what language they are in, etc., and merely discussing what makes the sentences true or false. (In mathematics, the word “proposition” has separately been used to refer to theorems -- e.g., “Euclid’s First Proposition.”)
  • quantum mechanics. The branch of physics that studies subatomic phenomena and their nonclassical implications for larger structures; also, the mathematical formalisms used by physicists to predict such phenomena. Although the predictive value of such formalisms is extraordinarily well-established experimentally, physicists continue to debate how to incorporate gravitation into quantum mechanics, whether there are more fundamental patterns underlying quantum phenomena, and why the formalisms require a “Born rule” to relate the deterministic evolution of the wavefunction under Schrödinger’s equation to observed experimental outcomes. Related to the last question is a controversy in philosophy of physics over the physical significance of quantum-mechanical concepts like “wavefunction,” e.g., whether this mathematical structure in some sense exists objectively, or whether it is merely a convenience for calculation.
  • recursion. A sequence of similar actions that each build on the result of the previous action.
  • separate magisteria. See “magisterium.”
  • sequences. Yudkowsky’s name for short series of thematically linked blog posts or essays.
  • set theory. The study of relationships between abstract collections of objects, with a focus on collections of other collections. A branch of mathematical logic frequently used as a foundation for other mathematical fields.
  • Singularity Summit. An annual conference held by MIRI from 2006 to 2012. Purchased by Singularity University in 2013.
  • strawman. An indefensible claim that is wrongly attributed to someone whose actual position is more plausible.
  • subjective. (a) Conscious, experiential. (b) Dependent on the particular distinguishing features (e.g., mental states) of agents. (c) Playing favorites, disregarding others’ knowledge or preferences, or otherwise violating some norm as a result of personal biases. Importantly, something can be subjective in sense (a) or (b) without being subjective in sense (c); e.g., one’s ice cream preferences and childhood memories are “subjective” in a perfectly healthy sense.
  • subjectivism. See “Berkeleian idealism.”
  • territory. See “map and territory.”
  • theorem. A statement that has been mathematically or logically proven.
  • Type-A materialism. David Chalmers’s term for the view that the world is purely physical, and that there is no need to try to explain the relationship between the physical facts and the facts of first-person conscious experience. Type-A materialists deny that there is even an apparent mystery about why philosophical zombies seem conceivable. Other varieties of materialist accept that this is a mystery, but expect it to be solved eventually, or deny that the lack of a solution undermines physicalism.
  • utility maximizer. An agent that always picks actions with better outcomes over ones with worse outcomes (relative to its utility function). An expected utility maximizer is more realistic, given that real-world agents must deal with ignorance and uncertainty: it picks the actions that are likeliest to maximize its utility, given the available evidence. An expected utility maximizer’s decisions would sometimes be suboptimal in hindsight, or from an omniscient perspective; but they won’t be foreseeably inferior to any alternative decision, given the agent’s available evidence. Humans can sometimes be usefully modeled as expected utility maximizers with a consistent utility function, but this is at best an approximation, since humans are not perfectly rational.
  • vertex. See “graph.”
  • winning. Yudkowsky’s term for getting what you want. The result of instrumental rationality.
  • AGI. See “artificial general intelligence.”
  • akrasia.
  • amplitude. A quantity in a configuration space, represented by a complex number. Many sources misleadingly refer to quantum amplitudes as "probability amplitudes", even though they aren't probabilities. Amplitudes are physical, not abstract or formal. The complex number’s modulus squared (i.e., its absolute value multiplied by itself) yields the Born probabilities, but the reason for this is unknown.
  • amplitude distribution. See “wavefunction.”
  • average utilitarianism.
  • Bell's Theorem.
  • Born rule.
  • collapse.
  • complex. (a) Colloquially, something with many parts arranged in a relatively specific way. (b) In information theory, something that's relatively hard to formally specify and that thereby gets a larger penalty under Occam's razor; measures of this kind of complexity include Kolmogorov complexity. (c) Complex-valued, i.e., represented by the sum of a real number and an imaginary number.
  • configuration space.
  • consequentialism. (a) The ethical theory that the rightness of actions depends only on those actions' consequences. Consequentialism is normally contrasted with ideas like deontology, which says that morality is about following certain rules (e.g., "don't lie") regardless of the consequences. (b) Yudkowsky's term for any reasoning process that selects actions based on their consequences.
  • Copenhagen Interpretation.
  • decoherence.
  • directed acyclic graph. A graph that is directed (its edges have a direction associated with them) and acyclic (there's no way to follow a sequence of edges in a given direction to loop around from a node back to itself).
  • dukkha.
  • Dutch book.
  • eudaimonia.
  • Eurisko.
  • FAI. See “friendly AI.”
  • falsificationism.
  • Fun Theory.
  • gray goo.
  • Gricean implication.
  • Mind Projection Fallacy.
  • one-boxing. Taking only the opaque box in Newcomb's Problem.
  • probability amplitude. See “amplitude.”
  • uniform probability distribution. A distribution in which all events have equal probability; a maximum-entropy probability distribution.
  • affect heuristic. People's general tendency to reason based on things' felt goodness or badness.
  • affective death spiral. AYudkowsky's term for a halo effect that perpetuates and exacerbates itself over time.
  • causal graph. A directed acyclic graph in which an arrow going from node A to node B is interpreted as "changes in A can directly cause changes in B."
  • correspondence bias. Drawing conclusions about someone's unique disposition from behavior that can be entirely explained by the situation in which it occurs. When we see someone else kick a vending machine, we think they are "an angry person," but when we kick the vending machine, it's because the bus was late, the train was early, and the machine ate our money.
  • cryonics. The low-temperature preservation of brains. Cryonics proponents argue that cryonics should see more routine use for people whose respiration and blood circulation have recently stopped (i.e., people who qualify as clinically deceased), on the grounds that future medical technology may be able to revive such people.
  • epistemology. (a) A world-view or approach to forming beliefs. (b) The study of knowledge.
  • existential risk. Something that threatens to permanently and drastically reduce the value of the future, such as stable global totalitarianism or human extinction.
  • frequentism. (a) The view that the Bayesian approach to probability—i.e., treating probabilities as belief states—is unduly subjective. Frequentists instead propose treating probabilities as frequencies of events. (b) Frequentist statistical methods.
  • Friendly AI. Artificial general intelligence systems that are safe and useful. "Friendly" is a deliberately informal descriptor, intended to signpost that "Friendliness" still has very little technical content and needs to be further developed. Although this remains true in many respects as of this writing (2018), Friendly AI research has become much more formally developed since Yudkowsky coined the term "Friendly AI" in 2001, and the research area is now more often called "AI alignment research."
  • group selection. Natural selection at the level of groups, as opposed to individuals. Historically, group selection used to be viewed as a more central and common part of evolution—evolution was thought to frequently favor self-sacrifice "for the good of the species."
  • halo effect. The tendency to assume that something good in one respect must be good in other respects.
  • humility. Not being arrogant or overconfident. Yudkowsky defines humility as "taking specific actions in anticipation of your own errors." He contrasts this with "modesty," which he views as a social posture for winning others' approval or esteem, rather than as a form of epistemic humility.
  • intelligence explosion. A scenario in which AI systems rapidly improve in cognitive ability because they see fast, consistent, sustained returns on investing work into such improvement. This could happen via AI systems using their intelligence to rewrite their own code, improve their hardware, or acquire more hardware, then leveraging their improved capabilities to find more ways to improve.

J

  • joint probability distribution. A probability distribution that assigns probabilities to combinations of claims. E.g., if the claims in question are "Is it cold?" and "Is it raining?", a joint probability distribution could assign probabilities to "it's cold and rainy," "it's cold and not rainy," "it's not cold but is rainy," and "it's neither cold nor rainy."
  • just-world fallacy. The cognitive bias of systematically overestimating how much reward people get for good deeds, and how much punishment they get for bad deeds.
  • koan. In Zen Buddhism, a short story or riddle aimed at helping the hearer break through various preconceptions.
  • many-worlds interpretation. The idea that the basic posits in quantum physics (complex-valued amplitudes) are objectively real and consistently evolve according to the Schrödinger equation. Opposed to anti-realist and collapse interpretations. Many-worlds holds that the classical world we seem to inhabit at any given time is a small component of an ever-branching amplitude.

    The "worlds" of the many-worlds interpretation are not discrete or fundamental to the theory. Speaking of "many worlds" is, rather, a way of gesturing at the idea that the ordinary objects of our experience are part of a much larger whole that contains enormously many similar objects.
  • meme. Richard Dawkins’s term for a thought that can be spread through social networks.
  • meta level. A domain that is more abstract or derivative than some domain it depends on, the "objectobject level." A conversation can be said to operate on a meta level, for example, when it switches from discussing a set of simple or concrete objects to discussing higher-order or indirect features of those objects.
  • modesty. Yudkowsky's term for the social impulse to appear deferential or self-effacing, and resultant behaviors. Yudkowsky contrasts this with the epistemic virtue of humility.
  • monotonic logicmonotonicity. Roughly, the property of never reversing direction. A monotonic function is any function between ordered sets that either preserves the order, or completely flips it. A non-monotonic function, then, is one that at least once takes an a<b input and outputs a>b, and at least once takes a c>d input and outputs c<d.

    A monotonic
    logic is one that will always continue to assert something as true if it ever asserted it as true. For example, if “2+If "2+2=4”4" is proved, then in a monotonic logic no subsequent operation can make it impossible to derive that theorem again in the future. In contrast, non-monotonic logics can “forget”"forget" past conclusions and lose the ability to derive them.
  • monotonicity. In mathematics, the property, loosely speaking, of always moving in the same direction (when one moves at all). If I have a preference ordering over outcomes, a monotonic change to my preferences may increase or decrease how much I care about various outcomes, but it won’t change the order -- if I started off liking cake more than cookies, I’ll end up liking cake more than cookies, though any number of other changes may have taken place. Alternatively, a monotonic function can flip all of my preferences. The only option ruled out is for the function to sometimes flip the ordering and sometimes preserve the ordering. A non-monotonic function, then, is one that at least once take an xy, and at least once takes an x>y and outputs x<y.
  • Moore’s Law. TheA 1965 observation that technological progress has enabledand prediction by Intel co-founder Gordon Moore: roughly every two years (originally every one year), engineers are able to double the number of transistors theythat can be fit on an integrated circuit approximately every two years from the 1960s tocircuit. This projection held true into the 2010s. Other exponential improvements inversions of this "law" consider other progress metrics for computing technology (some of which have also been called “Moore’s Law”) may continue to operate after the end of the original Moore’s Law. The most important of these is the doubling of available computations per dollar. The futurist Ray Kurzweil has argued that the latter exponential trend will continue for many decades, and that this trend will determine rates of AI progress.hardware.
  • object level. A base-The level of concrete things, as contrasted with the "meta" level. The object level tends to be a base case domain, especially one thator starting point, while the meta level is relatively concrete -- e.g., the topic of a conversation,comparatively abstract, recursive, or the target of an action. One might call one’s belief that murder is wrong "object-level" to contrast it with a meta-level belief about moral beliefs, or about the reason murder is wrong, or about something else that pertains to murderindirect in a relatively abstract and indirect way.relevance.
  • ontology. An account of the things that exist, especially one that focuses on their most basic and general similarities. Things are “ontologically distinct”"ontologically distinct" if they are of two fundamentally different kinds.
  • problem of induction. In philosophy, the question of how we can justifiably assert that the future will resemble the past (scientific induction) without relying on evidence that presupposes that very fact.
  • screening off. Making something informationallyevidentially irrelevant. A piece of evidence A screens off a piece of evidence B from a hypothesis C if, once you know about A, learning about B doesn’t affect the probability of C.
  • Simulation Hypothesis. The hypothesis that the world as we know it is a computer program designed by some powerful intelligence. An idea popularized in the movie The Matrix, and discussed more seriously by the philosopher Nick Bostrom.
  • superintelligence. An agent muchSomething vastly smarter (more intellectually resourceful, rational, etc.) than present-day humans. This can be a purely hypothetical agent (e.g., Omega or Laplace’s demon),predicted future technology, like smarter-than-human AI; or it can be a predicted future technology (e.g., Friendlypurely hypothetical agent, such as Omega or Unfriendly AI).Laplace's Demon.
  • transhuman. (a) Entities that are human-like, but much more capable than ordinary biological humans. (b) Related to radical human enhancement. Transhumanism is the view that humans should use technology to radically improve their lives—e.g., curing disease or ending aging.
Load More (10/16)