Conjecture is an alignment startup founded by Connor Leahy, Sid Black and Gabriel Alfour, which aims to scale alignment research... (read more)
The rationalist movement, rationality community,1 rationalsphere or rationalistsphere2 represents a set of modes of bayesian thinking from self-described rationalists or 'aspiring rationalists' typically associated with the Less Wrong diaspora and their associated communities... (read more)
Less Wrong is a community resource devoted to refining the art of human rationality which was founded in 2009. Site activity reached a peak in 2011-13 and a trough in 2016-17. This page mainly describes the history through 2016... (read more)
A Seed AI (a term coined by Eliezer Yudkowsky) is an Artificial General Intelligence (AGI) which improves itself by recursively rewriting its own source code without human intervention. Initially this program would likely have a minimal intelligence, but over the course of many iterations it would evolve to human-equivalent or even trans-human reasoning. The key for successful AI takeoff would lie in creating adequate starting conditions... (read more)
A Third Option dissolves a False Dilemma by showing that there are in fact more than two options... (read more)
Eliezer Yudkowsky is a research fellow of the Machine Intelligence Research Institute, which he co-founded in 2001. He is mainly concerned with the obstacles and importance of developing a Friendly AI, such as a reflective decision theory that would lay a foundation for describing fully recursive self modifying agents that retain stable preferences while rewriting their source code. He also co-founded LessWrong, writing the Sequences, long sequences of posts dealing with epistemology, AGI, metaethics, rationality and so on... (read more)
The old LessWrong wiki was a companion wiki site to LessWrong 1.0, it was built on MediaWiki software. As of September 2020, the LessWrong 2.0 team is migrating the contents of the old wiki to LessWrong 2.0's new tag/wiki system. The wiki import is complete... (read more)
The San Francisco Bay Area is a region in the US state of California. Many members of the rationalist community are located there, as are the Machine Intelligence Research Institute and the Center For Applied Rationality.. (read more)
Someone is well-calibrated if the things they predict with X% chance of happening in fact occur X% of the time. Importantly, calibration is not the same as accuracy. Calibration is about accurately assessing how good your predictions are, not making good predictions. Person A, whose predictions are marginally better than chance (60% of them come true when choosing from two options) and who is precisely 60% confident in their choices, is perfectly calibrated. In contrast, Person B, who is 99% confident in their predictions, and right 90% of the time, is more accurate than Person A, but less well-calibrated... (read more)
| User | Post Title | Wikitag | Pow | When | Vote |
Plan A is the optimal governance structure which is supposed to maximize the probability that the ASI's creation results in an excellent future. As far as I am aware, it was first named[1] Plan A in Greenblatt's post Plans A, B, C, and D for misalignment risk.
The AI Futures' version of Plan A is an international deal ruling out any AGI projects unaudited by the Consortium by carefully tracing all or almost all compute in the world. The AI projects audited by the Consortium have research, training runs and safety cases thoroughly studied by outsiders while ensuring that many more people can study frontier models to assess their alignment properties and use them for developing novel techniques.
Additionally, Plan A tries its best to resolve the issues related to misuse, concentration of power, risk of WWIII, disempowerment induced by job loss.
However, attempts to sketch the optimal plan have been made far earlier, see, e.g. Yudkowsky's Six Dimensions of Operational Adequacy in AGI Projects, which, however, assume that creating the AGI will become easy and aligning it is extraordinarily difficult.
The security mindset has much in common with what Eliezer Yudkowsky calls the AI safety mindset.
Here's some fundamental confusions that agent foundations tries to answer:answer, mostly informed by the post/paper Embedded Agency:
In the context of AI alignment, the concern is that a base optimizer (e.g., a gradient descent process) may produce a learned model that is itself an optimizer, and that has unexpected and undesirable properties. Even if the gradient descent process is in some sense "trying" to do exactly what human developers want, the resultant mesa-optimizer will not typically be trying to do the exact same thing.[1]
In 2003 Wei Dai brings up a similar idea in an SL4 thread.thread [2]'"friendly" humans?'.
The optimization daemons articleYudkowsky's "Optimization daemons" was published on Arbital was published probably in 2016.[1]
"Optimization daemons". Arbital.
Wei Dai. '"friendly" humans?' December 31, 2003.
SingluarSingular learning theory is a theory that applies algebraic geometry to statistical learning theory, developed by Sumio Watanabe. Reference textbooks are "the grey book", Algebraic Geometry and Statistical Learning Theory, and "the green book", Mathematical Theory of Bayesian Statistics.
Also notable is Azathoth, the blind idiot god of evolution, from Yudkowsky's An Alien God, which far predates Meditations on MolochMoloch..
Kant's thirdsecond formulation of the categorical imperative lets you build up most of the structure of the key moral ideas from a simple rule: "treat no person as purely a means to an end, but always also as an end in themselves". Many applications of the categorical imperative require baroque derivations to loop back and be justified from this premise (treated as a generative axiom) but "consent ethics" in general, and "slavery is forbidden" are both elementary proofs from this starting point. A slave is a person, turned into a tool and piece of property of another person... a literal "means" to ANY end that the owning person (or "Master") deems desirable and feasible.
Ursula von der Leyen
Mark Carney: p(doom) is "above zero"
Barack Obama: "Nonzero chance" of being wiped out by "killer robots"
António Guterres
Rishi Sunak
Prince Albert II
Naftali Bennett
Ted Lieu (CAIS Signatory)
Audrey Tang (CAIS Signatory)
Bernie Sanders: Bernie Sanders Reveals the AI ‘Doomsday Scenario’ That Worries Top Experts
David Chalmers (CAIS Signatory)
Toby Ord (CAIS Signatory)
Will MacAskill (CAIS(CAIS Signatory)
Others:
Chris Anderson - Dramer-in-Chief, TED (CAIS Signatory)
Lex Fridman (CAIS Signatory)
Grimes (CAIS Signatory)
Stephen Fry (endorsed "If Anyone Builds It, Everyone Dies")
Cenk Uygur: AI's Disturbing Behaviors Will Keep You Up At Night
Paul Tudor Jones: AI poses an imminent threat to humanity in our lifetime
Nate Silver
Steve Bannon
Bill Gates
Historically, slaves would be kept in bondage for life, or for a fixed period of time after which they would be gratedgranted freedom. Many historical cases of enslavement occurred as a result of breaking the law, becoming indebted, suffering a military defeat, or exploitation for cheaper labor; other forms of slavery were instituted along demographic lines such as race or sex.
The issue has gained modern salience given the possible existence of "digital people" who (1) are (1) owned as property, (2) perform cognitive labor, (3) aren't laboring consensually, and (4) don't get pay.
From a Bayesian standpoint this is how we can identify a huge machine strung with superconducting cables as having been produced by high-technology aliens, even before we have any idea of what the machine does. We're saying, "This looks like the product of optimization, a strategy X that the aliens chose to best achieve some unknown goal Y; we can infer this even without knowing Y because many possible Y-goals would concentrate probability into this X-strategy being used."
When you select policy πk because you expect it to achieve a later state Yk (the "goal"), we say that πk is your instrumental strategy for achieving Yk. The observation of "instrumental convergence" is that a widely different range of Y-goals can lead into highly similar π-strategies. (This becomes truer as the Y-seeking agent becomes more instrumentally efficient; two very powerful chess engines are more likely to solve a humanly solvable chess problem the same way, compared to two weak chess engines whose individual quirks might result in idiosyncratic solutions.)
If there's a simple way of classifying possible strategies Π into partitions X⊂Π and ¬X⊂Π, and you think that for most compactly describable goals Yk the corresponding best policies πk are likely to be inside X, then you think X is a "convergent instrumental strategy".
In this case "paperclips", "diamonds", "keeping a button pressed as long as possible", and "sapient beings having fun", would be the goals Y1,Y2,Y3,Y4. The corresponding best strategies π1,π2,π3,π4 for achieving these goals would not be identical - the policies for making paperclips and diamonds are not exactly the same. But all of these policies (we think) would lie within the partition X⊂Π where the superintelligence tries to "transport matter and energy efficiently" (perhaps by using superconducting cables), rather than the complementary partition ¬X where the superintelligence does not try to transport matter and energy efficiently.
If, given our beliefs P about our universe and which policies lead to which real outcomes, we think that in an intuitive sense it sure looks like at least 90% of the utility functions Uk∈UK ought to imply best findable policies πk which lie within the partition X of Π, we'll allege that X is "instrumentally convergent".
X being "instrumentally convergent" doesn't mean that every mind needs an extra, independent drive to...
This looks like the variables are reversed. If V is the intended approximation of U, heavy selection is on high values of V, not U. That selection tends to hit places where V diverges upward from U, so V is an unusually poor approximation of U — not the other way around.