The Open Agency Architecture ("OAA") is an AI alignment proposal by (among others) @davidad and @Eric Drexler. .. (read more)
Singular learning theory is a theory that applies algebraic geometry to statistical learning theory, developed by Sumio Watanabe. Reference textbooks are "the grey book", Algebraic Geometry and Statistical Learning Theory, and "the green book", Mathematical Theory of Bayesian Statistics.
Archetypal Transfer Learning (ATL) is a proposal by @whitehatStoic for what is argued by the author to be a fine tuning approach that "uses archetypal data" to "embed Synthetic Archetypes". These Synthetic Archetypes are derived from patterns that models assimilate from archetypal data, such as artificial stories. The method yielded a shutdown activation rate of 57.33% in the GPT-2-XL model after fine-tuning. .. (read more)
Open Threads are informal discussion areas, where users are welcome to post comments that didn't quite feel big enough to warrant a top-level post, nor fit in other posts... (read more)
A Black Marble is a technology that by default destroys the civilization that invents it. It's one type of Existential Risk. AGI may be such an invention, but isn't the only one... (read more)
AI Evaluations focus on experimentally assessing the capabilities, safety, and alignment of advanced AI systems. These evaluations can be divided into two main categories: behavioral and understanding-based... (read more)
AI Risk Skepticism is the view that the potential risks posed by artificial intelligence (AI) are overstated or misunderstood, specifically regarding the direct, tangible dangers posed by the behavior of AI systems themselves. Skeptics of object-level AI risk argue that fears of highly autonomous, superintelligent AI leading to catastrophic outcomes are premature or unlikely.
Slowing Down AI refers to efforts and proposals aimed at reducing the pace of artificial intelligence advancement to allow more time for safety research and governance frameworks. These initiatives can include voluntary industry commitments, regulatory measures, or coordinated pauses in development of advanced AI systems.
Aligned AI Proposals are proposals aimed at ensuring artificial intelligence systems behave in accordance with human intentions (intent alignment) or human values (value alignment)... (read more)
| User | Post Title | Wikitag | Pow | When | Vote |
The ability to persuade at a level beyond the median (or best) human. (this description intentionally left a bit vague since there is little consensus about what this term should mean)
Embedded Agency is the problem that an understanding of the theory of rational agents must account for the fact that the agents we create (and we ourselves) are inside the world or universe we are trying to affect, and not separated from it. This is in contrast with much current basic theory of AI or Rationality (such as Solomonoff induction or Bayesianism) which implicitly supposes a separation between the agent and the-things-the-agent-has-beliefs about. In other words, agents in this universe do not have Cartesian or dualistic boundaries like much of philosophy assumes, and are instead reductionist, that is agents are made up of non-agent parts like bits and atoms. This includes the question of the Boundary of the embedded agent, i.e. the questions which parts an agent considers part of itself and how it delineates other agents from its observations.
The ability to persuade at a level beyond the bestmedian (or best) human. (this description intentionally left a bit vague since there is little consensus about what this term should mean)
OneSince in general Humans can be assigned any values whatsoever), value learning has the pointers problem. It requires that the value can be recovered from the data but cannot prove it.
Practically, the required data has to include sufficient cases of conflict between goals, otherwise, .
People often agree on separate goals until those goals conflict; predicting unconflicted choices does not show how strongly one care should yield to another when both apply.
Another potential challenge is that human values are somewhat mutable and AGI could affect them.
Embedded Agency is the problem that an understanding of the theory of rational agents must account for the fact that the agents we create (and we ourselves) are inside the world or universe we are trying to affect, and not separated from it. This is in contrast with much current basic theory of AI or Rationality (such as Solomonoff induction or Bayesianism) which implicitly supposes a separation between the agent and the-things-the-agent-has-beliefs about. In other words, agents in this universe do not have Cartesian or dualistic boundaries like much of philosophy assumes, and are instead reductionist, that is agents are made up of non-agent parts like bits and atoms. This includes the question of the Boundary of the embedded agent, i.e. the questions which parts an agent considers part of itself and how it delineates other agents (or itself) from its observations.
Formerly Sequent
The pointers problem is about figuring out f.Value learning is an approach to learning f from humans.
Even if we can find f for human values, we would still have to build a system that acts on it and is robust to optimization pressure (Goodhart's law). And these two can fail independently of identifying f.
Achieving total corrigibility everywhereeverywhere, i.e., including in the correction mechanism itself, without manipulating its correctors, and preserving correction through self-modification, via some single, general mental state in which the AI "knows that it is still under construction" or "believes that the programmers know more than it does about its own goals" is termed 'the hard problem of corrigibility'.
Formerly SequentSequent
Website: https://resolution.org
April: i do very much struggle to imagine what we could actually do with, like, a decade, that would lead to us being any more prepared for ASI
Yudkowsky: Iterate rapidly on augmenting adult human intelligence.
Short of that, not much.
A dual operations is to ask what an f reports (eliciting latent knowledge).
The ability to persuade at a level beyond the best human.
Does anyone in particular own this glossary?
I'm asking because I had a dialogue with ChatGPT that compiled a much more extensive and richly sourced glossary. I used to do this sort of thing by hand - see "The New Hacker's Dictionary" - but an LLM makes an almost perfect neutral observer. Mine now exists as a markdown file containing 222 entries from all across the LW/SSC/ACX corpus and navigation paths for newbies.
What should we do about this?