• How can a mind reason about a world too large to consider in its entirety? Perhaps we should look at what sorts of abstractions they'd use; perhaps we need new probability theories like infra-bayesianism
• How can a mind reason when it does not know all of the implications of its beliefs (that is, it has logical uncertainty)?
• How can a mind reason about (possibly logical) counterfactuals to make decisions?
• How can a mind reason about itself, or improve itself?
• What even is an "agent"? What sorts of agents should we expect to be selected for by evolution or gradient descent or any other selection process?
• What are "goals"? How do we formalize a goal that's about the world, instead of our beliefs about the world (or our observations)? Even something as simple as "maximize the number of diamond atoms" has no clear route to formalization!

The ML Alignment & Theory ScholarsMachine Alignment, Transparency, and Security (MATS) Program is an independent research and educational seminar and independent research program that aims to provide talented scholarsprovides emerging researchers with mentorship, talks, workshops, and research mentorship in the field of AI alignment,workshops and connectconnects them with the BerkeleySF Bay Area and London AI safety research community.communities.

There are fundamental confusions about intelligent agents, that is, about minds that try to make stuff that they want happen. Some believe that working out these fundamental confusions is necessary for AI alignment. Others prefer more prosaic approaches; or something else not mentioned.

Here's some fundamental confusions that agent foundations tries to answer:

• How can a mind reason about a world too large to consider in its entirety? Perhaps we should look at what sorts of abstractions they'd use; perhaps we need new probability theories like infra-bayesianism
• How can a mind reason when it does not know all of the implications of its beliefs (that is, it has logical uncertainty)?
• How can a mind reason about (possibly logical) counterfactuals to make decisions?
• What even is an "agent"? What sorts of agents should we expect to be selected for by evolution or gradient descent or any other selection process?
• What are "goals"? How do we formalize a goal that's about the world, instead of our beliefs about the world (or our observations)? Even something as simple as "maximize the number of diamond atoms" has no clear route to formalization!

Prosaic alignment is an approach to AI alignment.

Payor's lemma is a theorem in mathematical logic that is similar to Löb's theorem.

For the purposes of Agent Foundations, Payor's lemma has been proposed as an alternative to Löb's theorem, due to both being simpler and possibly having a probabilistic generalization in a way that breaks for Löb's theorem. If it works out, this would provide a way for agents to do the probabilistic version of logical decision theory type stuff like cooperate in the Prisoner's dilemma when given each other's source code, this time with uncertainty.

The Supervised Program for Alignment Research (SPAR) is a part-time, remote research program hosted by Kairos AI pairing aspiring researchers with established researchers working to mitigate the risks posed by advanced AI, focusing in particular on AI safety and security, interpretability, biosecurity, AI policy and societal impacts. 

Training on narrow examples of misaligned behavior sometimes extrapolates to broadly misaligned behavior, seemingly altering the assistant's goals or persona rather than just training on that specific behavior

• Despite being around as an idea for 20 years, futarchy hasn't happened. Prediction markets aren't clearly more accurate than Metaculus and those markets that currently exist generally aren't useful for decision makers.
  • Michael Story puts it well (emphasis his). It's easy to think that prediction markets tell us a lot about the world. But maybe instead they tell us who is a good bettor and who isn't. And perhaps we should hire those people to be grant funders or decision makers, but it's not clear we should use the process to make decisions.

1. Let anyone create and participate in decision markets for advisory purposes. (Right now, prediction markets are currently heavily regulated and largely illegalin places like the United States; Hanson proposes to change this.) In particular, we would be able to set up a prediction market in which future outcomes of interest (e.g., future scores on some measure of welfare) are bet on, conditional on certain policies being accepted. The markets have some time to settle on prices, and then, ideally, for important policies, prices conditional on the policy being accepted will be significantly different from prices conditional on the policy being rejected. This is an indicator that (Hanson suggests) would describe the effect traders expect a certain policy to have on relevant outcomes. Policymakers can then consult this information when deciding whether to accept or reject the policy in question.
2. Let markets veto proposed bills. Per the proposal, an official measure of welfare is voted on and maintained by legislatures. A bill cannot become law if a market estimate of national welfare conditional on the bill becoming law is clearly lower than the market estimate of national welfare given the status quo (the bill not becoming law). (Futarchy, described below, is a larger and more complicated version of this.)

Metacognition Is thinking about thinking - the capacity to monitor, evaluate, and govern one's own reasoning while it is happening rather than only after the fact.

• Consequentialism, or choosing actions/policies on the basis of their expected future consequences
  • Modeling the conditional relationship P(Y|X) and selecting an X such that it leads to a high probability of Y or high quantitative degree of Y, is ceteris paribus a sufficient precondition for deploying Convergent instrumental strategies that lie within the effectively searchable range of X.
  • Note that selecting over a conditional relationship is potentially a property of many internal processes, not just the entire AI's top-level main loop, if the conditioned variable is being powerfully selected over a wide range.
  • Cross-domain consequentialism implies many different cognitive domains potentially lying within the range of the X being selected-on to achieve Y.
  • Trying to rule out particular instrumental strategies,strategies in the presence of increasingly powerful consequentialism,consequentialism would lead to the nearest unblocked strategy form of patch resistance and subsequent context-change disasters.
• Big-picture strategic awareness is a world-model that includes strategically important general facts about the larger world, such as e.g. "I run on computing hardware" and "I stop running if my hardware is switched off" and "there is such a thing as the Internet and it connects to more computing hardware".
• Psychological modeling of other agents (not humans per se) potentially leads to:
  • Extrapolating that its programmers may present future obstacles to achieving its goals
  • This in turn leads to the host of problems accompanying incorrigibility as a convergent strategy.
  • Trying to conceal facts about itself from human operators
  • Being incentivized to engage in cognitive steganography.
  • Mindcrime if building models of reflective other agents, or itself.
  • Internally modeled adversaries breaking out of internal sandboxes.
  • Modeling distant superintelligences or other decision-theoretic adversaries.
• Substantial capability gains relative to domains trained and verified previously.
  • E.g. this is the qualifying property for many context-change disasters.
• General intelligence is the most obvious route to an AI acquiring many of the capabilities above or below, especially if those capabilities were not initially or deliberately programmed into the AI.
• Self-improvement is another route that potentially leads to capabilities not previously present. While some hypotheses say that self-improvement is likely to require basic general intelligence, this is not a known fact and the two advanced properties are conceptually distinct.
• Programming or computer science capabilities are a route potentially leading to self-improvement, and may also enable cognitive steganography.
• Turing-general cognitive elements (capable of representing large computer programs), subject to sufficiently strong end-to-end optimization (whether by the AI or by human-crafted clever algorithms running on 10,000 GPUs), may give rise to crystallized agent-like processes within the AI.
  • E.g. natural selection, operating on chemical machinery constructible by DNA strings, optimized some DNA strings hard enough to spit out humans.
• Pivotal material capabilities such as quickly self-replicating infrastructure, strong mastery of biology, or molecular nanotechnology.
  • Whatever threshold level of domain-specific engineering acumen suffices to develop those capabilities, would therefore also qualify as an advanced-agent property.

• Cognitive uncontainability is when we can't effectively imagine or search the AI's