• Hello World, AI Doompop
• Annotations for AI Doompop

Embedded Agency is the problem that an understanding of the theory of rational agents must account for the fact that the agents we create (and we ourselves) are inside the world or universe we are trying to affect, and not separated from it. This is in contrast with much current basic theory of AI or Rationality (such as Solomonoff induction or Bayesianism) which implicitly supposes a separation between the agent and the-things-the-agent-has-beliefs about. In other words, agents in this universe do not have Cartesian or dualistic boundaries like much of philosophy assumes, and are instead reductionist, that is agents are made up of non-agent parts like bits and atoms. This includes the question of the Boundary of the embedded agent, i.e. the questions which parts an agent considers part of itself and how it delineates other agents from its observations.

AI Doompop is an emerging genre of music, generally upbeat (eg synthpop) and featuring AI-generated music and/or music videos.

A partial lineage of AI doompop can be found here.

Example 2: Suppose a funder, worried about the prospect of advanced AIs wiping out humanity, startingstarted offering grants for "AI safety". It may then be tempting to try to write papers that you know you can finish, like a paper on robotic cars causing unemployment in the trucking industry, or a paper on who holds legal liability when a factory machine crushes a worker. These have the advantage of being much less difficult problems than those involved in making something actually smarter than you be safe.

Notable posts:

• Hello World, AI Doompop

Agentic Misaligment

Hiding information within seemingly innocuous content.

In AI, it refers to models encoding hidden messages in their outputs, potentially enabling covert communication between models or evading human oversight through being Opaque.

In addition, the Axiom of Choice precludes the possibility of any infinite well ordered chain which includes no subsets without a greatest element (well ordered in both ascending and descending order). 

Let O be a random variable representing observations, and o be some particular value (some specific observations). P() is the prior probability distribution. U is a random variable representing the utility. E is the expectation operator. There is a set of possible actions, A. EDT recommends the following action:^[5]

However, Wei Dai noticed a problem with UDT 1.0, as detailed in Explicit Optimization of Global Strategy (Fixing a Bug in UDT1). In some cases, UDT 1.0 fails to coordinate with itself. The fix is as follows, with π∈Π being policies, IE, functions from observations to actions:

1. ^{^}

   Getting up early to get a good start on the day seems appealing the previous evening, but when the alarm rings, the relative reward of sleeping in another few minutes is larger.

2. ^{^}

   We can more rigorously define dynamic inconsistency as follows:

   If the agent is given the opportunity to commit to a decision early, there are cases where it strictly prefers a different choice than the one it would make in-the-moment.

   In Counterfactual Mugging, we understand Omega as "making a copy" of the agent at some point in time (EG, taking a detailed scan for use in a simulation). If a CDT agent is given the opportunity to commit to a decision in Counterfactual Mugging before this point in time, then it will think of the simulation as being downstream of its decision, so it will make the same decision as UDT. If a CDT agent is asked after the copy is made, but before the coin-flip result is revealed, then the CDT agent will decide to refuse, just like it does after the coin-flip is revealed.

   The analysis for EDT agents is a little bit simpler. If you ask the EDT agent to commit to a decision before the coin-flip is revealed, then it will agree to give up the money just like UDT would, since it sees the cost as smaller than the potential payout, and always thinks of its decision as strongly correlated with Omega's simulation.^[6] If the EDT agent is asked after the coin-flip, then EDT refuses.

3. ^{^}

   This supposes that the agent isn't doing any anthropic reasoning; specifically, they don't put any probability on the possibility that they themselves are the agent inside Omega's hypothetical simulation. If the agent reasons anthropically, CDT and EDT can reach the same conclusion as UDT.

4. ^{^}

   There are several different theories of causality, meaning there is a different version of CDT for each alternative.

5. ^{^}

   In contrast, CDT chooses the following:

   
   choice(o):=argmaxa∈AEP(U|do(choice(o)=a),O=o)

   The only difference between this and CDT is that the "do" operator has been wrapped around the choice of action. This notation says that the action is an "intervention" rather than a normal Bayesian conditioning. In the context of a causal Bayesian network, we cut the parents of variables with a "do" before we make inferences (again assuming Judea Pearl's theory of causality, rather than alternatives).

6. ^{^}

   Note that this is a questionable assumption: EDT need not think its decision is correlated with the simulation. This is merely a typical part of the problem set-up.

• If we try to suspend the AI to disk, or shut it down entirely, a corrigible AI will let us do so. (Even though, if suspended, the AI will then be unable to fulfill what would usually be its goals.) Shutting down the AI also has to stop processes and subagents it has caused.
• If we try to reprogram the AI's utility function or meta-utility function, a corrigible AI will allow this modification to go through. (Rather than, e.g., fooling us into believing the utility function was modified successfully, while the AI actually keeps its original utility function as obscured functionality; as we would expect by default to be a preferred outcome according to the AI's current preferences.)

Achieving total corrigibility everywhereeverywhere, i.e., including in the correction mechanism itself, without manipulating its correctors, and preserving correction through self-modification, via some single, general mental state in which the AI "knows that it is still under construction" or "believes that the programmers know more than it does about its own goals" is termed 'the hard problem of corrigibility'.

Could the Axiom of Choice, like the Axiom of Parallel lines, have physical implications? If it is false in our universe, then perhaps the Copenhagen interpretation of quantum mechanics must be false, as it involves a notion of true randomness. The axiom of choice is the only axiom of set theory which is incompatible with a constructivist approach to mathematics in which proving that anything exists requires an algorithm for its construction, which suggests that it is impossible to prove the existence of a randomly selected set. In such a universe, even selecting a single arbitrary element from an infinite set without an algorithm might be impossible, although proving that one exists within Z F does not rely upon the Axiom of Choice. This would seem to be incompatible with a particle collapsing into a partially random position.

re "The"The Axiom of Choice is necessary to select a set from an infinite number of pairs of socks, but not an infinite number of pairs of shoes." — Bertrand Russell, Introduction to mathematical philosophy

The axiom states that a certain kind of function, called a `choice' function, always exists. It is called a choice function, because, given a collection of non-empty sets, the function 'chooses' a single element from each of the sets. It is a powerful and useful axiom, asserting the existence of useful mathematical structures (such as bases for vector spaces of arbitrary dimension, and ultraproducts). It is a generally accepted axiom, and is in wide use by mathematicians. In fact, according to Elliott Mendelson in Introduction to Mathematical Logic (1964) "The status of the Axiom of Choice has become less controversial in recent years. To most mathematicians it seems quite plausible and it has so many important applications in practically all branches of mathematics that not to accept it would seem to be a wilfulwillful hobbling of the practicing mathematician. "

This is the statmentstatement that taking the mathematical product of non-empty sets will always yield a non-empty set. Consider infinitely many sets  indexed by the natural numbers. Then an element of the product  is a family (essentially an infinite tuple) of the form  where ,  and so on. However, to select such a tuple in the product amounts to selecting a single element from each of the sets. Hence, without the axiom of choice, it is not provable that there are any elements in this product.

A suprisingly good video explanation in laymanslayman's terms by Vsauce can be found Here.

This theorem is the main counterexample used as evidence of the falsehood of the Axiom of Choice. If not taken as evidence of its falsehood, thsithis is at least used as evidence of the counter-intuitiveness of AC.

Yet, mathematicians do disagree about this axiom, and whether it applies to our world as well. Some mathematicians take a Platonic view of mathematics, in which mathematical objects such as sets actually exist in some abstract realm, and for which the axiom of choice is either true or false, but we do not know which. Others take a highly constructive view (in many cases motivated by realism and the ability of the mathematics to model the world) in which either the axiom of choice is false, or infinite sets do not exist in which case the axiom of choice is provable and hence superfluous. Others take the view that the axiom is not true or false, but merely useless, and that anything provable from it is meaningless. Many seem not to care: the axiom is convenient for the mathematics they wish to do (whose application they are not...

In addition, the Axiom of Choice precludes the possibility of any infinite well ordered chain which includes no subsets without a greatest element (well ordered in both ascending and descending order).