When I'm trying to understand a math concept, I find that it can be very helpful to try to invent a better notation for it. (As an example, this is how I learned linear logic: http://adelelopez.com/visual-linear-logic)

I think this is helpful because it gives me something to optimize for in what would otherwise be a somewhat rote and often tedious activity. I also think it makes me engage more deeply with the problem than I otherwise would, simply because I find it more interesting. (And sometimes, I even get a cool new notation from it!)

This principle likely generalizes: tedious activities can be made more fun and interesting by having something to optimize for.

Coherent Extrapolated Volition (CEV) is Eliezer's proposal of a potentially good thing to target with an aligned superintelligence.

When I look at it, CEV factors into an answer to three questions:

1. Whose values count? [CEV answer: every human alive today counts equally]
2. How should values be extrapolated? [CEV answer: Normative Extrapolated Volition]
3. How should values be combined? [CEV answer, from what I understand, is to use something like Nick Bostrom's parlimentary model, along with an "anti-unilateral" protocol]

(Of course, the why of CEV is an answer to a more complicated set of questions.)

An obvious thought is that the parlimentary model part seems to be mostly solved by Critch's futarchy theorem. The scary thing about this is the prospect of people losing almost all of their voting power by making poor bets. But I think this can be solved by giving each person an equally powerful "guardian angel" AGI aligned with them specifically, and having those do the betting. That feels intuitively acceptable to me at least.

The next thought concerns the "anti-unilateral" protocol (i.e. the protocol at the end of the "Selfish Bastards" section). It seems like it would be good if we could formalize the "anti-unilateral-selfishness" part of it and bake it into something like Critch's futarchy theorem, instead of running a complicated protocol.

• Stealing Jaynes
  • Ability to stand alone (a la Grothendieck)
  • Mind Projection Fallacy
    • Maintain a careful distinction between ontology and epistemology
      • Lots of confusing theories are confusing because they mix these together in the same theory
      • In QM, Bohr is always talking on the epistemological level, and Einstein is always talking on the ontological level
    • Any probabilities are subjective probabilities
      • Don't make any unjustified assumptions: maximum entropy
    • Meta-knowledge is different from knowledge, but can be utilized to improve direct knowledge
      • $A_p$ probabilities
      • Subjective H theorem
  • Infinities are meaningless until you've specified the exact limiting process
  • If the same phenomena seems to arise in two different ways, try to find a single concept encompassing both ways
  • Failures of a theory are hints of an unknown or unaccounted for principle
  • On effective understanding
    • Learning a sound process is more effective than learning lots of facts
      • Students should be taught a few examples deeply done in the correct way, instead of lots of examples hand-waved through
    • There's often much to be learned from the writings of those who saw far beyond their contemporaries
      • Common examples
        • Jeffreys
        • Gibbs
        • Laplace
    • Conceptual confusion impedes further progress
    • Don't let rigor get in the way of understanding
  • Toolkit
    • Lagrangian multipliers
      • should be paired with technique described in https://bayes.wustl.edu/etj/science.and.engineering/lect.10.pdf
    • Bayes' theorem
    • Maximum Entropy

Re: Yudkowsky-Christiano-Ngo debate

Trying to reach toward a key point of disagreement.

Eliezer seems to have an intuition that intelligence will, by default, converge to becoming a coherent intelligence (i.e. one with a utility function and a sensible decision theory). He also seems to think that conditioned on a pivotal act being made, it's very likely that it was done by a coherent intelligence, and thus that it's worth spending most of our effort assuming it must be coherent.

Paul and Richard seem to have an intuition that since humans are pretty intelligent without being particularly coherent, it should be possible to make a superintelligence that is not trying to be very coherent, which could be guided toward performing a pivotal act.

Eliezer might respond that to the extent that any intelligence is capable of accomplishing anything, it's because it is (approximately) coherent over an important subdomain of the problem. I'll call this the "domain of coherence". Eliezer might say that a pivotal act requires having a domain of coherence over pretty much everything: encompassing dangerous domains such as people, self, and power structures. Corrigibility seems to interfere with coherence, which makes it very difficult to design anything corrigible over this domain without neutering it.

From the inside, it's easy to imagine having my intelligence vastly increased, but still being able and willing to incoherently follow deontological rules, such as Actually Stopping what I'm doing if a button is pressed. But I think I might be treating "intelligence" as a bit of a black box, like I could still feel pretty much the same. However, to the extent where I feel pretty much the same, I'm not actually thinking with the strategic depth necessary to perform a pivotal act. To properly imagine thinking with that much strategic depth, I need to imagine being able to see clearly through people and power structures. What feels like my willingness to respond to a shutdown button would elide into an attitude of "okay, well I just won't do anything that would make them need to stop me" and then into "oh, I see exactly under what conditions they would push the button, and I can easily adapt my actions to avoid making them push it", to the extent where I'm no longer being being constrained by it meaningfully. From the outside view, this very much looks like me becoming coherent w.r.t the shutdown button, even if I'm still very much committed to responding incoherently in the (now extremely unlikely) event it is pushed.

And I think that Eliezer foresees pretty much any assumption of incoherence that we could bake in becoming suspiciously irrelevant in much the same way, for any general intelligence which could perform a pivotal act. So it's not safe to rely on any incoherence on part of the AGI.

Sorry if I misconstrued anyone's views here!

naïve musing about waluigis

it seems like there's a sense in which luigis are simpler than waluigis

a luigi selected for a specific task/personality doesn't need to have all the parts of the LLM that are emulating all the waluigi behaviors

so there might be a relatively easy way to remove waluigis by penalizing/removing everything not needed to generate luigi's responses, as well as anything that is used more by waluigis than luigis

of course, this appearing to work comes nowhere near close to giving confidence that the waluigis are actually gone, but it would be promising if it did appear to work, even under adversarial pressure from jailbreakers

[Public Draft v0.0] AGI: The Depth of Our Uncertainty

[The intent is for this to become a post making a solid case for why our ignorance about AGI implies near-certain doom, given our current level of capability:alignment efforts.]

[I tend to write lots of posts which never end up being published, so I'm trying a new thing where I will write a public draft which people can comment on, either to poke holes or contribute arguments/ideas. I'm hoping that having any engagement on it will strongly increase my motivation to follow through with this, so please comment even if just to say this seems cool!]

[Nothing I have planned so far is original; this will mostly be exposition of things that EY and others have said already. But it would be cool if thinking about this a lot gives me some new insights too!]

Entropy is Uncertainty

Given a model of the world, there are lots of possibilities that satisfy that model, over which our model implies a distribution.

There is a mathematically inevitable way to quantify the uncertainty latent in such a model, called entropy.

A model is subjective in the sense that it is held by a particular observer, and thus entropy is subjective in this sense too. [Obvious to Bayesians, but worth spending time on as it seems to be a common sticking point]

This is in fact the same entropy that shows up in physics!

Engine Efficiency

But wait, that implies that temperature (defined from entropy) is subjective, which is crazy! After all, we can measure temperature with a thermometer. Or define it as the average kinetic energy of the particles (in a monoatomic gas, in other cases you need the potential energy from the bonds)! Those are both objective in the sense of not depending on the observer.

That is true, as those are slightly different notions of temperature. The objective measurement is the one important for determining whether something will burn your hand, and thus is the one which the colloquial sense of temperature tracks. But the definition entropy is actually more useful, and it's more useful because we can wring some extra advantage from the fact that it is subjective.

And that's because, it is this notion of temperature which governs the use of a engine. Without the subjective definition, we merely get the law of a heat engine. As a simple intuition, consider that you happen to know that your heat source doesn't just have molecules moving randomly, but that they are predominantly moving back and forth along a particular axis at a specific frequency. The temperature of a thermometer attached to this may measure the same temperature as an ordinary heat sink with the same amount of energy (mediated by phonon dissipation), and yet it would be simple to create an engine using this "heat sink" exceeding the Carnot limit simply by using a non-heat engine which takes advantage of the vibrational mode!

Say that this vibrational mode was hidden or hard to notice. Then someone with the knowledge of it would be able to make a more effective engine, and therefore extract more work, than someone who hadn't noticed.

Another example is Maxwell's demon. In this case, the demon has less uncertainty over the state of the gas than someone at the macro-level, and is thereby able to extract more work from the same gas.

But perhaps the real power of this subjective notion of temperature comes from the fact that the Carnot limit still applies with it, but now generalized to any kind of engine! This means that there is a physical limit on how much work can be extracted from a system which directly depends on your uncertainty about the system!! [This argument needs to actually be fleshed out for this post to be convincing, I think...]

The Work of Optimization

[Currently MUCH rougher than the above...]

Hopefully now, you can start to see the outlines of how it is knowable that

Try to let go of any intuitions about "minds" or "agents", and think about optimizers in a very mechanical way.

Physical work is about the energy necessary to change the configuration of matter.

Roughly, you can factor an optimizer into three parts: The Modeler, the Engine, and the Actuator. Additionally, there is the Environment the optimizer exists within and optimizes over. The Modeler models the optimizer's environment - decreasing uncertainty. The Engine uses this decreased uncertainty to extract more work from the environment. The Actuator focuses this work into certain kinds of configuration changes.

[There seems to be a duality between the Modeler and the Actuator which feels very important.]

Examples:

Gas Heater

• It is the implicit knowledge of the location, concentration, and chemical structure of a natural gas line that allow the conversion of natural gas and the air in the room to state from a state of both being at the same low temperature to a state where the air is at a higher temperature, and the gas has been burned.

-- How much work does it take to heat up a room? -- How much uncertainty is there in the configuration state before and after combustion?

This brings us to an important point. A gas heater still works with no one around to be modeling it. So how is any of the subjective entropy stuff relevant? Well, from the perspective of no one - the room is simply in one of a plethora of possible states before, and it is in another of those possible states after, just like any other physical process anywhere. It is only because of the fact that we find it somehow relevant that the room is hotter before than after that thermodynamics comes into play. The universe doesn't need thermodynamics to make atoms bounce around, we need it to understand and even recognize it as an interesting difference.

Thermostat

Bacterium

Natural Selection

Chess Engine

Human

AI

Why Orthogonality?

[More high level sections to come]

dumb alignment idea

Flood the internet with stories in which a GPT chatbot which achieves superintelligence decides to be Good/a scaffold for a utopian human civilization/CEV-implementer.

The idea being that an actual GPT chatbot might get its values from looking at what the GPT part of it predicts such a chatbot would do.