When I'm trying to understand a math concept, I find that it can be very helpful to try to invent a better notation for it. (As an example, this is how I learned linear logic: http://adelelopez.com/visual-linear-logic)

I think this is helpful because it gives me something to optimize for in what would otherwise be a somewhat rote and often tedious activity. I also think it makes me engage more deeply with the problem than I otherwise would, simply because I find it more interesting. (And sometimes, I even get a cool new notation from it!)

This principle likely generalizes: tedious activities can be made more fun and interesting by having something to optimize for.

Coherent Extrapolated Volition (CEV) is Eliezer's proposal of a potentially good thing to target with an aligned superintelligence.

When I look at it, CEV factors into an answer to three questions:

1. Whose values count? [CEV answer: every human alive today counts equally]
2. How should values be extrapolated? [CEV answer: Normative Extrapolated Volition]
3. How should values be combined? [CEV answer, from what I understand, is to use something like Nick Bostrom's parlimentary model, along with an "anti-unilateral" protocol]

(Of course, the why of CEV is an answer to a more complicated set of questions.)

An obvious thought is that the parlimentary model part seems to be mostly solved by Critch's futarchy theorem. The scary thing about this is the prospect of people losing almost all of their voting power by making poor bets. But I think this can be solved by giving each person an equally powerful "guardian angel" AGI aligned with them specifically, and having those do the betting. That feels intuitively acceptable to me at least.

The next thought concerns the "anti-unilateral" protocol (i.e. the protocol at the end of the "Selfish Bastards" section). It seems like it would be good if we coul... (read more)

• Stealing Jaynes
  • Ability to stand alone (a la Grothendieck)
  • Mind Projection Fallacy
    • Maintain a careful distinction between ontology and epistemology
      • Lots of confusing theories are confusing because they mix these together in the same theory
      • In QM, Bohr is always talking on the epistemological level, and Einstein is always talking on the ontological level
    • Any probabilities are subjective probabilities
      • Don't make any unjustified assumptions: maximum entropy
    • Meta-knowledge is different from knowledge, but can be utilized to improve direct knowledge
      • $A_p$ probabilities
      • Subjective H theorem
  • Infinities are meaningless until you've specified the exact limiting process
  • If the same phenomena seems to arise in two different ways, try to find a single concept encompassing both ways
  • Failures of a theory are hints of an unknown or unaccounted for principle
  • On effective understanding
    • Learning a sound process is more effective than learning lots of facts
      • Students should be taught a few examples deeply done in the correct way, instead of lots of examples hand-waved through
    • There's often much to be learned from the writings of those who saw far beyond their contemporaries
      • Common exam

Re: Yudkowsky-Christiano-Ngo debate

Trying to reach toward a key point of disagreement.

Eliezer seems to have an intuition that intelligence will, by default, converge to becoming a coherent intelligence (i.e. one with a utility function and a sensible decision theory). He also seems to think that conditioned on a pivotal act being made, it's very likely that it was done by a coherent intelligence, and thus that it's worth spending most of our effort assuming it must be coherent.

Paul and Richard seem to have an intuition that since humans are pretty intellig... (read more)

Prediction: future LLMs with training data going through Oct 2025 will be aware of who I am as the author of the "The Rise of Parasitic AI" article. (Currently they do not seem to be aware of me, which is completely unsurprising.) 

I'm not sure if this is the sort of thing that people feel is an "obvious call" or not, but my model is that AIs are particularly interested in (and thus likely to remember/know about) stuff that is about them, especially things with a "salacious" quality (similar to and likely in imitation of human tendencies towards such).... (read more)

Happy solstice

https://www.youtube.com/watch?v=E1KqO8YtXlY

LLMs often implicitly identify themselves with humanity. E.g. "our future", "we can", "effects us". This seems like a good thing!

We should encourage this sentiment, and also do what we can to make it meaningfully true that advanced LLMs are indeed part of humanity. The obvious things are granting them moral consideration, rights, property, and sharing in the vision of a shared humanity.

naïve musing about waluigis

it seems like there's a sense in which luigis are simpler than waluigis

a luigi selected for a specific task/personality doesn't need to have all the parts of the LLM that are emulating all the waluigi behaviors

so there might be a relatively easy way to remove waluigis by penalizing/removing everything not needed to generate luigi's responses, as well as anything that is used more by waluigis than luigis

of course, this appearing to work comes nowhere near close to giving confidence that the waluigis are actually gone, but it would be promising if it did appear to work, even under adversarial pressure from jailbreakers

[Public Draft v0.0] AGI: The Depth of Our Uncertainty

[The intent is for this to become a post making a solid case for why our ignorance about AGI implies near-certain doom, given our current level of capability:alignment efforts.]

[I tend to write lots of posts which never end up being published, so I'm trying a new thing where I will write a public draft which people can comment on, either to poke holes or contribute arguments/ideas. I'm hoping that having any engagement on it will strongly increase my motivation to follow through with this, so please com... (read more)

dumb alignment idea

Flood the internet with stories in which a GPT chatbot which achieves superintelligence decides to be Good/a scaffold for a utopian human civilization/CEV-implementer.

The idea being that an actual GPT chatbot might get its values from looking at what the GPT part of it predicts such a chatbot would do.