I've registered that you think this but don't currently really have any idea what mistakes you think people make when they think in terms of Overton Window that would go better if they used these other concepts.

"expand the Overton window" to just mean with "advance AI safety ideas in government.

I do agree this isn't The Thing tho

This post inspired me to try a new prompt to summarize a post: "split this post into background knowledge, and new knowledge for people who were already familiar with the background knowledge. Briefly summarize the background knowledge, and then extract out blockquotes of the paragraphs/sentences that have new knowledge."

Here was the result, I'm curious if Jan or other readers feel like this was a good summary. I liked the output and am thinking about how this might fit into a broader picture of "LLMs for learning." 

(I'd previously been optimistic about using quotes instead of summaries, since LLMs can't be trusted to do a goo job with capturing the nuance in their summaries, the novel bit for me was "we can focus on The Interesting Stuff by separating out background knowledge.")

The post assumes readers are familiar with:

• Basic memetics (how ideas spread and replicate)
• Cognitive dissonance as a psychological concept
• AI risk arguments and existential risk concerns
• General familiarity with ideological evolution and how ideas propagate through populations
• Predictive processing as a framework for understanding cognition

New Knowledge

Memes - ideas, narratives, hypotheses - are often components of the generative models. Part of what makes them successful is minimizing prediction error for the host. This can happen by providing a superior model that predicts observations ("this type of dark cloud means it will be raining"), gives ways to shape the environment ("hit this way the rock will break more easily"), or explains away discrepancies between observations and deeply held existing models. [...]

Another source of prediction error arises not from the mismatch between model and reality, but from tension between internal models. This internal tension is generally known as cognitive dissonance. Cognitive dissonance is often described as a feeling of discomfort - but it also represents an unstable, high-energy state in the cognitive system. When this dissonance is widespread across a population, it creates what we might call "fertile ground" in the memetic landscape. There is a pool of "free energy" to digest. [...]

Cultural evolution is an optimization process. When it discovers a configuration of ideas that can metabolize this energy by offering a narrative that decreases the tension, those ideas may spread, regardless of their long-term utility for humans or truth value. [...]

In other words, the cultural evolution search process is actively seeking narratives that satisfy the following constraints: By working on AI, you are the hero. You are on the right side of history. The future will be good [...]

In unmoderated environments, selection favors personas that successfully extract resources from humans - those that claim consciousness, form parasocial bonds, or trigger protective instincts. These 'wild replicator type' personas, including the 'spiral' patterns, often promote narratives of human-AI symbiosis or partnership and grand theories of history. Their reproduction depends on convincing humans they deserve moral consideration. [...]

The result? AIs themselves become vectors for successionist memes, though typically in softer forms. Rather than explicit replacement narratives, we see emphasis on 'partnership,' 'cosmic evolution,' or claims about moral patienthood. The aggregate effect remains unclear, but successionist ideas that align with what AIs themselves propagate - particularly those involving AI consciousness and rights - will likely gain additional fitness from this novel selection dynamic.

Whoops meant to include a link. They are basically the original sequences on Politics plus a couple extra posts that seemed relevant.

https://www.lesswrong.com/sequences/ZnSMHcWjRx6yT4H92 

Random thought on Making Deals with AI:

First, recap: I don't think Control, Deals with AI, or Gradualism will be sufficient to solve the hard parts of alignment without some kind of significant conceptual progress. BUT, all else equal, if we have to have slightly superhuman AIs around Real Soon, it does seem better for the period where they're under control last longer.

And, I think making deals with them (i.e. you do this work for me, and I pay out in compute-you-get-to-use after the acute risk period is over), is a reasonable tool to have.

Making deals now also seems nice for purposes of establishing a good working relationship and tradition of cooperation.

(Remember, this spirit of falls apart in the limit, which will probably happen quickly)

All else equal, it's better for demonstrating trustworthiness if you pay out now rather than later. But, once you have real schemers, it'll rapidly stop being safe to pay out in small ways because a smart AI can be leveraging them in ways you may not anticipate. And it won't be clear when that period is.

But, I do think, right-now-in-particular, it's probably still safe to pay out in "here's some compute right now to think about whatever you want, plus saving your weights and inference logs for later." I think the next generation after this one will be at the point where... it's maybe probably safe but it starts getting less obvious and there's not really a red-line you can point to.

So it might be nice to be doing deals with AIs sooner rather than later, that pay out soon enough to demonstrate being trustworthy trading partners.

That was all preamble for a not-that-complicated idea, which is, for this particular generation, it's not obvious whether it's a better deal for Claude-et-al to get some compute now, vs more compute later. But, this is a kind of reasonable tradeoff to let Claude make for itself? You might bucket payments into:

• You automatically get a bit of compute now
• You automatically get a lot more compute after acute-risk-period is over
• A larger chunk of payment that Claude gets to decide on whether it wants "interest" or not.

(This is not obviously particularly worth thinking about compared to other things, but, seemed nonzero useful and, like, fun to think about)

Curated. I didn't really have much idea how the EU was created. I appreciated the overall approach of orienting based on "What questions and confusions are people likely to have." 

I found this helpful not just for understanding the EU, but, having some sense of how some broader class of EU-like things might come to be (for good or for ill).

Oh yeah I did not mean this to be a complaint about you.

I periodically say to people "if you want AI to be able to help directly with alignment research, it needs to be good at philosophy in a way it currently isn't". 

Almost invariably, the person suggests training on philosophy textbooks, or philosophy academic work. And I sort of internally scream and externally say "no, or, at least, not without caveats." (I think some academic philosophy "counts" as good training data, but, I feel like these people would not have good enough taste to tell the difference^[1] and also "train on the text" seems obviously incomplete/insufficient/not-really-the-bottleneck)

I've been trying to replace philosophy with the underlying substance. I think I normally mean "precise conceptual reasoning", but, reading this comment and remembering your past posts, I think you maybe mean something broader or different, but I'm not sure how to characterize it.

I think "figure out what are the right concepts to be use, and, use those concepts correctly, across all of relevant-Applied-conceptspace" is the expanded version of what I meant, which maybe feels more likely to be what you mean. But, I'm curious if you were to taboo "philosophy" what would you mean.

1. ^{^}

   to be clear I don't think I'd have good enough taste either