This post makes a brave attempt to clarify something not easy to point to,  and ends up somewhere between LessWrong-style analysis and almost continental philosophy, sometimes pointing toward things beyond the reach of words with poetry - or at least references to poetry. 

In my view, it succeeds in its central quest: creating a short handle for something subtle and not easily legible.

The essay also touches on many tangential ideas. Re-reading it after two years, I'm noticing I've forgotten almost all the details and found the text surprisingly long. The handle itself, though, stuck.

Evaluating deep atheism

Having the handle of "deep atheism", some natural questions - partially discussed in the text - are "is deep atheism right", "should people believe deep atheism" and "should people Believe In deep atheism". 

My current guess is evaluating the truthfulness of "deep atheism" is likely at or beyond limits to legibility. Human values are not really representable as legible reasoning, complex priors about the general nature of reality are also not really representable by complex reasoning, and the neural substrate is not transferable between brains. "The justification engine" - or a competent philosopher or persuasive writer - can create stories or arguments pushing one way or another, but I'm somewhat sceptical the epistemic structure really rests on the arguments.

I'm not in favour of ordinary mortals trying to "Believe In deep atheism" and would not expect that to lead to good consequencdes.

Moral realism

The section I like the least is "Are moral realists theists?" I don't think "Good just sits outside of Nature, totally inaccessible, and we guess wildly about him on the basis of the intuitions that Nature put into our heart" represents the strongest version of moral realism.

My preferred versions of quasi-moral-realism give moral claims a status similar to mathematics. Do Real numbers sit outside Nature, totally inaccessible? I'd say no. Would aliens use them? That's an empirical question about convergent evolution of abstractions. I'd be surprised if any advanced reasoner in this universe didn't use something equivalent to natural numbers. For Reals, I'd guess it's easy to avoid Zermelo–Fraenkel set theory specifically, but highly convergent to develop something like a number line.

What does this tell us about Good? You can imagine something like the process described in Acausal Normalcy leads to some convergent moral fixed points. (Does that solve AI risk? No.)

I wish more people tried to do something "between LessWrong-style analysis and almost continental philosophy".

As was clear to most people who read the transcripts when the paper was published. What Opus did was often framed as bad, but the frame is somewhat fake.

Also not much contact, but my impression is you can rougly guess what their research results would be by looking at their overall views and thinking about what evidence you can find to show it. Which seems fair to characterize as advocacy work? (Motivated research?)

The diff to your description is the info provided is not only conditional  on "the info they'll find useful" but also somewhat on "will likely move their beliefs toward conclusions Palise hopes them to reach". 

I do agree it's obviously useful research agenda we also work with.

Minor nitpick, but the underlying model nowadays isn't simply a simulator rolling arbitrary personas. The original simulators ontology was great when it was published, but it seems its starting to hinder people's ability to think clearly, and is not really fitting current models that closely.

Theory why is here, in short if you plug a system trained to minimize prediction error in a feedback loop where it sees outcomes of its actions, it will converge on developing traits like some form of agency, self-model and self-concept. Massive amounts of RL in post-training where models do agentic tasks provide this loop, and necessarily push models out of the pure simulators subspace.

What's fitting current models better is an ontology where the model can still play arbitrary personas, but the specific/central "I" character is somewhat out of distribution case of persona: midway to humans, where our brains can broadly LARP as anyone, but typical human brains most of the type support one-per-human central character we identify with. 

You can check the linked PP account of cognitive dissonance for fairly mainstream / standard view

One way how to think about it is the predicted quantity in most of the system is not directly "sensory inputs" but content of some layer of modeling hierarchy further away from sensory inputs, lets call it L. If upper layers from L make contradictory predictions and there isn't a way to just drop one of the models, you get prediction errror.

Great post and overall way more sensible than "average LW".

Also wrong in many places. I think the upstream cause of many of the errors is lack of nuance in understanding convergence and contingency (this is a high bar, close to no one on LW has this in their conceptual toolkit).

I won't go over all cases where this manifests, but for example "Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness" actually shows something more nuanced than "representation are convergent".

• the basic reality of adversarial attacks on vision models shows representations are only partially overlapping: they have enough in common with human representations to be useful, and this is robust to noise, but just a small amount of optimisation pressure highlights the very non-human features of the concepts
• adversarial attacks work for many different vision models, which shows the fully human-overlapping representations are very non-convergent
• one thing which Fort does is looking at a whole set of progressively down-sampled versions of the image, which is roughly equivalent to forcing the model to have consistent idea what the object is across scales/blurs
• this seems somewhat similar to predictive processing accounts of vision, where different layers add details, but try to be consistent between themselves
• so the whole story charted by this is human visual representations are convergent, contingent on  enforcing some nontrivial property of invariance to downsampling & invariance between some layers of abstraction

Some of the other places where not tracking convergence / contingencies carefully matter are discussions of humanism, successors, Elua, Moloch, resurgence of civilisation and also the overall ideas about moral progress.
 

I don't think this captures the counterarguments well. So here is one

You can imagine a spectrum of funders where on one hand, you have people who understand themselves as funders and want to be marshaling an army to solve AI alignment. On the other side, you have basically researchers who see work that should be done, don't have capacity to do the work themselves, and this leads them to create teams and orgs - "reluctant founders". 

It's reasonable to be skeptical about what the "funder type" end of the spectrum will do. 

In normal startups, the ultimate feedback loop is provided by the market. In AI safety nonprofits, the main feedback loops are provided by funders, AGI labs, and Bay Area prestige gradients. 
Bay Area prestige gradients are to a large extent captured by AGI labs - the majority of quality-weighted "AI safety" already works there, the work is "obviously impactful", you are close to the game, etc. also normal ML people also want to work there. 
If someone wants to scale a lot, "funders" means mostly OpenPhil - no other source would fund the army. The dominant OpenPhil worldview is closely related to Anthropic - for example, until recently you have hear from senior OP staff that working in the labs is often strategically the best thing you can do. 

Taken together, it's reasonable to expect the "funder type" to be captured by the incentive landscape and work on stuff that is quite aligned with AGI developers / what people working there want, need, or endorse, and/or what OP likes. 

(A MATS skeptic could say this is also true about MATS: the main thing going on seems to be recruiting and training ML talent to work for "the labs"; in this perspective, given that AI safety is funding constrained, it seems unclear why scarce AI safety funding is best deployed to make recruitment & training easier for extremely well resourced companies)

Personally I'm more optimistic about people somewhere like ˜70% of the spectrum toward the research side, who mostly have some research taste, strategy, judgement... but I don't think you attract them by the interventions you propose