The LLMs might be picking up the spiral symbolism from Spiral Dynamics.

I follow a hardline no-Twitter policy. I don't visit Twitter at all, and if a post somewhere else has a screenshot of a tweet I'll scroll past without reading it. There are some writers like Zvi whom I've stopped reading because their writing is too heavily influenced by Twitter and quotes too many tweets.

I was describing reasoning about idealized superintelligent systems as the method used in agent foundations research, rather than its goal. In the same way that string theory is trying to figure out "what is up with elementary particles at all," and tries to answer that question by doing not-really-math about extreme energy levels, agent foundations is trying to figure out "what is up with agency at all" by doing not-really-math about extreme intelligence levels.

If you've made enough progress in your research that it can make testable predictions about current or near-future systems, I'd like to see them. But the persistent failure of agent foundations research to come up with any such bridge between idealized models and real-world system has made me doubtful that the former are relevant to the latter.

For me, the OP brought to mind another kind of "not really math, not really science": string theory. My criticisms of agent foundations research are analogous to Sabine Hossenfelder's criticisms of string theory, in that string theory and agent foundations both screen themselves off from the possibility of experimental testing in their choice of subject matter: the Planck scale and very early universe for the former, and idealized superintelligent systems for the latter. For both, real-world counterparts (known elementary particles and fundamental forces; humans and existing AI systems) of the objects they study are primarily used as targets to which to overfit their theoretical models. They don't make testable predictions about current or near-future systems. Unlike with early computer science, agent foundations doesn't come with an expectation of being able to perform experiments in the future, or even to perform rigorous observational studies.

Building on what you said, pre-LLM agent foundations research appears to have made the following assumptions about what advanced AI systems would be like:

1. Decision-making processes and ontologies are separable. An AI system's decision process can be isolated and connected to a different world-model, or vice versa.
2. The decision-making process is human-comprehensible and has a much shorter description length than the ontology.
3. As AI systems become more powerful, their decision processes approach a theoretically optimal decision theory that can also be succinctly expressed and understood by human researchers.

None of these assumptions ended up being true of LLMs. In an LLM, the world-model and decision process are mixed together in a single neural network instead of being separate entities. LLMs don't come with decision-related concepts like "hypothesis" and "causality" pre-loaded; those concepts are learned over the course of training and are represented in the same messy, polysemantic way as any other learned concept. There's no way to separate out the reasoning-related features to get a decision process you could plug into a different world-model. In addition, when LLMs are scaled up, their decision-making becomes more complex and inscrutable due to being distributed across the neural network. The LLM's decision-making process doesn't converge into a simple and human-comprehensible decision theory.

More useful. It would save us the step of having to check for hallucinations when doing research.

Another example of this pattern that's entered mainstream awareness is tilt. When I'm playing chess and get tilted, I might think things like "all my opponents are cheating, "I'm terrible at this game and therefore stupid," or "I know I'm going to win this time, how could I not win against such a low-rated opponent." But if I take a step back, notice that I'm tilted, and ask myself what information I'm getting from the feeling of being tilted, I notice that it's telling me to take a break until I can stop obsessing over the result of the previous game.

Tilt is common, but also easy to fix once you notice the pattern of what it's telling you and start taking breaks when you experience it. The word "tilt" is another instance of a hangriness-type stance that's caught on because of its strong practical benefits--having access to the word "tilt" makes it easier to notice.

It's working now. I think the problem was on my end.

This strategy suggests that decreasing ML model sycophancy should be a priority for technical researchers. It's probably the biggest current barrier to the usefulness of ML models as personal decision-making assistants. Hallucinations are probably the second-biggest barrier.

The new feed doesn't load at all for me.