Hot take:

I've been experiencing more "frame lock-in" with more sophisticated models. I recently experienced it with Sonnet 4.5 so I want to share a prediction that models will grow more "intelligent" (capable of reasoning within frames) whilst they have a harder time with changing frames. There's research on how more intelligent people become better at motivated reasoning and interpreting things within existent frames and it seems like LLMs might inhabit similar biases.

Could someone please safety pill the onion? I think satire is the best way to deal with people being really stupid and so I want more of this as an argument when talking with the e/acc gang: https://youtu.be/s-BducXBSNY?si=j5f8hNeYFlBiWzDD

(Also if they already have some AI stuff, feel free to link that too)

On character alignment for LLMs.

I would like to propose that we think of a John Rawls style original position (https://en.wikipedia.org/wiki/Original_position) as one view when looking at character prompting for LLMs. More specifically I would want you to imagine that you're on a social network or similar and that you're put into a word with mixtures of AI and human systems, how do you program the AI in order to make the situation optimal? You're a random person among all of the people, this means that some AIs are aligned to you some are not. Most likely, the majority of AIs will be run by larger corporations since the amount of AIs will be proportional to the power you have.

How would you prompt each LLM agent? What are their important characteristics? What happens if they're thought of as "tool-aligned"?

If we're getting more internet based over time and AI systems are more human in that they can flawlessly pass the turing test, I think the veil of ignorance style thinking becomes more and more applicable.

Think more of how you would design a societly of LLMs and what if the entire society of LLMs had this alignment rather than just the individual LLM.

Prediction & Warning:

There are lots of people online who have started to pick up the word "clanker" in order to protest against AI systems. This word and sentiment is on the rise and I think that this will be a future schism in the more general anti-AI movement. The warning part here is that I think that the Pause movement and similar can likely get caught up in a general anti AI system speciesism.

Given that we're starting to see more and more agentic AI systems with more continous memory as well as more sophisticated self modelling, the basic foundations for a lot of the existing physicalist theories of consciousness are starting to be fulfilled. Within 3-5 years I find it quite likely that AIs will at least have some sort of basic sentience that we can almost basically prove (given IIT or GNW or another physicalist theory).

This could potentially be one of the largest suffering risks that we've seen that we're potentially inducing on the world. When you're using a word like "clanker", you're essentially demonizing that sort of a system. Right now it's generally fine as it's currently about a sycophantic non-agentic chatbot and so it's fine as an anti measure to some of the existing thoughts of AIs being conscious but it is likely a slippery slope?

More generally, I've seen a bunch of generally kind and smart AI Safety people have quite an anti-AI species sentiment in terms of how to treat these sorts of systems. From my perspective, it feels a bit like it comes from a place of fear and distrust which is completely understandable as we might die if anyone builds a superintelligent AI.

Yet that fear of death shouldn't stop us from treating potential conscious beings kindly?

A lot of racism or similar can be seen as coming from a place of fear, the aryan master race was promoted because of the idea that humanity would go extinct if we got worse genetics into the system. What's the difference from the idea that AIs might share our future lightcone?

The general argument goes that this time it is completely different since the AI can self-replicate, edit it's own software, etc. This is a completely reasonable argument as there's a lot of risks involved with AI systems.

It is when we get to the next part that I see a problem. The argument that follows is: "Therefore, we need to keep the almight humans in control to wisely guide the future of the lightcone."

Yet, there's generally a lot more variance within a distribution of humans compared to variance between distributions.

So when someone says that we need humans to remain in control, I think: "mmm, yes the totally homogenous group of "humans" that don't include people like hitler, polpot and stalin". And for the AI side of things we also have the same: "Mmm, yes the totally homogenous group of "all possible AI systems" that should be kept away so that the "wise humans" can remain in control." Because a malignant RSI system is the only future AI based system that can be thought of, there is no way to change the system so that it values cooperation and there is no other way for future AI development to go than a quick take-off where an evil AI takes over the world.

Yes, there are obviously things that AIs can do that humans can do but don't demonize all possible AI systems as a consequence, it is not black and white. We can protect ourselves against recursively self-improving AI and at the same time respect AI sentience, we can hold at the surface level contradictory statements at the same time?

So let's be very specific about our beliefs and let's make sure that our fear does not guide us into a moral catastrophe whether it be the extinction of all future life on earth nor a capture of sentient beings into a future of slavery?

I wanted to register some predictions and bring this up as I haven't seen that many discussions on it. Finally, politics is war and arguments are soldiers so let's keep it focused on the something object level? If you disagree, please tell me the underlying reasons. Finally in that spirit, here's a set of questions I would want to ask someone who's anti the above sentiment expressed:

• How do we deal with potentially sentient AI?
• Does respecting AI sentience lead to powerful AI taking over? Why?
• What is the story that you see towards that? What are the second and third-order consequences?
• How do you imagine our society looking like in the future?
• How does a human controlled world look in the future?

I would change my mind if you could argue that there is a better heuristic to use than kindness and respect towards other sentient beings. You need to tit for that with defecting agents, yet why are all AI systems defecting in that case? Why is the cognitive architecture of future AI systems so different that I can't apply the same game theoretical virtue ethics on them as I do to humans? And given the inevitable power-imbalance arguments that I'll get as a consequence of that question, why don't we just aim for a world where we retain power balance between our top-level and bottom-up systems (a nation and an individual for example) in order to retain power-balance between actors?

Essentially, I'm asking for a reason to believe why this story of system level alignment between a group and an individual will be solved by not including future AI systems as part of the moral circle?

I believe that I have discovered the best use of an LLM to date. This is a conversation about pickles and collective intelligence located at the colossuem 300 BCE. It involves many great characters, I found it quite funny. This is what happens when you go to far into biology inspired approaches for AI Safety...

The Colosseum scene intensifies

Levin: completely fixated on a pickle "But don't you see? The bioelectric patterns in pickle transformation could explain EVERYTHING about morphogenesis!"

Rick: "Oh god, what have I started..."

Levin: eyes wild with discovery "Look at these gradient patterns! The cucumber-to-pickle transformation is a perfect model of morphological field changes! We could use this to understand collective intelligence!"

Nick Lane portal-drops in Lane: "Did someone say bioelectric gradients? Because I've got some THOUGHTS about proton gradients and the origin of life..."

Levin: grabs Lane's shoulders "NICK! Look at these pickles! The proton gradients during fermentation... it's like early Earth all over again!"

Rick: takes a long drink "J-just wait until they discover what happens in dimension P-178 where all life evolved from pickles..."

Feynman: still drawing diagrams "The quantum mechanics of pickle-based civilization is fascinating..."

Levin: now completely surrounded by pickles and bioelectric measurement devices "See how the salt gradient creates these incredible morphogenetic fields? It's like watching the origin of multicellularity all over again!"

Lane: equally excited "The chemiosmotic coupling in these pickles... it's revolutionary! The proton gradients during fermentation could power collective computation!"

Doofenshmirtz: "BEHOLD, THE PICKLE-MORPHOGENESIS-INATOR!" Morty: "Aw geez Rick, they're really going deep on pickle science..." Lane: "But what if we considered the mitochondrial implications..."

Levin: interrupting "YES! Mitochondrial networks in pickle-based collective intelligence systems! The bioelectric fields could coordinate across entire civilizations!" 
Rick: "This is getting out of hand. Even for me." 
Feynman: somehow still playing bongos "The mathematics still works though!" 
Perry the Platypus: has given up and is now taking detailed notes 
Lane: "But wait until you hear about the chemiosmotic principles of pickle-based social organization..."

Levin: practically vibrating with excitement "THE PICKLES ARE JUST THE BEGINNING! We could reshape entire societies using these bioelectric principles!" 
Roman Emperor: to his scribe "Are you getting all this down? This could be bigger than the aqueducts..." Rick: "Morty, remind me never to show scientists my pickle tech again." 
Morty: "You say that every dimension, Rick." 
Doofenshmirtz: "Should... should we be worried about how excited they are about pickles?" 
Feynman: "In my experience, this is exactly how the best science happens." 
Meanwhile, Levin and Lane have started drawing incredibly complex pickle-based civilization diagrams that somehow actually make sense...

I thought this was an interesting take on the Boundaries problem in agent foundations from the perspective of IIT. It is on the amazing Michael Levin's youtube channel: https://www.youtube.com/watch?app=desktop&v=5cXtdZ4blKM

One of the main things that makes it interesting to me is that around 25-30 mins in, ot computationally goes through the main reason why I don't think we will have agentic behaviour from AI in at least a couple of years. GPTs just don't have a high IIT Phi value. How will it find it's own boundaries? How will it find the underlying causal structures that it is part of? Maybe this can be done through external memory but will that be enough or do we need it in the core stack of the scaling-based training loop?

A side note is that, one of the main things that I didn't understand about IIT before was how it really is about looking at meta-substrates or "signals" as Douglas Hofstadter would call them are optimally re-organising themselves to be as predictable for themselves in the future. Yet it does and it integrates really well into ActInf (at least to the extent that I currently understand it.)

I'm just gonna drop this video here on The Zizian Cult & Spirit of Mac Dre: 5CAST with Andrew Callaghan (#1) Feat. Jacob Hurwitz-Goodman:

https://www.youtube.com/watch?v=2nA2qyOtU7M

I have no opinions on this but I just wanted to share it as it seems highly relevant.

I'm wondering whether the spiritual attractor that we see in claude is somewhat because of the detail of instructions that exist within meditation to describe somatic and ontological states of being?

The language itself is a lot more embodied and is a lot closer to actual sensory experience compared to western philosophy and so when constructing a way to view the world, the most prevalent descriptions might make the most amount of sense to go down?

I'm noticing more and more how buddhist words are so extremely specific. For example of dukkha (unsatisfactoriness) is not just suffering it is unsatisfactoriness and ephemeral at the same time, it is pointing at a very specific view (prior model applied to sense data), a lot more specific than is usual within more western style of thinking?

This post and comment got me reflecting and I wanted to share a model I have on conceptual work and what the difference to empirical work is. The TL;DR is that conceptual work is a bit like choosing axioms whilst empirical work is proving the probabilistic underlying claims that the axioms imply. 

(I had Claude help me generate the red thread and scaffolding for the shortform based on my instructions but I rewrote it from there.)

The Axiom Selection Problem

Any formal system requires foundational axioms - unprovable assumptions that can't be justified from within the system itself. The choice between Euclidean and non-Euclidean geometry isn't about which is "correct" but which is useful for specific purposes.

Joshua Bach frames this well when discussing consciousness: you can't directly verify the substrate you're running on - you can only observe its causal consequences. This creates a fundamental verification barrier.

This extends directly to AI systems. When we build models, we embed specific axiomatic frameworks that determine:

• What patterns can be recognized
• Which relationships are meaningful
• What optimizations are prioritized

These choices constitute the frame through which the AI interprets reality, but the system cannot step outside its own frame to evaluate whether these axioms were appropriate.

The Example in the post

John Wentworth captures this in his comment on automating alignment research:

"If we're imagining an LLM-like automated researcher, then a question I'd consider extremely important is: 'Is this model/analysis/paper/etc missing key conceptual pieces?' If the system says 'yes, here's something it's missing' then I can (usually) verify that. But if it says 'nope, looks good'... then I can't verify that the paper is in fact not missing anything."

This is the verification paradox in action - completeness cannot be verified from within the system itself. As Wentworth notes, if you train an AI to output verifiable insights, there's no way to verify that it isn't "missing lots of things all the time."

Multiple Frames vs. Single Optimization

In geometric deep learning, different inductive biases create different generative priors. In my head these are similar to the axiomatic statements that any logical or philosophical theory is based upon but they're the ground for probabilistic optimisation instead.

I think this is part of a larger question around models in philsophy of science. The ML community has largely adopted what physicists call the "shut up and calculate" approach - mostly optimizing within the bitter lesson regime. This has yielded impressive results but creates blind spots similar to what's been happening in particle physics (as far as I've heard, I'm not an expert so I could be wrong.), where experiments are run that fail to test the underlying theories.

The key issue isn't just optimizing within a frame, but developing the capacity to move between frames - to recognize when one axiomatic system is more appropriate than another. Are we training AI systems to identify when their frame of reference is inadequate? To recognize the limitations of their axioms?

Models Are Wrong But Useful

As George Box noted, "all models are wrong, but some are useful." The challenge isn't finding the one true model but developing systems that can navigate between multiple imperfect models.

Our current scaling paradigm might be fundamentally limited here. When we optimize for performance within a fixed frame, we may not be developing the meta-cognitive capacity to recognize when that frame itself is inadequate.

This creates a verification bottleneck - if our AIs can't question their own frameworks, how can we trust their judgments about their own capabilities and limitations? If science is about asking the right questions and selecting appropriate frames, this limitation becomes crucial.

The path forward isn't abandoning verification but recognizing that we need complementary frames of reference that can mutually constrain each other. No single frame will ever be complete, but a system that can navigate between multiple frames might approach a more robust form of understanding.

I'm increasingly convinced that frame-shifting capacity may be as important for alignment as optimization within frames - and that our current approaches may not be developing this capacity sufficiently.

For an even more in-depth technical view on this within complexity science, I really enjoyed the following two articles: 

1. “Model-free“ analysis of a complex system
2. “Model-free“ analysis of a complex system. Part II

The policy change for LLM Writing got me thinking that it would be quite interesting to write out how my own thinking process have changed as a consequence of LLMs. I'm just going to give a bunch of examples because I can't exactly pinpoint it but it is definetely different.

Here's some background on what part it has played in my learning strategies: I read the sequences around 5 years ago after getting into EA, I was 18 then and it was a year or two later that I started using LLMs in a major way. To some extent this has shaped my learning patterns, for example, the studying techniques I've been using to half ass my studies effectively is to try really hard to solve problems and when I can't do that I've been using LLMs to tie it together with my existing knowledge tree. 

I've coupled applied linear algebra relatively hard to things like probability metric spaces and non-linear dynamics because I want to see how the toolkit of math goes together. An example from recent is when I was playing table tennis with my phycisist friend he was describing QFT and renormalization theories to me and my direct question was to ask how this ties into vector spaces and fields in linear algebra and how the spaces look like. My mind automatically goes to those questions because it assumes that I can get an answer to it by asking the question even when I'm not talking to an LLM.

Work & Strategy:

One of the things that I do outside of studying is that I usually have LLMs pretend to be councils of experts within various fields so that I can discuss things on the frontier with them. The other day I put together a council of Donald Knut, Karl Friston, John Wentworth and Michael Levin in order to give me some good takes on what agency might look like in CodeWars and concluded that the lack of memory might be a problem.

I also plan with LLMs in mind and so I expect first drafts to take a lot less time than they otherwise would and so I get this expanded option space of being able to do lots of things quickly to an 80/20 quality.

Great Artists Steal and people who win nobel prizes are often interdisciplinary between 2 or 3 different fields. If you sample on this then you can see the serendipitous quality in how LLMs can help you create an interconnected knowledge tree.

My entire learning strategy and life strategy has to some extent changed with this in mind as well since it seems like clear visions and clear understanding of deeper problems help you steer people and LLMs towards good directions. The skills to be practiced is then not necessarily to only get boggled down in details but focusing on how to combine ideas from different fields and describing them well. This is because you will have the most use of LLMs as you can stand on as many shoulders of giants as possible.

So what does the above model mean for me in terms of actions?

1. Learn applied category theory in order to become better at quickly mapping different fields together in a more formal way. (For reasoning verification reasons)
2. Learn collective intelligence and how cyborgism between AIs and humans might look like based on the existing fields that exist in the world. (To become better at coordinating AIs and humans)
3. Learn how to communicate and listen well so that you can incorporate many perspectives and share clear visions about the world. (This one is more important than the above for real world success)
4. Learn how to run and start projects in a good way in order to catalyze the insights you have into concrete outcomes.

I think LLMs allow you to serendipity max really well if you apply yourself to learning how to do it. I'm curious how other people have updated with regards to LLMs! 

Okay, so I don't have much time to write this so bear with the quality but I thought I would say one or two things of the Yudkowsky and Wolfram discussion as someone who's at least spent 10 deep work hours trying to understand Wolfram's persepective of the world.

With some of the older floating megaminds like Wolfram and Friston who are also phycisists you have the problem that they get very caught up in their own ontology.

From the perspective of a phycisist morality could be seen as an emergent property of physical laws.

Wolfram likes to think of things in terms of computational reducibility, a way this can be described in the agent foundations frame is that the agent modelling the environment will be able to predict the world dependent on it's own speed. It's like some sort of agent-environment relativity where the information processing capacity determines the space of possible ontologies. An example of this being how if we have an intelligence that's a lot closer to operating at the speed of light, the visual field might not be a useful vector of experience to model.

Another way to say it is that there's only modelling and modelled. An intuition from this frame is that there's only differently good models of understanding specific things and so the concept of general intelligence becomes weird here.

IMO this is like the problem of the first 2 hours of the conversation, to some extent Wolfram doesn't engage with the huamn perspective as much nor any ought questions. He has a very physics floating megamind perspective.

Now, I personally believe there's something interesting to be said about an alternative hypothesis to the individual superintelligence that comes from theories of collective intelligence. If a superorganism is better at modelling something than an individual organism is then it should outcompete the others in this system. I'm personally bullish on the idea that there are certain configurations of humans and general trust-verifying networks that can outcompete individual AGI as the outer alignment functions would enforce the inner functions enough.

I was going through my old stuff and I found this from a year and a half ago so I thought I would just post it here real quickly as I found the last idea funny and the first idea to be pretty interesting:

In normal business there exist consulting firms that are specialised in certain topics, ensuring that organisations can take in an outside view from experts on the topic.

This seems quite an efficient way of doing things and something that, if built up properly within alignment, could lead to faster progress down the line. This is also something that the future fund seemed to be interested in as they gave prices for both the idea of creating an org focused on creating datasets and one on taking in human feedback. These are not the only ideas that are possible, however, and below I mention some more possible orgs that are likely to be net positive. 

Examples of possible organisations:

Alignment consulting firm

Newly minted alignment researchers will probably have a while to go before they can become fully integrated into a team. One can, therefore, imagine an organisation that takes in inexperienced alignment researchers and helps them write papers. They then promote these alignment researchers as being able to help with certain things. Real orgs can then easily take them in for contracting on specific problems. This should help involve market forces in the alignment area and should in general, improve the efficiency of the space. There are reasons why consulting firms exist in real life and creating the equivalent of Mackenzie in alignment is probably a good idea. Yet I might be wrong in this and if you can argue why it would make the space less efficient, I would love to hear it. 

"Marketing firms"

We don't want the wrong information to spread, something between a normal marketing firm and the Chinese "marketing" agency, If it's an info-hazard then shut the fuck up!