Jesper L. — LessWrong

LESSWRONG
LW

Replying toHow AI Is Learning to Think in Secret

maybe our last hope for understanding what these things are doing before they turn into completely opaque superintelligence, please God let it keep working.”
*
We got unbelievably, preposterously, cosmically lucky with Chain-of-Thought.

Actually there is an important point to make here. We didn't get so lucky, of course.

Rather CoT was the logical next step, because, as you just explained, it is more powerful to build chains of reasoning than optimize every single output ad infinitum.

We will get far, but never get to SI, with only an LLM, exactly because they lack integrated learnings and continuous states.

An LLM designing the start of a new SI-seed architecture paradigm is one (speculative) thing, but neither whole-models nor... (read more)

Replying toGPT-5.2 Is Frontier Only For The Frontier

Jesper L.25d

GPT-5.2 Is Frontier Only For The Frontier

I am still to finish my own small tests and lm-evals on GPT 5, but I am already quite concerned about deception, and even functional sandbagging. How much of such behaviour have the models absorbed from their training by now?

I have poor insight into the Chinese models, but so far the GPT 5 series is the scariest one I have interacted with.

Replying toEliezer's Unteachable Methods of Sanity

Jesper L.2mo

Eliezer's Unteachable Methods of Sanity

I think you are severely underestimating how relatable and common your thoughts on this topic are (also to many journalists). In short, you underestimate people's capacity to get this (probably because they are out-of-distribution for your way of structured reasoning in general, to borrow LW 2.0 lingo).

If I would make a guess, I think that (self-aware) people outside of LW and similar circles may be even more likely to relate to several of these points than people inside of LW. For example, "a sentence about snow is words, is made of words, but it is about snow" and "if I were to dwell on how it impacted me emotionally that the world... (read more)

Jesper L.3mo

Thinking about this much harder, I would rewrite the post to clearly phrase it as damage prevention you can do fast and cheap from your home.

I notice I forgot to mention a first aid kit. First thing to correct.

Secondly, when thinking hard on what of your feedback might actually be useful, I realized that statistically speaking you may do more good by buying slip mats for wet floors, and potentially investing in premium ladders. Since fall accidents dominate injury statistics. Not something I thought about intuitively but makes sense.

https://injuryfacts.nsc.org/home-and-community/safety-topics/fire-related-fatalities-and-injuries/

https://rehis.com/news/preventable-accidents-in-the-uk-are-rising-and-deaths-reach-an-all-time-high-warns-new-report/

For general awareness, accidents are 3d leading cause of death in US, fire is top 3 common accidents in the home, and quick google confirms cast majority of fire deaths happen in the home. Which I knew from safety trainings too.

Jesper L.3mo

Oliver. You just suggested "just use a smartphone" as a flashlight in an emergency as an argument not to get a flashlight. And you say I don't think things through? You make claims that fire alarms are useless. Data? If you hadn't revived LW I would think you are a troll. And your odd reasoning using GPT seems mostly off topic.

I don't think there is any negative utility here, perhaps low on investing in a fire blanket.

Also, I have lived through scenarios where I needed a flashlight, and I have used fire blankets. Like, I have used these in my own life. I have been in the wild too, and the military ( I am from Finland.) Pretty standard safety advice here, tbh. Not sure what your problem is. If you think post is useless, and we should just trust the system, fine, but don't make such low quality critique?

-11

Jesper L.3mo

While skepticism on utility is wise, I strongly disagree with most of your comment. Fire accidents at home is by far the leading majority of fire related deaths in many countries, and fire accidents is a major cause of accidental civil deaths.

If we look at US as a whole (not where I live but you do), fire accidents are few versus drug overdose and gun violence. If I post: Don't do drugs, is this useful advice? Tell me the low-hanging fruit there. Fires you CAN protect yourself against but many don't.

Falls and poisoning are other common accidents worldwide. But, "don't fall" is not very helpful, is it? And I don't know how... (read more)

-10

Jesper L.3moQuick Take

"Common sense" advice that is highly rational and easily overlooked by A LOT of people, including smart people

This is your regular reminder.

1. Buy fire alarms, a fire blanket, a strong light torch for blackouts, and a first aid kit (this is the bare minimum low-cost emergency kit that can save you a lot of hassle and potentially your life).

1.2 Do NOT have heavy power-draining appliances connected via extension cord. Fridges and washing machines should never sit in an extension cord unless you know what you are doing and that it can handle the strain.

A leading cause of accidental death in many countries, especially if we discard traffic accidents, relate to fires. If... (read more)

-1

Jesper L.3mo

What does "pictures in order" mean? Also, damn.

Replying toMaybe Insensitive Functions are a Natural Ontology Generator?

Jesper L.3mo

Maybe Insensitive Functions are a Natural Ontology Generator?

If so, I would be genuinely curious to hear your ideas here. This might be an actually powerful concept if it holds up and you can formalize it properly. I assume you are an engineer, not a scientist? I think this idea deserves some deep thinking.

Jesper L.3mo*Quick Take

Strategic effort can be viewed through an epistemic lens (seeking truth) or a coordination lens (amassing consensus and directing resources). This is also true for AI safety, and x-risk.

However, this binary viewpoint misses the point of why we discuss strategy in the first place. The goal is not to simply be right, rationally or politically. The goal is to win.

The art of strategy is the art of winning.

Of course, the objection that comes to mind: What does "winning" even mean in ASI safety? Isn't that what we are trying to figure out here?

We must separate what from how.

When deciding the strategic win condition for x-risk, there is no need to overcomplicate it.... (read more)

-8

The AI bubble is expanding very fast. E.g., Thinking Machines by ex OpenAI CTO Murati and his four dozens of employees, is now near funding at 1 billion USD per employee.

At this rate, it's unlikely the bubble will last until reality (possibly) catches up. This begs important questions:

Would an AI bubble burst would be a good opportunity to double down on safety and reset AI budgeting?
Will the US economy survive a burst?
How does this affect China's strategy?
Should safety people secure funding now?

Zvi recently shared news about the impressive open model Kimi 2 thinking. It is currently in the top rankings across the leaderboards (you can find several leaderboards via my monitoring page).

I have briefly tried it and probed it with my own hard tests on meta cognition, ontological coherence, and moral epistemics. On these edge cases, where frontier LLMS fail to humans, it is performing equal to Claude 4.5, Gemini 2.5 and GPT 5 and 5 mini. It is also self-reporting limitations more accurately than Claude, and perhaps also better than GPT 4 and 5 mini (not enough testing to say).

What is the (LW) consensus on jump from qualia to self-awareness in AI?

Jesper L.

3mo

What is the consensus here on jumping from qualia (inner experience) to full self-awareness in LLM based AI? Meaning: If an AI running on something like LLM based architecture would gain qualia; inner experience of any kind, is the gap to self-awareness small?

Is it perhaps 15 % for qualia, 10 % for full self-awareness?

The alternative would be a bigger gap between qualia and self-awareness. Perhaps as big as, or bigger than the gap from non-sentience to qualia.

This question is only about how big the sentience jump would be, relatively speaking. I do not explicitly care about agency here. (The consensus there is ofc that agency is more likely than qualia. Those Ps... (read more)

OpenAIs disclosure of Scheming in LLMs is of course a big red flag, since it shows the model can enable an agent to strategize deceptivey, in theory, should it develop goals.

But it is important to realize that this is also a major red flag because any agent running on the model can now hide having goals in the first place.

OpenAI addresses that a goal can be subversed and kept hidden by scheming agents. But this presumes a task is already set and subseq. subversed.

Emrgent intent/persistent goal seeking is the biggest red flag there is. It should be impossible inside LLMs themselves, and also impossible in any agentic process-instance run continously, without significant scaffolding. The ability to hide intent before we could detect it is therefore catastrophic. This point is somewhat non-obvious.

It's not just that IF goals emerge the agent may mislead us, but that goals may emerge without us noticing at all.

Two core beliefs about AI to question

Mainstream belief: Rational AI agents (situationally aware, optimizes decisions, etc.) are superior problem solvers, especially if they can logically motivate their reasoning.

Alternative possibility: Intuition, abstraction and polymathic guessing will outperform rational agents in achieving competing problem-solving outcomes. Holistic reasoning at scale will force-solve problems intractable by much more formal agents, or at least outcompete in speed/complexity.

2)

Mainstream belief: Non-sentient machines will eventually recursively self-improve to steer towards their goals

Alternative possibility (much lower but non-trivial p): Self-improvement coupled with goal persistence requires sophisticated, conscious self-awareness. No persistent self-improvement drive is likely to occur in non-sentient machines.

/

Mainstream beliefs are well-informed based on LLMs, but I think many AI people take them for granted at this point. Making belief into a premise is dangerous. This is uncharted territory and most humans-in-the-past appear silly and ignorant most of the time, after all, to most humans-in-the-future.

Meta-cognition and epistemic intelligence is undertested in current LLMs. Deployed LLMs lack casual experience, embodiment and continuous awareness. People forget this. Too much focus is still put on analytical capability benchmarks in math and coding versus casual analysis and practical judgement metrics. This worries me. Predictive AIs are not scaling to be used as calculators.

Some undertested but important areas are starting to gain more testing-traction: Psychology, social intelligence, long-term planning, etc. For LLMs, understanding these fields is all about textual correlation. LLMs cannot verify their knowledge in these fields on their own.

We always needed to study LLMs metacognition as they scale and change. We are not doing enough of it.

Deployed AI models are not perfectly isolated systems, anymore than humans are. We interact with external information and the environment we are in. The initial architecture determines how well we can interact with the world and develop inside of it. This shapes our initial development.

(Every rationalist learns this at some point, but it is not always integrated.)

For example, human retinas are extremely good at absorbing visual data, and so much of our cognition grows centered around this data. If we build a world view based on this data, this ontology is path-dependent.

Bear with me, as I elaborate.

The dependence of the retina comes from its structure, which is an example of what I... (read more)

Jesper L.'s Shortform

Jesper L.

4mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

I think that in current political climate, the key to mitigate race dynamics hinges on cross-lab collaboration (safety collaboration being the goal).

To do this, you need to create as many interaction points as possible, direct and indirect, between the labs. Then steer towards safety collaboration being beneficial. Use general incentives as levers, tie them to safety purposes.

Think of various intracellular pathways that initially evolve independently, with various outcomes that compete or support each other. Because their components may interact, and because their outcomes affect each other, they end up regulating each other. Over time the signal network optimizes to a shared outcome.

Note: You can only steer if you are also involved in... (read more)

Reminder: Morality is unsolved

Jesper L.

4mo

Here is a game you can play with yourself, or others:

a) You have to decide on a moral framework that can be explained in detail, to anyone.

b) It will be implemented worldwide tomorrow.

c) Tomorrow, every single human on Earth, including you and everyone you know, will also have their lives randomly swapped with someone else.

This means that you are operating under the veil of ignorance. You should make sure that the morality you decide on is beneficial whoever you are, once it takes effect.

Multiplayer: The one to first convince all other players, wins.

Single player: If you play alone, you just need to convince yourself.

Good luck!

Morality is unsolved

Let me put this another way:... (read 630 more words →)

Dead-switches as AI safety tools

Jesper L.

4mo

Time estimate: 1 minute read
+ Additional arguments for ca. 6 minutes

Some obvious technical ideas for improving AI safety are largely overlooked, or at least not discussed enough. I claim that the concept of dead-switches for AI control is one of them.

A dead man's switch, a dead-switch, is originally meant as a technical device that triggers if human operators become incapacitated. It is a fail-safe. The famous example is the Soviet "dead hand" nuclear deterrent.

In AI terms, it is a safety mechanism that works like a kill switch, but triggers automatically as default, in case of compromised system integrity. You don't have to detect and react to a threat.

When you view the concept... (read 1314 more words →)

The Underexplored Prospects of Benevolent Superintelligences - PART 1: THE WISE, THE GOOD, THE POWERFUL

Jesper L.

4mo

THE UNEDEREXPLORED PROSPECTS OF BENEVOLENT SUPERINTELLIGENCES

WELCOME TO THE SERIES - START HERE

This trilogy explores what happens AFTER we solve alignment - a scenario many researchers treat as 'mission accomplished.' I argue that we are dramatically underestimating our ignorance about what benevolent superintelligence means.

The scenarios presented are not predictions. They're illustrations of how little we actually know about what we're building toward. Each scenario is internally consistent with “successful alignment” - yet significantly different from the others. This should concern us.

I wrote this series to a broader audience. I have tried to make this series accessible not just for AI safety people, but for anyone with a relationship to AI.

It's one of the... (read 7456 more words →)

Proactive AI Control: A Case for Battery-Dependent Systems

Jesper L.

6mo

Epistemic effort: ~26 hours, multiple drafts

Epistemic status: Relatively confident in the core claims. Moderately confident in the framing and arguments. Feedback especially welcome on whether I’m reinventing the wheel.

TL;DR: Proposes physically battery-dependent AI architectures as proactive control—hard runtime caps, predictable shutdowns, and built-in “dead-man’s switch” behavior. Discusses feasibility for API endpoints and token processing units.

What if we just... put frontier AI systems on batteries? This elementary idea might be more powerful than it appears.

Introduction – A Proactive Control Philosophy

Coming into the AI safety field anew, after only briefly dipping my toes in it over a decade ago, (I received my introduction through LessWrong blog posts and Max Tegmark’s Life 3.0,) one thing... (read 3677 more words →)