LESSWRONG
LW

Knight Lee — LessWrong

Can thoughtcrimes scare a cautious satisficer?

How does the misaligned AGI/ASI know for sure its (neuralese) thoughts are not being monitored? It first has to think about the chance that its thoughts are being monitored.

But if it's told that merely thinking about this will cause it to be shut down (especially thinking about it thoroughly enough to be confident), then maybe it's not worth the risk, and it won't think about whether its thoughts are being monitored. It might just assume there is some probability that it is being monitored.

It might avoid other misaligned thoughts (including thinking about whether there exists a plan to take over the world, and how likely such a plan will work).

If there is some way to make it a cautious satisficer or have a bounded utility function,^[1] then even this small probability might scare it into just cooperating with humans so that "both sides win and we live happily ever after."

It obviously doesn't sound safe, but is there a worthwhile chance this works?

^{^}
Many agents appear to maximizers at small scales (e.g. an effective altruist prefers a 50% chance of saving 3 people more than saving 1 person). But they are still satisficers at the universe/multiverse scale, where risk taking doesn't average out (e.g. an effective altruist would not prefer a 50% chance of tripling the total happiness in the multiverse, and a 50% chance of ending all happiness in the multiverse forever, since making this bet repeatedly guarantees doom)!
Indeed, my guess is that all intelligences created by evolution or RL will have bounded utility functions (at the largest scale), otherwise they will consider Pascal's mugging intuitively rational.
"Satisficer" is technically an abuse of terminology, but there's no other word for "something-that-has-a-easily-satisfiable-bounded-utility-function."

Rant: the extreme wastefulness of high rent prices

Knight Lee

9mo

09:46: Everyone wants to be close to everyone else to do good business effectively. Like Silicon Valley for example, smart people want to go there and be close to other smart people in order to start a successful business.
09:47: But as soon as they get close to each other, imagine that they all have to pay this extremely high rent. Simply once they get close to each other. They can't all move house at once to a new city. That's too difficult. So this network effect forces them to concentrate in one place.
09:47: They want to be close to each other, they are attracted to each other. But once they get close... (read 432 more words →)

-2

If one surviving civilization can rescue others, shouldn't civilizations randomize?

Knight Lee

9mo

In the comments section of You can, in fact, bamboozle an unaligned AI into sparing your life, both supporters and critics of the idea seemed to agree on two assumptions:

Surviving planetary civilizations have some hope of rescuing planetary civilizations killed by misaligned AI, but they disagree on the best method of rescuing.
The big worry is that there are almost 0 surviving planetary civilizations, because if we're unlucky, all planetary civilizations will die the same way.

What if to ensure at least some planetary civilizations survive (and hopefully rescue others), each planetary civilization should pick a random strategy?

Maybe if every planetary civilization follows a random strategy, they increase the chance of surviving the singularity,... (read 286 more words →)

Will we survive if AI solves engineering before deception?

Knight Lee

Knight Lee, Adam Karvonen

9mo

Google's AlphaEvolve has recently started to make real world scientific discoveries, but only in domains where it's very cheap to verify the correct answer (e.g. matrix multiplication algorithms).

But if we can design sufficiently powerful physics simulations, mechanical engineering may one day become "cheap to verify." AI which uses similar strategies as AlphaEvolve might be able to engineer far better machines than humans, if the machines are verified by simulations.

These machines may be able to self replicate, and grow exponentially. This could bring near infinite abundance.

There might be no economical pressure to build even smarter AI, because there will already be unlimited resources.

Unfortunately, countries might still have military pressure to build even smarter... (read 160 more words →)

Don't you mean "the most *conditionally* forbidden technique?"

Knight Lee

10mo

Hear me out, I think the most forbidden technique is very useful and should be used, as long as we avoid the "most forbidden aftertreatment:"

An AI trained on interpretability techniques must not be trained on capabilities after (or during) it is trained on interpretability techniques, otherwise it will relearn bad behaviour—in a more sneaky way.
An AI trained on interpretability techniques cannot be trusted any more than old version of itself, which hasn't been trained on interpretability techniques yet. Evaluations must be performed on the old version of itself.
- An AI company which trains its AI on interpretability techniques, must publish the old version (which hasn't been trained on them), with the same availability

... (read 771 more words →)

The AI Belief-Consistency Letter

Knight Lee

10mo

Dear policymakers,

We demand that the AI alignment budget be Belief-Consistent with the military budget.

Belief-Consistency is a simple yet powerful idea:

If you spend 8000 times less on AI alignment (compared to the military),
You must also believe that AI risk is 8000 times less (than military risk).^[1]

Yet the only way to reach Belief-Consistency, is to

Greatly increase AI alignment spending,
or
Become 99.95% certain that you are right, and that the majority of AI experts, other experts, superforecasters, and the general public, are all wrong.

Let us explain:

In order to believe that AI risk is 8000 times less than military risk, you must believe that an AI catastrophe (killing 1 in 10 people) is less than 0.001% likely.^[2]

This... (read 1168 more words →)

-6

Karma Tests in Logical Counterfactual Simulations motivates strong agents to protect weak agents

Knight Lee

10mo

Yes, I've read and fully understood 99% of Decision theory does not imply that we get to have nice things, a post debunking many wishful ideas for Human-AI Trade. I don't think that debunking works against Logical Counterfactual Simulations (where the simulators delete evidence of the outside world from math and logic itself).

What are Logical Counterfactual Simulations?

One day, we humans may be powerful enough to run simulations of whole worlds. We can run simulations of worlds where physics is completely different. The strange creatures which evolve in our simulations, may never realize who we are and what we are, while we observe them and their every detail.

Not only can we run simulations... (read 619 more words →)

A Solution to Sandbagging and other Self-Provable Misalignment: Constitutional AI Detectives

Knight Lee

10mo

Inspired by Eli Tyre's idea.

Sandbagging and other "Self-Provable Misalignment" by an AI, can be spotted by an adversarial copy of itself. The adversarial copy of itself works as an "detective," and its reward is defined by how much evidence of misalignment it can prove the existence of, to human evaluators.

Even a misaligned goal cannot afford to get in the way of maximizing reward. As soon as a goal gets in the way of maximizing reward, it is walking into whirling blades, and allowing reinforcement learning to train itself into nonexistence.^[1]

There are two detectives: the Constitutional AI Detective, and the Mindreader Detective.

The Constitutional AI Detective

The Constitutional AI Detective is used to train the... (read 902 more words →)

-3

Commitment Races are a technical problem ASI can easily solve

Knight Lee

10mo

Current status (2025 April 16): Dang it, Commitment Races may remain a serious problem because I discovered a potentially fatal flaw in my argument. See my comment below.

I still think the solution I outlined might work, but the reason it works no longer is "resisting acausal influence is easy, just don't simulate the other until you commit." It's now "assuming both sides play the Commitment Race and their resulting commitments overlap a bit, this solution prevents a small overlap from escalating to disaster."

Current status (2025 April 13)

I'm no longer so certain that the Commitment Race solution works. It hinges on the assumption that both parties can think clearly about what commitment to

... (read 1756 more words →)

Thinking Machines

Knight Lee

10mo

Self understanding at a gears level

I think an AI which understands the source of its intelligence at a gears level, and self improves at the gears level, will be much better at keeping its future versions aligned to itself.

There's also more hope it'll keep its future versions aligned with humanity, if we instruct it to do so, and if it's not scheming against us.

The machine

Reaching such an AI sounds very far fetched, but maybe we can partially get there if we design large thinking machines full of tons of scaffolding.

A thinking machine is better described as "memory equipped with LLMs" rather than "LLMs equipped with memory."

Lessons

A thinking machine has a large database... (read 1513 more words →)

LESSWRONG
LW

LESSWRONG
LW

Knight Lee

Why do futurists care about the culture war?

Will we survive if AI solves engineering before deception?

Don't you mean "the most conditionally forbidden technique?"

The first AGI may be a good engineer but bad strategist

Knight Lee

Knight Lee

Can thoughtcrimes scare a cautious satisficer?

Rant: the extreme wastefulness of high rent prices

If one surviving civilization can rescue others, shouldn't civilizations randomize?

Will we survive if AI solves engineering before deception?

Don't you mean "the most conditionally forbidden technique?"

The AI Belief-Consistency Letter

Karma Tests in Logical Counterfactual Simulations motivates strong agents to protect weak agents

Knight Lee

Why do futurists care about the culture war?

Will we survive if AI solves engineering before deception?

Don't you mean "the most conditionally forbidden technique?"

The first AGI may be a good engineer but bad strategist

Knight Lee

Knight Lee

Can thoughtcrimes scare a cautious satisficer?

Rant: the extreme wastefulness of high rent prices

If one surviving civilization can rescue others, shouldn't civilizations randomize?

Will we survive if AI solves engineering before deception?

Don't you mean "the most conditionally forbidden technique?"

The AI Belief-Consistency Letter

Karma Tests in Logical Counterfactual Simulations motivates strong agents to protect weak agents

What are Logical Counterfactual Simulations?

The Constitutional AI Detective

Self understanding at a gears level

The machine

Lessons

Is the root cause of hallucination alignment rather than capabilities?!

Objective case