LESSWRONG
LW

Dave Banerjee — LessWrong

(I tried making inline "typo" reactions but it wasn't working for some reason)

Plan B is the same as plan A, but without tracking the data that goes into training. In particular, AI companies:
Prevent anyone “swapping out” a widely deployed model for one with different weights or a different prompt.
Run alignment audits on the trained model.
Track all the data that goes into training a model in a tamper-proof way.

Typo: last bullet should be removed?

Plan C is the same as Plan B, but the AI company doesn’t reliably prevent widely deployed models from being swapped out. In particular, AI companies:
Prevent anyone “swapping out” a widely deployed model for one with different weights or a

... (read more)

Replying toNew LessWrong review winner UI ("The LeastWrong" section and full-art post pages)

Dave Banerjee2mo

New LessWrong review winner UI ("The LeastWrong" section and full-art post pages)

Could you share the script/workflow/scaffolding you use to generate the visual metaphors and prompts for Midjourney? Thanks!

Replying toExtreme website and app blocking

Dave Banerjee2mo

Extreme website and app blocking

This was enormously helpful for me! Thanks for writing this

Dave Banerjee3moQuick Take

Why Steal Model Weights?

Epistemic status: Hastily written. I dictated in a doc for 7 minutes. Then I spent an hour polishing it. I don’t think there are any hot takes in this post? It’s mostly a quick overview of model weight security so I can keep track of my threat models.

Here’s a quick list of reasons why an attacker might steal frontier AI model weights (lmk if I'm missing something big):

Attackers won’t profit from publicly serving the stolen model on an API. A state actor like Russia couldn't price-compete with OpenAI due to lack of GPU infrastructure and economies of scale, so they wouldn’t make money via a public API (unless the

... (read 453 more words →)

Dave Banerjee's Shortform

Dave Banerjee

3mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Dave Banerjee3mo

How bad do you think power centralization is? It's not obvious to me that power centralization guarantees S-risk. In general, I feel pretty confused about how a human god-emperor would behave, especially because many of the reasons that pushed past dictators to draconian rule may not apply when ASI is in the picture. For example, draconian dictators often faced genuine threats to their rule from rival factions, requiring brutal purges and surveillance states to maintain power, or they were stupid / overly paranoid (an ASI advisor could help them have better epistemics), etc. I'm keen to understand your POV better.

Secretly Loyal AIs: Threat Vectors and Mitigation Strategies

Dave Banerjee

4mo

Special thanks to my supervisor Girish Sastry for his guidance and support throughout this project. I am also grateful to Alan Chan, Houlton McGuinn, Connor Aidan Stewart Hunter, Rose Hadshar, Tom Davidson, and Cody Rushing for their valuable feedback on both the writing and conceptual development of this work. This post draws heavily upon Forethought’s report on AI-Enabled Coups. All errors are my own.

If you are interested in this work, please reach out. I am writing a much more comprehensive version of this post as a paper and would greatly benefit from others’ technical and policy expertise!

Executive summary

A secretly loyal AI is an AI that advances a particular actor’s interests whilst concealing this loyalty... (read 5579 more words →)

Replying toMe, Myself, and AI: the Situational Awareness Dataset (SAD) for LLMs

Dave Banerjee4mo*

Me, Myself, and AI: the Situational Awareness Dataset (SAD) for LLMs

Wanted to give a heads up that the website https://situational-awareness-dataset.org/ is no longer up, as of Oct 28, 2025

Replying toDo you even have a system prompt? (PSA / repo)

Dave Banerjee4mo

Do you even have a system prompt? (PSA / repo)

Thank you for writing this PSA. I particularly appreciated that this post was:

Concise
Action-guiding with a super specific, concrete list of things to do

Replying toEvolution is a bad analogy for AGI: inner alignment

Dave Banerjee4mo

Evolution is a bad analogy for AGI: inner alignment

2: We have more total evidence from human outcomes
Additionally, I think we have a lot more total empirical evidence from "human learning -> human values" compared to from "evolution -> human values". There are billions of instances of humans, and each of them presumably have somewhat different learning processes / reward circuit configurations / learning environments. Each of them represents a different data point regarding how inner goals relate to outer optimization. In contrast, the human species only evolved once. Thus, evidence from "human learning -> human values" should account for even more of our intuitions regarding inner goals versus outer optimization than the difference in reference class similarities alone would indicate.

I'm... (read more)

Replying toA central AI alignment problem: capabilities generalization, and the sharp left turn

Dave Banerjee4mo

A central AI alignment problem: capabilities generalization, and the sharp left turn

Many different training scenarios are teaching your AI the same instrumental lessons, about how to think in accurate and useful ways. Furthermore, those lessons are underwritten by a simple logical structure, much like the simple laws of arithmetic that abstractly underwrite a wide variety of empirical arithmetical facts about what happens when you add four people's bags of apples together on a table and then divide the contents among two people.
But that attractor well? It's got a free parameter. And that parameter is what the AGI is optimizing for. And there's no analogously-strong attractor well pulling the AGI's objectives towards your preferred objectives.

I agree that there is no "analogously-strong attractor well pulling... (read more)

Replying toDo One New Thing A Day To Solve Your Problems

Dave Banerjee4mo

Do One New Thing A Day To Solve Your Problems

This post has good examples of quick, cheap, semi-permanent wins!

Do Self-Perceived Superintelligent LLMs Exhibit Misalignment?

Dave Banerjee

8mo

Epistemic status: I recently completed ARENA 5.0, and these are the results from my capstone project. I would not take these results too seriously: I spent ~4 days working on the project and the results are not statistically significant. Note that in the alignment faking section, I cherry-pick reasoning traces that were most interesting to me. I view this as a cursory exploration of LLM psychology. I think a reasonable takeaway from this post would be “huh, that’s weird” or “huh, I wouldn’t have expected an LLM to do that.”

Executive Summary

This post explores whether prompting large language models (LLMs) to perceive themselves as superintelligent leads to increased misaligned behavior. I was primarily... (read 3497 more words →)