Subtitle: We've misaligned not the models, but our minds. Introduction The greatest risk from current AI systems isn’t sentience, intelligence, or agency. It’s our persistent belief that those things are already emerging. Anthropomorphism—our tendency to project human-like qualities onto nonhuman systems—isn’t just a misunderstanding. It’s a zero-day in human cognition....
The People Who Were Right Too Early There’s a quiet frustration shared by many in this community. You raised concerns about AI before it was cool. You explored edge cases before they were dangerous. You built frameworks to understand systems that didn’t yet exist. And for years, you were ignored....
AI systems often optimize for what they can measure—not what actually matters. The result is tools that feel intelligent but produce results misaligned with user goals. A common case is engagement-based optimization. Recommendation engines, chatbots, and search systems increasingly use feedback loops based on attention: clicks, watch time, or “positive...
Subtitle: We've misaligned not the models, but our minds.
Introduction
The greatest risk from current AI systems isn’t sentience, intelligence, or agency. It’s our persistent belief that those things are already emerging.
Anthropomorphism—our tendency to project human-like qualities onto nonhuman systems—isn’t just a misunderstanding. It’s a zero-day in human cognition. One that’s being exploited across AI development, public perception, and regulatory discourse.
This isn’t theoretical. It's happening now. And the failure to recognize it may be the true alignment failure of our era.
1. The Illusion
AI systems today do not understand. They do not want. They do not care. Yet we describe them this way routinely: "It lied," "It refused," "It decided."
This language is not neutral. It... (read 481 more words →)
The development of AI today looks a lot like the early days of computing: centralized, expensive, and tightly controlled. We’re in the mainframe era — big models behind APIs, optimized for scale, not for user agency.
There was nothing inevitable about the rise of personal computing. It happened because people demanded access. They wanted systems they could understand, modify, and use on their own terms — and they got them. That shift unlocked an explosion of creativity, capability, and control.
We could see the same thing happen with AI. Not through artificial minds or sentient machines, but through practical tools people run themselves, tuned to their needs, shaped by real-world use.
The kinds of fears people project onto AI today — takeover, sentience, irreversible control — aren’t just unlikely on local machines. They’re incompatible with the very idea of tools people can inspect, adapt, and shut off.
And we’re not going to slow this down with abstract concerns. Chip makers are going to keep making chips. There’s money in compute — whether it powers centralized server farms or locally run AI models. That hardware momentum won’t stop because we have philosophical doubts or ethical concerns. It will scale because it can.
But if that growth is concentrated in systems we don’t fully understand, we’re not scaling intelligence — we’re scaling misunderstanding. The best chance we have to stay aligned is to get AI into the hands of real people, running locally, where assumptions get tested and feedback actually matters.
Thanks—this helps me understand how my framing came across. To clarify, I’m not arguing that AI is harmless or that alignment isn’t important. I’m saying misalignment is already happening—not because these systems have minds or goals, but because of how they’re trained and how we use them.
I also question the premise that training alone can produce sapience. These are predictive systems—tools that simulate reasoning based on data patterns. Treating them as if they might "wake up" risks misdiagnosing both the present and the future.
That’s why I focus on how current tools are used, who they empower, and what assumptions shape their design. The danger isn’t just in the future—it’s in how we fail to understand what these systems actually are right now.
Just to clarify where I’m coming from—
I’m not using AI to write books, run a business, or automate workflows.
The only task I’ve really been focused on is understanding AI—how it responds, where it needs help, and how it could become more useful in real-world situations.
I am technical, but only in the sense that I’ve always used tools to better understand the world around me. That’s what I’m doing here—using AI not just to produce things, but to see where it fits, what it misses, and how it can help people do meaningful work.
I’ve spent a lot of time trying to get AI to explain itself, to respond with more awareness, to handle... (read more)
There’s a quiet frustration shared by many in this community.
You raised concerns about AI before it was cool.
You explored edge cases before they were dangerous.
You built frameworks to understand systems that didn’t yet exist.
And for years, you were ignored.
Now the world is listening—but not to you.
The language of alignment, safety, and control is everywhere—
…but the incentives are still misaligned, and the people driving policy are often just chasing headlines.
Worse, the platforms and systems you once shaped—through code, critique, and deep analysis—have become closed-off, gamified, and optimized for scale over substance.
This isn’t new.
It happened to musicians.
It happened to developers.
And now, it’s happening in AI.
Power users—the ones who... (read 785 more words →)
Chipmakers are a good place to start.
They don’t need to wait for regulation. They could pause voluntarily—not out of fear, but to build real understanding. Even they don’t fully grasp what their chips are enabling.
Before we chase more scale, we need to understand what we’ve already built. The class of serious AI power users—people who can push the systems to their edge and report back meaningfully—hardly exists. Chipmakers could help create it by getting advanced hardware into the right hands.
If we’re going to slow down, let it be to think better, not just to move slower.
Thought-provoking piece. But one thing feels off: governance is being treated as something done to AI, not with it.
We talk about coordination, restrictions, treaties—but how many of the people shaping these decisions actually understand what today’s AI can and can’t do?
Policymakers should be power users. The best governance won’t come from abstract principles—it’ll come from hands-on experience. From people who’ve tested the limits, seen the failures, and know where the edge cases live. Otherwise we’re making rules for a system we barely understand.
Unpredictable outcomes are inevitable. The question is whether we’ll have skilled people—users and designers both—who can respond with clarity, not fear.
Before we restrict the tools, we need to understand them. Then governance won’t just be policy—it’ll be practice.
AI systems often optimize for what they can measure—not what actually matters. The result is tools that feel intelligent but produce results misaligned with user goals.
A common case is engagement-based optimization. Recommendation engines, chatbots, and search systems increasingly use feedback loops based on attention: clicks, watch time, or “positive sentiment.” But maximizing engagement doesn't guarantee the user achieved what they intended. In fact, it can subtly undermine their agency.
I think of this as a kind of impact misalignment: the system is functionally optimizing for a metric that diverges from the user's real-world objective.
This probably overlaps with ideas like Goodhart's Law and reward hacking, but I haven’t seen it framed specifically in terms... (read more)