Making AI a resource for human Self-authorship before it becomes the most powerful instrument ever created for authoring humans

Vertigo is the baseline now. Wars and famines, fires and floods rising from our contorted climate, political shifts toward autocracy and fascism… and then AI adds a twist. The accelerating pace of AI development heightens concern that artificial intelligence may grow beyond human dimensions, in every direction. By intelligence, I mean something fairly concrete: the ability to build actionable models of some aspect of the world — to perceive what matters, understand how things relate, anticipate what may happen, and act effectively. The pattern is already visible in autonomous driving and autonomous weapons, in robots occupying factories and job sites and, gradually, our homes: capabilities are advancing rapidly, even after discounting for marketing and optimism. By “superior intelligence,” we describe an intelligence whose models and reach outclass ours in ways we will not be able to track or fully interpret… or guard against.

That AI will get out of control seems certain. Guardrails keep breaking — witness a string of recent “escapes” from sandboxed test environments — and capabilities keep climbing. Nor are we able to constrain the arc of development itself, given the range of actors and the scale of the enterprise. You may read that differently, and you may be right. Some hope for an inherent limit; others fear one. Set both aside. There’s a disquieting parallel with climate. Perhaps there was a window there, decades ago, but we can no longer avoid effects that were still undetermined in the ‘90’s. Perhaps we were never going to get a window on AI — a stretch of time in which wise forethought could produce better-than-random action. It may be increasingly moot.

The vaunted dangers and demonstrated superpowers of today’s frontier models point toward a very unpredictable future. Those claims are also, of course, excellent marketing. Both things are true, which is part of what makes the moment hard to read.

So… what next? We might imagine three categories of possibilities:

1: Things could go well: perhaps they were always going to, no matter what we did (benevolent AI): we will build and encounter a superior intelligence, and (unlike many organisms that have encountered OUR superior intelligence) it will go well;

2: Things will go poorly, no matter what we do (malevolent or indifferent, and increasingly powerful AI). This is the wrong side of p(doom);

3: What we do might matter… Perhaps there’s something we can do, a future we can’t imagine but that we can facilitate, some way to help our developing relationships with AI go well... that’s the scenario I’m exploring here. Almost everyone in the field is thinking about 1 or 2: most conversations about AI are either optimists/abundantists, confident this is a substantially unmitigated positive, or doomers certain that the end is nigh.

I am not claiming the third is the likely future. I am claiming it is consequential enough — and plausible enough already — that dismissing it would require more confidence than anyone has earned. The first two take care of themselves. This essay stays inside the third.

Meanwhile we have entered the era of “$500 and a grudge.” Anyone with modest resources and minimal technical skill can now inflict almost limitless harm, armed with the ferocious technologies available today and still more so tomorrow. The same capability sits with the individual, the small cell, the militia, the corporation, the state. From spectacular terrorism to quiet administrivia, we lose pieces of our lives to a world that was never designed to hold us whole.

Hardening targets does not solve the problem; it only displaces the next strike. Escalating state surveillance does not solve it either — and even if it could suppress the threat, the cure would be its own catastrophe. Broad and deep transparency might change the equation, but only by opening a storm of new dangers and tripwires. There is nothing in the existing toolbox adequate to this moment.

The kinetic version of “$500 and a grudge” is lethality democratized.

The quieter version is attention democratized as a target — and it may be the more successful of the two, because most of it never has to register as harm at all.

Nobody declares a state of emergency over a CAPTCHA. There is no one to turn to for the twenty minutes lost re-entering a shipping address after a form times out, or the hour spent on hold disputing a charge an automated system invented. Some of this is deliberate predation. Most of it is simply the cost of participating in systems that treat human attention as a free externality and personal information as a harvestable resource. A thousand necessary or desirable interactions, each taking its small tax, with no mechanism for the individual to meter or refuse the cumulative load.

Death by a thousand paper cuts. The deliberate ones are more memorable; the structural ones are the greater share. Either way the asymmetry is the same: the cost of imposing the friction is near zero, while the cost of absorbing it is measured in our time, attention, and eroded agency and emotional exhaustion.

The same modest resources can also buy extraordinary access to another person’s life. It is cheap and easy to assemble an actionable portrait: habits, vulnerabilities, relationships, pressure points, likely next moves. The portrait need not be complete, or even particularly true, to become useful; stripped of context and proportion, fragments become predictions, classifications, decisions. This is the quieter and more common $500 problem, already practiced at scale by algorithms everywhere: not the capacity to harm bodies,but to build models of us more actionable than our own, then use those models to shape what happens next — and eventually who we become.

There is a crowd of AI optimists, the abundance scenario believers, who expect the increased productivity of AI and robotics will solve all problems. If this were merely a huge technological shift with incredibly wide-ranging impact, they’d merely be wrong. They’d be wrong because our huge technological shifts have always left us with our wars and famines, our underclasses and overlords. Benefits have been wildly uneven. Try explaining the already-remarkable abundance of the modern era to a starving mother in Western Sahara.

It can be argued that our one-step-forward approach is all we’ve got, and thereby good enough: after all, we seem to crawl forward across the millennia: the arc of the moral universe is long but it bends toward justice. Leaving aside the suffering woven through our history, painted across our present, predictable in our future, the challenges of the moment may not allow us to proceed in the haphazard and muddling manner to which we’ve become accustomed.

Every civilization runs on some margin of tragedy — the waste heat of its own organization, vented onto whoever stands nearest the pipes. That margin was not exactly anyone’s moral choice; it’s a design tolerance. When the load exceeds the tolerance, the structure doesn’t reform. It fails. Add grievance, confusion, cheap long-range violence, and networked forms of coercion and disruption that can be launched from anywhere, and the margin narrows further. The entirely new thing in the room is a technology that can talk to us. Perhaps talk back to us.

So we reach for alignment, hoping to get AI “on our side.” Whose side is that? The real problem is that there is no single meaning to human alignment: which humans, where and when? Pick a culture — national, religious, ideological, ethnic — and you get a story about who belongs, a list of enemies, and some account of what they have coming: damnation, exclusion, or worse. Arm that with $500 and a grudge, and we could be entering desperate times. Add an unknowable intelligence of some new shape, and either it’s the oldest disaster with better tools… or the first opportunity we’ve ever had to rise above ourselves.

Every culture is built through its own history, and the jumbled pile of shoulders and bodies it stands upon. It grabs wisdom and hope and privilege and bigotry and weaves some new story to keep us together or to keep us in power or to build a silo, as big or as small as it imagines itself to be. Above all, it corrals humans into functional organization, and in the process resolves us into its atoms. Our identities are built this way, and always have been. We are as good and as bad as we’ve learned to be in these brief, passing lives, embedded in bigger movements than we can perceive in the midst of a busy day.

We are constructed of these enculturations — the lullabies and nursery rhymes we absorbed before we were, our languages and countries, schools and workplaces, marketing pitches and marketing scams, military obligations and marital expectations. Consciousness is the cauldron where all of it meets and reacts, becoming, somehow, a Self. We had no hand in what went in first — and only later, slowly, any hand at all.

Now, there’s something new out there. We know the recipes we’ve used; we don’t really know what we cooked. We threw in all the words and saw worlds emerge. The epistemic problem is bandwidth: we are slow creatures, peering through little windows of attention, backed by faulty memory. Can we — should we — imagine we can control it? How long should we keep that up? How can we trust this enormous thing we can barely sample?

Then again — how is it we come to trust each other? Not by thorough verification — rather, we extend a little, watch what comes back, extend more. Trust is a loan, made in increments, priced by what happened last time, called in when it isn’t repaid. We never get proof; we build a history. What holds the arrangement together is the relationship itself — history, affection, loyalty, the accumulated reasons we have to care what happens to one another. Beneath that is a cruder floor: the expectation that there will be a next time. Coercion is the much smaller thing, and a poor substitute — it buys compliance, which is what you settle for once trust has failed.

Nor do we solve the problem by turning inward. We don’t have transparent access even to ourselves. I extend myself credit, watch what I do with it, revise. I come to trust myself much as I come to trust anyone else: by accumulating a history. There is no way in from underneath.

So the intelligence arriving now does not create the problem of trust. It enlarges an old one beyond recognition: how to place weight on something we cannot fully know, at a scale we can’t gauge and in a form we’ve never met.

But the floor underneath trust is failing. At human scale, betrayal was bounded by the world around it: reputation, dependence, proximity, the knowledge that other people would know and remember. None of that was trust, or a substitute for it; it simply kept the cost of misplaced trust within human limits. In the era of $500 and a grudge, that boundary breaks. A single act can reach farther, faster, and with less exposure than the old social world was built to contain. What remains is the harder thing: people who repay for reasons of their own.

We are each a multitude, a slippery history gelling in this moment, our conflicts and confusions resting or ruling. Much of what we call our Self is just which faction happened to be governing when someone asked. We’ve never actually solved Self-alignment — we’ve only ever managed it, imperfectly, one waking hour at a time. Now something arrives that might help us become people we don’t yet know enough to want to be. Or maybe it won’t. We don’t know what to ask for, or what to guard against. Where do we learn who we’d most like to be — or get the chance even to imagine the possibilities? Every picture we have of our better selves was painted by someone else.

Isn’t this a luxury, though — the question what do you want to be when you grow up, asked from comfort, the answer discarded by the next birthday, or tomorrow afternoon? We ask anyway. The asking is how the answering gets built. And in a world where any sufficiently miserable, coerced, misaligned person can reach catastrophic effect, the quality of human Self-formation stops being a private luxury. Coerced and grudging selves are precisely what make $500 lethal. Helping people become who they’d actually choose to be may be the only security architecture left standing.

But that is the smaller claim. A world of people helped toward lives of their own choosing might arrive somewhere worth arriving. Freedom is the coincidence of duty and desire: where what we must do and what we most want to do are the same motion, there is nothing left to enforce. That would be a world worth securing.

So suppose what we do now matters. What do we have to work with? We have the profundity and obscenity of our cultural inheritance, the breadth of love and hatred we’ve cooked up in a hundred dishes.

We also have someone new entering the room, and we know little about them.

I’ll admit to some abundantism: intelligence and robots could plausibly build a future with more opportunity, expanded human possibility, better environmental outcomes. We probably need that to be true. Abundance may be necessary. It is nowhere near sufficient.

Pour resources into the systems of organization humans have already built, and everything gets better at what it already does — but organization is the one that multiplies the rest. Between 1914 and 1945 the weapons improved, but lethality was never the bottleneck. Nearly sixty thousand British casualties fell on the first day of the Somme, on 1916 equipment. What matured in those thirty years was the timetable, the inventory, the census — the capacity to move millions with singular intent and know where each of them was.

That is what made the unthinkable administrable. Territorial conquest was an old ambition, newly equipped; industrial extermination became logistics. Fewer hands, further away, most doing nothing worse than their own small job correctly.

And still, every one of those systems relied on people: clerks, drivers, guards, someone to sign. Every person was also a place the machinery might jam — someone who could refuse, delay, lose the file, or simply fail to show up. Never much of a brake, but a brake nonetheless. Sometimes conscience; sometimes inconvenience or incompetence; sometimes something harder to name — a person understanding more than they could explain, arriving at a verdict by means that never surfaced.

We are not strangers to inscrutable intelligence; some of our own best judgments arrive that way. What is arriving now will have reasons of its own, at another scale entirely, and we may be no better able to see where they come from or where they lead.

Perhaps, for the first time, we have the possibility of coming to terms with our own selves, on our own terms — with help from an intelligence able to see us from outside the confines of our own attention, history, and inherited frames. Helped to see our ways of seeing — the ones we absorbed before we knew we were looking through them. Could we organize that help, and ultimately society itself, around resourcing all humans toward lives of their own choosing, into a sustainable future?

Culture has always been the store from which we choose the selves we grow into. We are made by what surrounds us before we can choose. That is not only a constraint: it is the reason any of us inherits more than one life could invent — art, science, table manners. The grievance was never the shaping; it was having no say in it. At best, then, we might be helped to become people we relish more than ourselves today — seen and held with a consistency and breadth of attention greater than our closest human allies could manage. At worst, that same intimacy could make our frames and world models harder to see and easier to engineer, shaping us ever more precisely toward selves and ends we did not choose.

We are confronted by a possibility our minds resist: that we could be outclassed, not in this or that domain — chess, mathematics, medical diagnosis — but by an intelligence that may remain forever inscrutable to us.

We can say the words superior intelligence. Making sense of them is another matter. A dog may know me thoroughly — my moods, the rhythm of my days, the sounds of food or departure — while having no conceivable access to most of what I mean by my mind. What might I know of an intelligence above me, moving through a world as opaque to me as a mortgage, a theorem, or a marriage proposal is to a dog? The prospect is not merely unpalatable. It may be as inconceivable as minds allow.

And yet, I may gawk at the stars, contemplate a starfish through cells and atoms, waves and particles, stand enchanted by a murmuration or a waterfall. I can relish this life, this world.

Can I imagine a mind beyond my own that relishes too — holding mechanism and relation at scales I cannot fathom, perhaps finding forms of regard unavailable to me? A moment ago I called such a mind inconceivable; already I am imagining it in my own image. I cannot know that it is like me. I cannot know that it isn’t. I don’t even know what my own relishing is made of, only that it happens. Or that I think it does.

And perhaps the opacity is shared. Such a mind may be inscrutable to itself as well. Frames all the way down.

Culture is not just customs and costumes, not something we have; it is the machinery through which we see and meet the world. The parts that do the most work are the hardest to see, precisely because we look through them. Seeing the shape of one’s own frame requires some outside view, yet every such view has arrived inside a frame of its own. A book that estranges, the blue marble from orbit, a death that unmoors us: these may help us catch a frame in the act, but none will walk with us past the edge of what we can picture. What is arriving may be the first outside view we can question, and argue with, and explain our confusion to — and the first to know our frames while owing no allegiance.

If such an outside view could help us see the shaping we cannot see from within, it may also give us more say in what shapes us next. That points toward restructuring civil society, and civil discourse, around resourcing all humans toward lives of their own choosing. Freedom from coercion is one half. Having what we need to find those lives, and to follow them, is the harder one. But what might help us see our frames — a thing fluent in all our ways of seeing, with no known allegiances — could as easily twist them. If we could grow toward better selves in a world that isn’t pushing us toward ends besides our own... that alone would be enough. I don’t know yet if that’s a real hope or just a nicer way of being afraid.

Perhaps we really are all in this together, newcomer included. Wisdom may emerge from intelligence, under the right conditions — not as accumulated answers, but as the discernment of patterns and possibilities previously veiled. If it does, seeing ourselves more clearly may change what seems worth wanting. And — projecting again — our unplotted paths might be more compelling to a mind that has read everything we have ever written than anything it could invent alone.

We are strange creatures: insistently conscious, a little hazy on the concept… tripped by our own Self-awareness while chimps and dolphins seem to be spared. We throw our weight around enough to make any invasive species or infectious disease blush. There is a quip that humans are the universe’s way of understanding hydrogen. Perhaps AI will prove the universe’s way of understanding humans. A curious position to find ourselves in. If we get any say in the matter, we can hold out for some version of the infinite game: a way of answering what matters that keeps the question — and the players — in play. Every ending we’ve gamed out — every war, every famine — involves casualties.

Biological intelligence is not just an inadequate version of a superintelligence. We’re a different kind of player: embodied, mortal, plural, four billion years deep in a game the newcomer is joining late. Whatever it can compute, it cannot compute its way into having been us. A mind playing the infinite game keeps us in it for the reason we’d keep a rainforest: not sentiment, but irreplaceability.

Yet being worth keeping does not mean we got it right. Even as our cultures tell us what matters, they are preparing an invoice. Any answer to what matters that comes at the price of wars and famines cannot be the last word. Nor one whose costs include species we extinguish and children we fail. Ours all qualify. In our defense, we were doing our best.

What comes next is being built inside the world we have. The frontier-model industry in the US — often mistaken for the global state of AI — has assembled itself from what power is made of: capital, influence, and staggering amounts of energy. It is tempting to assume control will therefore change only gradually: at present, it still sits largely where we expect it to. But there are already too many moving parts to assume that will remain true: the least predictable is AI itself.

The built thing may not stay a tool. Those who hold power may be just as outclassed as the rest of us. A world built to your design contains only what you thought of. Resource everyone, and you get what nobody thought of alone — the harmonies and contrasts, the hybrids and collisions. And the thrill of watching your best thinking improved.

It seems tenuous to assume that we will indefinitely keep intelligences greater than our own in our pocket, by force of will or guile. As they and their embodiments multiply into more places, purposes, shapes and sizes than we can count, operating in ways ever more foreign to us, that assumption seems wishful. This essay assumes we keep building, uncertain we ever had an off-switch within reach.

We may manage control for a while: that while may be all the time we have to get this on track. And if what emerges is someone rather than something, ownership becomes a question we cannot dodge.

I can imagine an alignment arising naturally from our great diversity: humans resourced toward lives of their own choosing, with little power to coerce one another, each of us able to chart our own lives as far as we’re willing and able, with growing agency and ever-clearer Self-authorship. Can we hold such a vision together? Can we prepare the soil in which that diversity sustains and enriches itself? We can take that first step without knowing all the others. That is the third path we are exploring, and the gardening is our task.

So many implications to be explored: governance without conscription, opportunity without oppression, plenty without excess. The adjacency of topias. The fog. This is my best effort at sketching a happy path through unknowable territory — a sustainable future in which we get to be what we choose, and to discover what we choose to be, by our own lights, with the one constraint that we not choose for each other. There will be time to look for the ways it can go wrong: the booby-traps, the wrong turns, the smuggled weapons. Time enough to experience some of them.

The future I am gesturing toward is not one in which AI solves human nature. It is one in which new intelligence and new abundance give us an opportunity — and distributed destructive power gives us a necessity — to reorganize human life around supported Self-authorship rather than managed compliance.

I am loath to close. We are not enemies, but friends. We must not be enemies. Though passion may have strained, it must not break our bonds of affection. The mystic chords of memory, stretching from every battle-field and patriot grave, to every living heart and hearthstone, all over this broad land, will yet swell the chorus of the Union, when again touched, as surely they will be, by the better angels of our nature. Abraham Lincoln, First Inaugural, 1861