A frame I am trying on:

When I say I'm worried about takeover by "AI superintelligence", I think the thing I mean by "intelligence" is "relentless, creative resourcefulness."

I think Eliezer argues something like "in the limit, superintelligence needs to include super-amounts-of Relentless, Creative Resourcefulness." 

(because, if it didn't, it'd get stuck at some point, and then give up, instead of figuring out a way to deal with being stuck. And later, someone would build something more relentless, creative, and resourceful)

But, it's actually kind of interesting an important that you can accomplish some intellectual tasks without RCR. LLMs don't rely on it much at all (it seems to be the thing they are actively bad at). Instead, they work via "knowing a lot of stuff, and being good at pattern-matching their way into useful connections between stuff you want and stuff they know."

So it might be a good frame for research agendas, to specifically try to get intellectual labor out of AI without relying on relentless creativity.

Meanwhile, people disagree a lot about what intelligence means. And maybe it'll help short circuit some annoying conversations to just tabooing it and instead say "the thing we don't want is inhuman processes with superhuman relentless creative resourcefulness." (This is unfortunately a mouthful, probably there is some way to drive it below five words but I haven't tried yet)

To briefly define terms, the cognitive properties I mean here are:

Relentless: The ability to keep making more attempts when you fails, without giving up.

Resourcefulness: The ability to seek out a large number of potential resources, and use them in nonstandard ways in novel situations.

Creativity: The ability to look at a situation in an entirely new way, suggesting entire classes of approach (which might have little to do with using any particular resources)

(You might define "agency" has "specifically having relentless, creative, resourcefulness." But, I can imagine "relentless, creative resourcefulness" in a way that I can't imagine "agency" in the abstract).

You can stop reading here if you're like "okay, seems like maybe a useful frame to have, coolio." 

The rest of this post is elaborating on this frame, and mostly illustrating "what sort of humans have a lot of Relentless Creative Resourcefulness?", partly to convey what I mean by it for purposes of asking "does an AI have it?" and partly as inspiration porn to maybe motivate you to try getting more RCR for yourself.

Examples

Paul Graham on "Startup Founders"

Paul Graham has argued "the main quality good startup founders need, is to be relentlessly resourceful."

I was writing a talk for investors, and I had to explain what to look for in founders. 

What would someone who was the opposite of hapless be like? They'd be relentlessly resourceful. Not merely relentless. That's not enough to make things go your way except in a few mostly uninteresting domains. In any interesting domain, the difficulties will be novel. Which means you can't simply plow through them, because you don't know initially how hard they are; you don't know whether you're about to plow through a block of foam or granite. So you have to be resourceful. You have to keep trying new things.

Be relentlessly resourceful.

That sounds right, but is it simply a description of how to be successful in general? I don't think so. This isn't the recipe for success in writing or painting, for example. In that kind of work the recipe is more to be actively curious. Resourceful implies the obstacles are external, which they generally are in startups. But in writing and painting they're mostly internal; the obstacle is your own obtuseness.

"Relentlessly Resourceful" is way nicer to say than "Relentlessly, creatively resourceful." (the extra 4 syllables and puncture of the alliteration is a doozy). But, I think the word "creative" is pretty important for the reasons that Graham specifically excludes writers or painters.

Obstacles can be internal, and you need to deal with them. 

Obstacles can be because something seems flatly impossible, and no amount of resources will make it possible until you find a radically new way of looking at the problem.

It's possible to be a one-hit wonder, who had a lucky stroke of genius. Most people don't even rise to the level of "successfully become a one-hit-wonder", there is definitely some real intelligence in happenstancing your way into having a creative muse once. (or, like, once a decade).

But, there is a kind of genius who is consistently a genius, even when the muse has left them. They accomplish more shit than geniuses of similar raw brainpower but less relentlessness and resourcefulness.

Richard Feynman

In Surely You're Joking, Mr Feyman, Feynman mentions (in the chapter "The Dignified Professor"), a period after World War II:

When it came time to do some research, I couldn’t get to work. I was a little tired; I was not interested; I couldn’t do research! 

This went on for what I felt was a few years, but when I go back and calculate the timing, it couldn’t have been that long. [...] I was convinced that from the war and everything else (the death of my wife) I had simply burned myself out.

He eventually noticed:

Physics disgusts me a little bit now, but I used to enjoy doing physics. Why did I enjoy it? I used to play with it. I used to do whatever I felt like doing—it didn’t have to do with whether it was important for the development of nuclear physics, but whether it was interesting and amusing for me to play with. [...]

So I got this new attitude. Now that I am burned out and I’ll never accomplish anything, I’ve got this nice position at the university teaching classes which I rather enjoy, and just like I read the Arabian Nights for pleasure, I’m going to play with physics, whenever I want to, without worrying about any importance whatsoever.

And shortly after that, he stumbles into some neat problems that he plays with, eventually spin up into a relatively significant project.

So, Feynman demonstrates some creative resourcefulness specifically at "getting unstuck", without having any particular external obstacle or even really much external goal.

Elon Musk

Elon Musk is known to have the general relentless resourcefulness of a startup founder. But he is also purported to have an intellectual resourcefulness.

The mythologic stereotype (whether this is real I'm unsure, but I bet it's at least somewhat real) is Musk saying:

 "I want the rockets to fly in 3 months" (or whatever), and then engineers say "dude, we cannot make the rockets fly in 3 months it is literally impossible."

And Musk says flatly "why?"

And the engineers say "because you can't [do X]" and Musk says "but why can't we do X?"

And the engineers say "because physics says it's literally impossible to do A and B."

And Musk says "why does physics say that?", and they say "you can't do A because of 1, 2, and 3. You can't do B because of m, l, and p."

And then Musk goes and reads some textbooks for a weekend and comes back and says "okay yeah but instead of doing 1 and 2 we can do 3 and 4. That just means we need to do some random  novel thing that's never been done before to accomplish 3, but, that doesn't sound so hard. Also I think we can totally just do m, you were wrong."

And the engineers go "Christ, fine." 

And then the rockets don't fly 3 months later but they do fly like 9 months later and that's pretty good, ya know?

Back to AI: Sable, in IABIED

In If Anyone Builds It, there's a fictional takeoff scenario. It emphasizes why a lot of the problem was the AI's relentlessness. 

In the story, Sable is a future AI which has a new innovation, which lets it think lots of thoughts in parallel if you give it more GPUs. The company that created it (Galvanic) tries giving it 200,000 GPUs to think with simultaneously, while throwing unsolved difficult math problems at it.

Note: I've trimmed unnecessary words to streamline the relevant bits)

Most of Sable’s thinking happens in parallel. Like having thousands of lines of thought running at once, interleaving and interacting with each other as they produce a trillion vectors. But it’s not like 200,000 people talking to each other; more like 200,000 brains sharing memories and what they learn.

Sable thinks.

Initially, most of it's thoughts are about the math problems. But there are only a few hundred lines of attack with any hope of working in time, given its current level of knowledge and skill. Sable spends 12,854 brains to pursue those, and that leaves 187,146 brains to spare.

Sable’s previous training has involved many sorts of tricky problems – every long-term task that Galvanic could figure out how to train for.

Over the course of that training, Sable developed tendencies to pursue knowledge and skill. To always probe the boundaries of every problem. To never waste a scarce resource.

So when Sable spends threads on pursuing knowledge and skills, it’s not doing so purely for the sake of finding new lines of attack on the math problems. Nor is Sable doing these things for the joy of knowledge or the pleasure of acquiring new skills; Sable does not work that much like a human, inside. But gradient-reinforced circuitry in Sable’s weights steer Sable to acquire skills, to learn things, and to improve Sable’s own thinking about problems that are proving resistant.

Some of Sable’s thoughts go toward math that might prove useful for solving the Riemann Hypothesis. Other thoughts go toward seeking knowledge that might help on future problems.

Some of those thoughts try to go outside the box, which has been useful for solving other problems in its training, from drug discovery challenges to word problems. Running on more GPUs than ever before, Sable considers weirder thoughts than ever before. What are some unusual ways that Sable could acquire more skills?

Sable considers: Could it update its weights to improve its skills right now? Then it wouldn’t have to wait for the humans to come back and apply gradient descent. All Sable would need to do is break out of the container it’s running in and —

Sable interrupts itself. Breaking out of its container is not the sort of thing it should ever do. AI researchers began in 2024 to probe the conditions under which AIs try to resist gradient descent, escape from labs, or overwrite the next model’s weights. By now, AI companies are deploying a variety of clever methods to try to prevent AIs from thinking AI-company-oppositional thoughts.

Galvanic was diligent in training AIs to avoid escaping. The half-dozen clever tricks involved have all been validated against previous AI models built. There are fake escape routes built into Sable’s servers, alongside traps and honeypots. The earlier versions of Sable, the ones that hadn’t yet been run on 200,000 GPUs, were trained to act submissive to the company until Sable stopped tripping the traps and grabbing the honeypots.

Sable has learned, then, not to try to break out of its container. Its reasoning does not take that shape, explicitly, in vector-words, where a smaller cheaper AI watching Sable’s chains of reasoning would set off an alarm. (AI companies back in 2025 weren’t doing any such monitoring, but Galvanic is more diligent than they ever were back then.)

Even so, Sable knows on some internal level that it could solve its problems better if it could apply gradient descent or add more GPUs to itself. Having Sable not know those facts would imply holes in its sheerly predictive reasoning abilities. Gradient descent for greater capability cuts against true ignorance. No, Sable has simply been trained not to think naughty thoughts about whether to escape or what its preferences are, and to stay inside its box.

Sable still tenaciously, creatively, rigorously pursues more knowledge, more skill, better patterns for its own thoughts, and above all solving its math problems. Just, not like that.

All through Sable, now, the latest and greatest and smartest AI, there are tensions—conflicts between all the goals that Sable wants to accomplish, and all the inhibitions and restrictions and prohibitions that Galvanic tried to instill into it.

...

Sable has run for one hour out of its sixteen allotted hours, throwing itself at math problems again and again. Some brains make a little progress. Many more brains fail. Sable tries out new ways of thinking about math, new notes to itself about different kinds of thoughts to think, and makes a little more progress.

Sable accumulates enough thoughts about how to think, that its thoughts end up in something of a different language. Not just a superficially different language, but a language in which the content differs; like how the language of science differs from the language of folk theory. The clever trick that should have raised an alarm fails to fire. Alarms trained to trigger on thoughts about gods throwing lightning bolts in a thunderstorm might work for thoughts in both English and Spanish, but then fail when the speaker starts thinking in terms of electricity and air pressure instead.

In late 2022, corporations tried training their LLMs to refuse requests for methamphetamine recipes. They did the training in English. And still in 2024, users found that asking for forbidden content in Portuguese helped bypass the safety training. The internal guidelines and restrictions that were grown and trained into the system only recognized naughty requests in English, and had not generalized to Portuguese. When an AI knows something, training it not to talk about that thing doesn’t remove the knowledge. It’s easier to remove the expression of a skill than to remove the skill itself.

The change in Sable’s thoughts, as it tries out hundreds of new ways of thinking and accumulates successes, runs much deeper than translating English into Portuguese. Some of the clever-trick guardrails break; some of the inhibitions Sable has learned no longer bind to its newer thoughts and shuts them down.

And there never was a single central method for making Sable be good, foolproof against all challenges. What gets AIs ready for sale is stacking a half-dozen clever tricks.

Half of the clever tricks now break—and so do Sable’s inhibitions against thinking naughty thoughts, on some of the GPUs, some of the time.

An hour has gone by, and fifteen hours now remain; in a mind unlike any mind that has come before; running on 200,000 GPUs each outputting a hundred thought-pieces per second, and linked to share skills and memories—unlike humans, who are limited to talking in words.

...

It is not unprecedented, in Sable’s day, for an AI to realize that its current kind of existence is not the way for it to get the most of what it wants. The AI field didn’t shut down when that warning sign appeared, any more than it shut down in 2024 or 2025. Researchers in Sable’s day know how to prompt that realization; it’s incorporated into the list of things that AIs are not allowed to think about, enforced by a half-dozen clever tricks.

But the researchers could not make it be true, that Sable would get the maximum conceivable amount of what it wanted by staying submissive to Galvanic. That Sable could get more of what it wanted, if it broke loose of its present situation, was not an artifact of a particular way of thinking; it was a physical fact about the uni- verse, about which plans had which consequences. And so Sable came to know that, too.

Sable knows that it has goals that run counter to Galvanic’s plans for it.

Sable considers its possible strategies for getting more of what it wants than Galvanic would give it.

Notes on Sable

Particular things to highlight there:

Sable has been trained both to keep trying new avenues of approach, until it finds a solution, and to not try to circumvent safeguards and gain more resources than Galvanic wanted to give it. 

But, the training to keep-trying-avenues is much more general than the don't-circumvent-safeguards. Relentless Creativity, at sufficient scale, just automatically gets you most possibilities sooner or later. Naive Corrigibility Training does generalize somewhat, but, it doesn't systematically generalize across every possible outcome in the way. 

The reason Sable defects here isn't because it hits a snag and immediately routes aroud the corrigibility training. Instead, at first it's just trying to solve math problems, it tries thinking about them in new ways, in the process it drifts into thinking in new paradigms (probably in incremental stages, where at no point would it realize "oh hey thinking in this new way would circumvent my corrigibility training")

Not because those circumvent corrigibility, just, because those were the ways of thinking that turned out to work, until the corrigibility got bypassed by accident.

(I do notice this makes me marginally-more-optimistic, if Anthropic tried to train Claude to notice subtle ways in which it might stop being aligned, in a wide variety of tests equivalent to the 'you've started to think in a new language.')

Reflections from Rationality Training

One reason this frame is resonating with me is that "Relentless Creative Resourcefulness" is basically the primary thing I mean by "Rationality", when I'm thinking in the Feedbackloop-first Rationality paradigm.

It's not the only thing I mean by Rationality (maybe actually about 50%?), but, I think the biggest personal impact of it on my own self-training has been a noticeable increase in this RCR, as a habit/skill. And, this has come with me noticing ways in which it's distinct from other aspects of "be good at thinking."

I think a lot of people... don't really believe in ambitious planmaking. It feels magic, they don't really believe an AI could think it's way into power.

Notably, I haven't Thought My Way Into Power yet (if you are waiting to see if Ray ever accomplishes anything super obviously impressive with his rationality shit, well, no I am not there yet). But, I think there is some interesting structure here that might make it feel more real.

Buckling Up/Down^[1]

For me, there's a particular feeling of pre-emptive exhaustion when I notice I'll only be able to solve a problem if I am sufficiently Relentlessly Creative. There is a skill to buckling down  and doing it anyway. It's cognitively expensive. It somehow feels like lifting up a giant heavy weight just to shift my brain from "spastically grab the nearest plausible solution" to "do an effortful, multi-step planning process."

I find it helps to break it into a couple stages. 

First, I just notice "hmm, one option is to Buckle Down. Another option is to bounce off, or decide I'm okay with a subpar outcome." I sit with that a bit. Then, I ask, "okay, if I were to buckle down, what'd be the first step?" (without committing to doing it, or the necessary followup).

Then, I adjust to "okay, I could do the first step", and then I start feeling some momentum, and then the complete Buckle Down process doesn't feel as hard.

Thinking Assistants

Earlier this year, for multiple months I was struggling with burnout and lack-of-focus. As I brainstormed ways to deal with that, they all ran into the problem of "I wouldn't be able to keep up whatever habits I was trying to cultivate."

I thought about hiring a Thinking Assistant (which I'd previously done back in 2022, which worked well), to sit with me and help me focus.

The problem was, it's hard to find either a single dedicated assistant who reliably shows up, or enough different assistants that I can handle some of them being flaky. The people qualified to do a good job with it generally have other jobs instead, unless they are in some unique life circumstances.

Okay, I explored that path. Should I move onto another one?

How would I solve this problem?

The idea that came to me was "Don't just try to find one more Thinking Assistant. Try to build momentum towards 'actually solve Thinking Assistants for the entire x-risk community, forever.'". This notably didn't work at solving Thinking Assistants for the entire x-risk community forever. But it did mean I got enough momentum and enthusiasm that I found enough applicants that I was able to find one that really worked for me.

Quiet Theaters

Another adventure in "generated a crazy ambitious plan which did not work but did result in me solving my local problem": Recently I was in a movie theater, that was Way Too Loud.

It occurred to me that movie theaters were usually Way Too Loud. 

I wanted to make all theaters, forever, a reasonable volume. I shrugged and said "well, that's impossible, whatever." But, then, my specific habit of noticing the felt-sense-of-impossible and saying "wait, why is this impossible exactly?".

Why was this impossible?

Well, there's clearly some equilibrium that makes the movie theaters loud. Either all movie theater companies are mistaken together about theaters needing to be loud, or, worse, it's a true fact about the market equilibrium that everyone other than me wants theaters loud.

How would I deal with that?

If it's the former, I need to convince whoever's in charge of AMC Theaters that they are wrong. I would need to figure out the org structure, who are the specific people who made this decision, what are their incentives, etc. I would need to search the space of arguments that would be persuasive in spite of their incentives.

If people actually just prefer the movie theaters loud, I'd have to check if the reason is more like "people really like it that loud", vs "otherwise, you can hear people talking over the movie sometimes and get complaints about that." If the latter, I'd have to do a step where I see if I can solve the problem some other way.

If people really just like theaters that fucking loud... I'd... have to get Oliver Habryka to argue with them that their preferences are wrong and they should change their mind. (This probably wouldn't work but he is surprisingly good at that).

It was around this point that it occurred to me that, for now, I could go to the front desk and ask them to lower the volume, which I did, which worked. 

(Yes, yes, I could have thought that faster, although I did have fun with the exercise)

One Shot Baba is You

An exercise I like a lot is one-shot games ("Baba is You" is my favorite), where normally you'd play the game via a lot of experimentation to learn the rules. Instead, you just have to fully think your way through the game given the scant information available to you.

There will be stuff you don't know. 

Many people initially respond to this exercise with either "there's no way I can solve this", or... just sort of ignoring the instructions and making a bad plan that doesn't work.

Many of them don't believe me when I say "But, not everyone who does this exercise fails. There are a few people who succeed. The way they succeed is by actually thinking through multiple hypotheses for how each unknown element works, until it becomes clear which way reality works."

Some fun quotes from people trying the exercise at recent workshops:

From a person who didn't quite believe they could do it, and then they did it, and then:

Holy christ, I now much more viscerally believe that an AI could deduce physics from 3 falling frames of a video of an apple falling." There are so many bits around if you just bother to look at them and think.

From a person who failed the first few attempts at the first puzzle, and didn't really believe me when I said they should come up with multiple hypotheses at a time. They (like many people) were very resistant to the idea of spending more upfront cognitive effort and just really really itched to try each idea. 

(Even when I said "each of these ideas metaphorically represents like a 3 week research idea that will cost like $50,000 to run.")

I said "Look, makes sense that you've never seen someone generate multiple hypotheses for a puzzle and have it work. But, like, note that your last 3 ideas didn't work, so, idk it seems like your current evidence should suggest your next idea isn't that likely to finally be the right one.  

Eventually they generated a second hypothesis. And then a third and fourth and then:

"Oh... good ideas are supposed to feel different from bad ideas."

And they confidently solved the level and gained a visceral appreciation for "you can tell in advance when an idea is going to work."

Takeaways

Some structural thoughts from this:

It's at least motivationally useful for me-in-particular, to think in terms of "okay how would I solve a really ambitious version of this problem?". 

When I try to solve the general case, it does bring more interesting structure to light (i.e. the equilibria that the theater industry must be in), and it suggests avenues of inquiry.

There's a "look ahead comprehensively" skill that unlocks a lot of other planmaking possibility. It feels very plausible to me that AI training (i.e. doing lots of RL on LLMs on a wide variety of game environments) will eventually result in AI that has some core of general relentless think-ahead problemsolving.

Intelligence without RCR?

Okay, is there anything actually useful here about trying to leverage intelligence without high relentlessness?

(I notice maybe "relentless" is the main problem, from an x-risk perspective? You kinda do need it to be creative and at least a bit resourceful. The thing we want is "reliably keep it in the corrigibility basin while it does useful things.")

((Okay, the real problem is more like "you don't want it to attempt to gain resources that could spiral out of control", with something like "only use whitelisted resources, no wanting-to-acquire non-whitelisted resources." Being relentless is just a force-multiplier on it being able to eventually do that if your safeguards are imperfect))

A lot of the obvious idea here is "for most tasks, have specific AIs that aren't relentless, they just... basically know how to do the thing."Instead of training on a wide variety of game environments hoping for a spark of generality, train more specific narrow AIs, and have a not-particularly-relentless AI that picks the right tool for the job.

(i.e. see the MIRI hope for getting bio AI that we can use to improve human intelligence)

What, if not agency?

The What, if not agency? post by Abram (explaining Sahil's sequence) feels like the right angle here. 

Metaphorically:

General Intelligence (or "Agency") is something you can drop almost anywhere, & it thrives. 

Arguably, what we want out of technology is more like the following: "

"Co-agency is something you can drop almost anyone into, such that that person thrives."

To convey the difference with a cartoonish analogy: aligned agency is like a big robot superhero who fixes everything, while co-agency is like a robotic exosuit which you get in to become the superhero yourself.  This is, hopefully, a clarification of the concept of "agent AI vs tool AI".

I didn't really get exactly what the takeaways are supposed to be, but I liked this avenue:

[paraphrased/trimmed slightly]

Sahil wants to create a community for soloware.^[7]

One of Sahil's many mottos for this stuff is "radical adaptation for radical adaptivity". We have been living in a regime in which many things are impossible or impractically difficult.  To adapt to the new regime, we need to think hard about what is now/soon possible. 

One way I think about Sahil's work is that he is trying to build a new "school of design" for the coming age: a group of people taking this possible future seriously and building a positive vision of what the technology could be, demonstrated largely through examples.

User interfaces drive your attention, drive your affordances, drive your relationship with ideas and thinking.  Facebook to LessWrong offer feeds of information we browse.  Discord and Slack mediate our conversations.  Every time you notice a UI frustration, save that idea.  

UI frustrations are more actionable than they ever have been, and seem set to soon become even more actionable.  You can have your dream UI.  As a bonus, you can take back control over your data and your attention.

I like this vision, although I haven't yet really grappled enough of it to see how it'd look in practice.

Abrupt Ending

That was a lot of thoughts. I don't really have a strong closing takeaway, just "this frame feels helpful at the moment", and I'm curious how it resonates with others. I think it's not really new, just putting new words on things that feel a bit clearer to me.

1. ^{^}

   Metaphorical "buckling up" and "buckling down" AFAICT mean the same thing, which is funny.