Ryan had suggested that, on his model, spending ~5%-more-than-commercially-expedient resources on alignment might drop takeover risks down to 50%. I'm interested in how he thinks this scales: how much more resources, in percentage terms, would be needed to drop the risk to 20%, 10%, 1%?

An update here.

Following up on [1] and [2]...

So, I've had a "Claude Code moment" recently: I decided to build something on a lark, asked Opus to implement it, found that the prototype worked fine on the first try, then kept blindly asking for more and more features and was surprised to discover that it just kept working.

The "something" in question was a Python file editor which behaves as follows:

The remarkable thing isn't really the functionality (to a large extent, this is just a wrapper on ast + QScintilla), but how little effort it took: <6 hours by wall-clock time to generate 4.3k lines of code, and I've never actually had to look at it, I just described the features I wanted and reported bugs to Opus. I've not verified the functionality comprehensively, but it basically works, I think. 

How does that square with the frankly dismal performance I've been observing before? Is it perhaps because I skilled up at directing Opus, cracked the secret to it, and now I can indeed dramatically speed up my work?

No.

There was zero additional skill involved. I'd started doing it on a lark, so I'd disregarded all the previous lessons I've been learning and just directed Opus same as how I've been trying to do it at the start. And it Just Worked in a way it Just Didn't Work before.

Which means the main predictor of how well Opus performs isn't how well you're using it/working with it, but what type of project you're working on.

Meaning it's very likely that the people for whom LLMs works exhilaratingly well are working on the kinds of projects LLMs happen to be very good at, and everyone for whom working with LLMs is a tooth-pulling exercise happen not to be working on these kinds of projects. Or, to reframe: if you need to code up something from the latter category, if it's not a side-project you can take or leave, you're screwed, no amount of skill on your part is going to make it easy. The issue is not that of your skill.

The obvious question is: what are the differences between those categories? I have some vague guesses. To get a second opinion, I placed the Python editor ("SpanEditor") and the other project I've been working on ("Scaffold") into the same directory, and asked Opus to run a comparative analysis regarding their technical difficulty and speculate about the skillset of someone who'd be very good at the first kind of project and bad at the second kind. (I'm told this is what peak automation looks like.)

Its conclusions seem sensible:

tl;dr: "they think in types, not states", "they're anxious when there's no library to lean on", "they trust abstractions more than they build them", and "tend to think about one abstraction level at a time".

Or, what I would claim is a fine distillation: "bad at novel problem-solving and gears-level modeling".

Now, it's a bit suspicious how well this confirms my cached prejudices. A paranoiac, which I am, might suspect the following line of possibility: I'm sure it was transparent to Opus that it wrote both codebases (I didn't tell it, but I didn't bother removing its comments, and I'm sure it can recognize its writing style), so perhaps when I asked it to list the strengths and weaknesses of that hypothetical person, it just retrieved some cached "what LLMs are good vs. bad at" spiel from its pretraining. There are reasons not to think that, though:

• I first asked for an "objective" difficulty assessment of the codebases, and then to infer strengths/weaknesses by iterating on this assessment. I don't think "the user wants to confirm his prejudices, I should denigrate myself" is really inferrable from that initial task, and the follow-up was consistent with it.
• By its own assessment (see the spoiler above), both codebases "are well-architected" (the difference was in how much effort ensuring the well-architecturedness required on my part). If it recognizes that it wrote both, and considers both well-written, why would it jump to "the user is asking what I'm bad at"? I suppose it could have noticed signs of external tampering in Scaffold...
• I obviously provided it no additional context regarding myself or my experiences and beliefs, just the raw code + "run a comparative analysis" and "infer hypothetical strengths/weaknesses" prompts.

Overall... Well, make of that what you will.

The direction of my update, though, is once again in favor of LLMs being less capable than they sound like, and towards longer timelines.

Like, before this, there was a possibility that it really were a skill issue on my part, and one really could 10x their productivity with the right approach. But I've now observed that whether you get 0.8x'd or 10x'd depends on the project you're working on and doesn't depend on one's skill level – and if so, well, this pretty much explains the cluster of "this 10x'd my productivity!" reports, no? We no longer need to entertain the "maybe there really is a trick to it" hypothesis to explain said reports.

Anyway, this is obviously rather sparse data, and I'll keep trying to find ways to squeeze more performance out of LLMs. But, well, my short-term p(doom) has gone down some more.

I am interested in trying out the new code simplifier to see whether it can do a good job

Tried it out a couple times just now, it appears specialized for low-level, syntax-level rephrasings. It will inline functions and intermediate-variable computations that are only used once and try to distill if-else blocks into something more elegant, but it won't even attempt doing things at a higher level. Was very eager to remove Claude's own overly verbose/obvious comments, though. Very relatable.

Overall, it would be mildly useful in isolation, but I'm pretty sure you can get the same job done ten times faster using Haiku 4.5 or Composer-1 (Cursor's own blazing-fast LLM).

Curious if you get a different experience.

Eh, I don't think this really disproves the OP (who mentions bejeweled iPhones as well). This isn't really a "$100k phone", it's a normal $1k phone with $99k of jewelry tastelessly bolted on. You're not getting a 50,000 mAh battery, a gaming GPU, and Wi-Fi 12 plus 6G support in a 300 gram package here. Which, indeed: why not?

I've previously left a comment describing my being fairly unimpressed with Claude Code Opus 4.5, and leaving a few guesses regarding what causes the difference in people's opinion regarding its usefulness. Eight more days into it, I have new comments and new guesses.

tl;dr: Very useful (or maybe I'm also deluding myself), very hard to use (or maybe I have skill issues). If you want to build something genuinely complicated with it, it's probably worth it, but it will be an uphill battle against superhuman-speed codebase rot, and you will need significant technical expertise and/or the ability to learn that expertise quickly.

First, what I'm attempting is to use it to implement an app that's not really all that complicated, but which is still pretty involved, still runs fairly nontrivial logic at the backend (along the lines of this comment), and about whose functionality I have precise desiderata and very few vibes.

Is Opus 4.5 helpful and significantly speeding me up? Yes, of that there is no doubt. Asking it questions about the packages to use, what tools they offer, and how I could architecture solutions to various problems I run into, is incredibly helpful. Its answers are so much more precise than Google's, it distill information so much better than raw code documentation does, and it's both orders of magnitude faster than StackOverflow and is smarter than the median answer you'd get there.

Is Claude Code helpful and speeding me up? That is more of an open question. Some loose thoughts:

• Once I and Opus 4.5 have nailed down the architecture, you have to actually put it into code. If I've made the correct call regarding the scope, Claude Code can be mostly trusted with implementation as well.
• What is "the correct call regarding the scope"? I'm still gauging this, but some pointers:
  • You can trust it to implement an algorithm that traverses a complicated but well-specified data structure and transforms it in some way.
  • You can trust it to implement the infrastructure for trivial data roundtrips through nontrivial architectures (e. g., taking a user action and propagating it across several nested concern boundaries).
  • You cannot trust it to do (1) and (2) together: to implement the data-roundtrip infrastructure if it requires implementing, rather than only navigating, some complicated logic.
• Which is to say: the scope has to be pretty small. Smaller than you'd think from all the hype. No, even smaller than that.
• At which point: do I actually get speed-ups from delegating the implementation to Claude, as opposed to interrogating the chatbot version of Opus 4.5 regarding how it can be implemented, then doing it manually (or with Cursor's sometimes-helpful sometimes-frustrating autocomplete assist?).
  • On one hand, this does compromise learning. I don't develop as precise a sense of how specific tools can be used/built as I would have if I were forced to think through the lower-level details manually. This is potentially bad in the very-long term.
  • On the other hand, this removes a fair amount of mental strain and frustration from me. The alternative to my babbling a feature to Claude may not be "I interrogate Opus 4.5 and write the code manually", it may be "I go watch a movie, because I don't have it in me this late in the day". And, really, would it really be helpful for me to learn the minutia of how this one-off Python package I'll never use again works? (Perhaps!)
• Which is to say...
  • Over the short term, and for short-term projects, this definitely provides a speed-up.
  • Over the medium term, e. g. 4 weeks, or projects of similar scope? Dunno.
  • Over the long term, e. g. months/years, or projects of similar scope? Dunno.
• Regarding bug-fixing:
  • It's often advised to use test-driven development with LLMs, because it keeps them honest and is so easy. Just ask the LLM to write up the tests beforehand, glance over them to make sure they're solid, and off you go! This is not good advice, in my experience. Waste of time and tokens. LLMs don't write good tests, fixing those up would take longer than either doing the thing manually or having the LLM generate the feature raw then working out the kinks post-factum, etc.
    • (You can try asking the LLM to code up some domain-specific interface for you instead, so that you get a firehose of data on what you're coding and on any bugs that transpire... But honestly, I wish someone would just vibe-code some non-terrible general-purpose call-graph debugging tools instead. Maybe I should try doing it myself.)
  • If the LLM-generated code has bugs in it, can you make them go away by describing the bug to the LLM and asking it to fix it? Sometimes. Often no, and often it takes longer than going looking for the bug directly.
  • A broad issue with LLMs is that their activity in the codebase is always... "entropy-increasing". Almost every time they take action, they nudge some stuff out of place, add unnecessary features/functions, unnecessary comments, unnecessary guardrails for states the program can't enter, or they implement something then change their mind but fail to clean the unused thing back up, etc. This happens even if you ask the LLM to go tighten things up/see if the codebase can be streamlined. This also means that if an LLM doesn't one-shot the bug, if it tries A then B then C, that's going to leave a trail of failed bug-fixes which may or may not be screwing something else up.
    • So if you want to build something nontrivially complex which can serve as a foundation for further complexity, you have to periodically go in and manually straighten things out.
    • (Sidenote: I did not know about code-simplifier prior to reading the OP. I'll try it, I guess.)
• The LLM merely lacking context is a significant factor sometimes:
  • It's not really possible to convey your full intent to it, both because language is too low-bandwidth and because of context-window limitations.
  • It's often unable to bug-fix efficiently just because it can't see what's going wrong. (I guess I can screenshot things, but I know Claudes don't have great vision.)
• What's up with people who run twelve hundred simultaneous Claude sessions? Are they all LARPers?
  • Maybe.
  • Maybe not? After I've chewed up the project into bite-sized pieces, I do end up having to wait a while on Claude while it executes. If I had a better handle on what Claude can handle, and how to babysit it, and had the mental habit of breaking projects up into parallelizable instead of sequential implementation steps... If I've spent more than a week figuring this out... I can maybe see it?
• Is it possible to "vibe-code"?
  • Not really, no.
  • The moment you lose track of what's going on and start relying on trust and "vibes", the moment you make the wrong call about the scope, it's over for you, Claude runs off and spills a bunch of spaghetti all over the codebase. If that happens, you shouldn't even try to fix it, just roll it back, chew the task up into finer pieces, and redo.
  • More broadly, if you don't have the technical mindset to break the project down into small pieces that could be implemented and tested independently, and you can't quickly get up to speed if you run into some novel technical/architectural problem that requires a thought-out technical solution, I don't think you're getting anywhere, no.
    • (You can attain the needed competence more rapidly than ever before via asking Opus 4.5 as well, but, like, you do need to attain that competence, at which point are you really "vibe-coding"?)
• What's up with the people who claim they can vibe-code, that their workflows look like this picture where they sit back and relax and Claude does everything? My guesses in increasing order of respectability:
  • They're doing fake work, or are straight-up just attention-seeking AI influencers.
  • They're deluding themselves and their projects are going to crash and burn.
  • They're not doing anything actually nontrivially complicated, for certain values of "nontrivially complicated". See the footnote in my previous comment.
    • See also this as a concrete example. Sure, it sounds impressive when the guy frames it as impressive, but I'm also getting the sense it's just plugging a bunch of APIs into each other.
  • They're drastically underestimating how much metis/know-how goes into their workflows, how much they really babysit LLMs, and how very far LLMs are from real autonomy.
• Is Claude Code on track to replace software engineers? Ha! No. It does not have a "4-hour time horizon". 20 minutes, maybe. Whatever METR's graph is measuring, I currently don't feel very daunted by the prospect of it continuing to go exponential. I may eat my words here, sure.
• Is Rust a perfect language for agents? Haha, good one. I tried getting Opus to write Rust code (on its own suggestion), it was not good at it, like 10x slower and 3x buggier. I went back to Python. I'll try checking later if it's any good at translating from Python to Rust, once the Python code already exists... (And for fairness' sake, I suppose I was using Opus, and he was perhaps talking about Codex.)
• Is it fun to use Claude Code to code? No. Not for me. Keeping the project from degenerating into a non-functional mess is a constant uphill battle. I find it mentally taxing, in a different way to how writing code manually is mentally taxing.

To sum my current view up: Seems useful, but hard to use. You'll have to fight it/the decay it spreads in its wake every step of the way, and making a misstep will give your codebase lethal cancer.

We'll see how I feel about it in one more week, I suppose.

Seconded. Went from a skeptical "big if true" at the post title to rolling my eyes once I saw "iruletheworldmo".

For reference, check out this leak by that guy from February 2025:

I guess it's kind of entertainingly written.

It seems like, yes, he is saying that wealth levels get locked in by early investment choices, and then that it is ‘hard to justify’ high levels of ‘inequality’ and that even if you can make 10 million a year in real income in the post-abundance future Larry Page’s heirs owning galaxies is not okay.

I say, actually, yes that’s perfectly okay, provided there is stable political economy and we’ve solved the other concerns so you can enjoy that 10 million a year in peace.

I dunno about that. I think it is not okay for directionally the same reasons it wouldn't be okay if we got an "infinitesimally aligned" paperclip maximizer who leaves the Solar System alone but paperclips the rest of the universe: astronomical waste.

Like, suppose 99% of the universe ends up split between twenty people, with them using it as they please, in ways that don't generate much happiness for others. Arguably it's not going to be that bad even in the "tech-oligarch capture" future (because Dario Amodei has made a pledge to donate 10% of his earnings or whatever^[1]), but let's use that to examine our intuitions.

One way to look at it is: this means the rest of civilization will only end up 1% as big as it could be. This argument may or may not feel motivating to you; I know "more people is better" is not a very visceral-feeling intuition.

Another way to look at it is: this means all the other people will only end up with 1% of the lifespan they could have had. Like, in the very long term, post-scarcity isn't real, the universe's resources are finite (as far as we currently know), and physical entities need to continuously consume those to keep living. If 99% of resources are captured by people who don't care to share them, everyone else will end up succumbing to the heat death much faster than in the counterfactual.

This is isomorphic to "the rich have their soldiers take all timber and leave the poor to freeze to death in the winter". The only reason it doesn't feel the same way is because it's hard to wrap your head around large numbers: surely you'd be okay with only living for 100 billion years, instead of 10 trillion years, right? In the here and now, both numbers just round up to "effectively forever". But no, once you actually get to the point of you and all your loved ones dying of negentropic starvation 100 billion years in, it would feel just as infuriatingly unfair.

I understand the tactical moves of "we have to pretend it's okay if the currently-powerful capture most of the value of the universe, so that they're more amicable to listening to our arguments for AI safety and don't get so scared of taxes they accelerate AI even further" and "we have to shut down the discussion of 'but which monkey gets the banana?' at every turn because it competes with the 'the banana is poisoned' messaging". But if we're keeping to Simulacrum Level 1, no, I do not in fact believe it's okay.

I also don't necessarily agree that those moves are pragmatically good. It's mostly pointless to keep talking to AI-industry insiders; if we're doing any rhetoric, it should focus on "outsiders". And if it is in fact true that the current default trajectory of worlds in which the AGI labs' plans succeed may lead to something like the above astronomical-waste scenarios, making those arguments to the general public is potentially a high-impact move. "Ban the AGI research because otherwise the rich will take all the stuff" is a much more memetically viral message than "ban the AGI because Terminator".

(To be clear, I'm not arguing we should join various coalitions making false arguments to that end, e. g. the datacenter water thing. But if there are true arguments of that form, as I believe there are...)

1. ^{^}

   I do not trust that guy to keep such non-binding promises, by the way. His track record isn't good, what with "Anthropic won't advance the AI frontier".

I note that the wording in the more direct sources (rather than paraphrases) is "preventive war" and "bomb them", which doesn't actually strictly imply preventive nuclear bombings. It's plausible that "bomb them" and "war with the USSR" could only mean "nuclear war" in-context... But it'd also be really funny if this is another "Eliezer advocates nuking foreign datacenters" situation.