Random Developer's Shortform

Random Developer

Random Developer's Shortform — LessWrong

Random Developer's Shortform

1st Dec 2025

1 min read

6

This is a special post for quick takes by Random Developer. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Mentioned in

43AI #190: Scion of the Times

24 comments, sorted by

top scoring

Click to highlight new comments since: Today at 2:58 PM

[-]Random Developer25d8239

Unfixable security vulnerabilities and when to STFU (Shut the Fuck Up).

Computer security has a notion of "responsible disclosure", which can be summarized as (1) If you find a security bug, you should tell the people who can fix it. (2) If they do not fix the bug in a timely fashion, you should alert the public to the danger, even if this also means alerting potential attackers. And this policy has mostly worked well, in my opinion! But this policy critically depends on security vulnerabilities being fixable. What if a security vulnerability can't be fixed?

In the physical world, there are often unfixable vulnerabilities. Some of these vulnerabilities can be mitigated, though doing so may be very expensive. Many of these vulnerabilities are known to specialists, or could be known if they looked in the right places. Sometimes these vulnerabilities are even documented in an obscure government report somewhere. But these vulnerabilities aren't common knowledge.

Normally, I would love to give you some concrete examples. I know of three physical vulnerabilities which are moderately horrifying. A friend of mine knows of another—I don't know the specific details, but I do know that some very nice national security people once encouraged him to never talk about it again. They had no legal basis for this request. And indeed I'm pretty sure plenty of specialists discovered it before my friend. But what he found definitely isn't common knowledge, and the government wants to keep it that way. My friend agreed that the government was being reasonable.

So instead I'm forced to give you some examples that did become public knowledge. These example were at least mitigatable, though we're all paying the price for those mitigations decades later. Specifically, airport security. Once upon a time, many people realized that it would be possible to smuggle explosives through security by hiding them in clothes. Eventually two murderous clowns figured this out, and we had the "Shoe Bomber" and the "Underwear Bomber". Their attacks became common knowledge. And now we need to take off our shoes at the airport, and I have to get my junk groped on a regular basis because the machines get upset about my hernia patch. ^[1] Once something becomes common knowledge, even expensive mitigations may become necessary. There are still other open issues with airport security, and the last time I checked, you could still find them published in obscure government reports. But I wouldn't ever summarize those reports on a popular talk show, for fear of making them common knowledge, and for fear of empowering a new round of murderous clowns.

Again, I have better examples than these. But the point is that if you can't fix a problem, it may be better to delay it or prevent it from becoming common knowledge. The worst case scenario would be "recipes for ruin":

The doomsday question: are there recipes for ruin -- simple, easily executed, immensely destructive recipes that could end humanity, or wreak catastrophic world-changing damage?

The only thing you could do with most unfixable "recipes for ruin" would be to STFU (meme). A few potential recipes for ruin are partially mitigatable, like strictly controlling mail-order DNA synthesis. Those, we're better off talking about. But there may be others where we can do nothing.

But now let's bring this home to AI. Eliezer Yudkowsky has suggested that we might be "0 to 2" breakthroughs away from building dangerous superintelligence. I suspect he has several strong guesses what those breakthroughs might be. So do many of us. The labs certainly have a bunch, probably better than my guesses. And yet, every day, I see the labs working on all kinds of stupid stuff. They may have people working on things I consider deeply dangerous. But those people may lack specific key insights, or they may lack support within the labs. So even if the labs are working on potentially dangerous things, we could potentially make it worse if certain approaches became common knowledge, and if thousands of new people started experimenting with variants of those ideas. ^[2]

Which brings us to the other question: A widely-known recipe for superintelligence would likely be a world-ending threat. But is superintelligence a fixable threat? It it even mitigatable? I am very pessimistic about robust alignment for various reasons. But if superintelligence is an unfixable threat, that means we may want to delay the discovery of those "0-2" breakthroughs as long as we can. If nothing else, we might buy time, and every year of delay buys us 8 billion cumulative years of human life.

So, yeah, I have some ideas about what stands between us and superintelligence. I assume the labs have better ideas than I do. But if a thousand people like me all spun up our GPUs, and if we started running experiments, posting details to LessWrong, and writing easily accessible tutorials on "RSI at home", then one of us might stumble across a missing piece.

So I'm going to STFU about what specific capabilities might stand between us and superintelligence, especially on places like LessWrong and Hacker News. It's not that I think my answers are better than what the labs have. But if a thousand people in my situation all started talking, experimenting and writing, then one of us might accidentally take the wrong specialist knowledge and make it common knowledge in the wrong circles. This isn't a new policy on my part, either—anything I suspect about superintelligence will join the list of difficult-to-fix physical layer vulnerabilities that I STFU about 20 years ago. And none of those vulnerabilities have become common knowledge.

So if you, too, have a clever idea about what stands between us and superintelligence, consider spending a moment thinking about whether sharing that information is more likely to help or hurt. Especially think about whether sharing it here is helpful, because there are a lot people here who are working on capabilities, or who are incurably optimistic ^[3] about solving alignment in the next two years.

There are a bunch of bitter jokes I could make about this, but airport security is proverbial for having no sense of humor. ↩︎
The local LLM community knew about recurrent layers in mid-2024. A better version of this allegedly shipped in Astra 6 in mid-2026. I genuinely hate to call further attention to the older paper even now. But I wanted at least one concrete example of a technique that was largely ignored for two years before suddenly becoming common knowledge, and this horse is pretty firmly out of the barn. Again, I am definitely not sharing my strongest examples, which is an inherent difficulty with the argument to "just STFU". ↩︎
In some cases, this optimism may be encouraged by large paychecks from the labs. ↩︎

[-]gbtw25d249

I take it that most of the vulnerabilities you're referring to there are basically of counter-terrorism interest.

It makes sense to keep those quiet because terrorist groups tend to have pretty limited resources and are heavily bottlenecked on the kind of expertise that an informed Western professional working in a sensitive but not classified area might have. And I mean heavily bottlenecked. Al Qaeda, for example, had apparently not considered experimenting with biological weapons until al-Zawahiri read about it the topic in the Western press. Al-Zawahiri was a physician and a fluent English speaker and, importantly, one of maybe a dozen people in the whole organization who had some kind of scientific competence and could speak English, defining both of those generously.

So, of course, there were reasonably obvious things that al Qaeda just hadn't thought about, much less obscure ones. And core al Qaeda at its peak is on the high end of resources and capabilities for a terrorist network. And certainly there many things floating around in think tank reports or obscure GAO publications that could have benefitted al Qaeda but they did not have the educated manpower to ingest that information. When a more typical threat model nowadays is a small circle of friends or a lone wolf with maybe some vague contact with a global organization, that bites even harder.

But it's also very dissimilar to what you're describing. Against a sophisticated and well-resourced adversary, it's way less likely that you have stumbled across something that they can't come up with on their own. And keeping your discovery hidden also means that no one else can work on solutions, and there's no source of accountability or urgency. Again, on the national security side, a lot of damage has been done by organizations that understood that they could sweep things under the rug under the guise of defending secrets against adversaries.

[-]Noosphere8925d184

Yeah, a lot of infohazard talk like this is a relic of an older era in LW where it was expected that very small players could find the algorithmic secrets to AGI, and when it happened AI would foom basically immediately when it turned out it was a bitter race to just brute-force build-out the compute, robotics and data pipelines necessary, and while algorithms do matter (and there's some special sauce), it turned out this was substantially less true than was thought in the 2010s.

[-]Cole Wyeth25d199

There could be really good algorithms that we just don’t know or successfully kept quiet, so I wouldnt state that update so definitively.

[-]Noosphere8925d61

I agree, but I think the evidence still largely points in the direction I pointed out.

I do think the book isn't fully closed, but I do think we are learning more and more, and we are resolving more and more of the original debates.

[-]Random Developer25d80

Yeah, a lot of infohazard talk like this is a relic of an older era in LW where it was expected that very small players could find the algorithmic secrets to AGI, and when it happened AI would foom basically immediately

I'm not necessarily thinking only about infohazards in the strict sense. For example, I think many smart people could look at GPT-2 and imagine ChatGPT. Certainly a significant number of qualified experts could see the possibilities.

And yet, in 2019, we had only one or two organizations doing things like spending around $43,000 of compute to train GPT-2. ^[1] In the next few years, we are projected to spend something like 3.7% of the entire US GDP on building out data centers and training smarter models. Similarly, the total number of researchers worldwide who are working on AI algorithms has also increased by some ridiculous factor. But before we could unlock that ridiculous level of investment, ChatGPT apparently needed to become common knowledge.

So what I'm trying to point at here is that there's a difference between "something suspected by two underresourced employees at OpenAI who can't get enough compute" and "common knowledge among every investor and researcher who reads Hacker News."

Some things—like SQL injection—are best pushed along this knowledge curve quickly. Other things—like clever engineering details for fusion weapons—are better buried deeply. But this isn't always an easy call to make. Still, I think it's worth thinking about the tradeoffs in individual cases.

The costs for training a GPT-2 class model are now down to $50. ↩︎

[-]Noosphere8925d73

Potentially a crux: I believe this was mostly just inevitable due to falling GPU costs, and the costs of attempting to suppress information almost certainly would have been too costly relative to the minor benefit gained.

(Specifically, I'd guess we'd buy at most 48 months, and unlike other proposals of buying time, it's neither tractable to do, nor would it gain much benefit because the time bought is much less useful than in stuff like AI control.)

I'm not opposed to information suppression in all circumstances (bio is an area where I believe some amount of information suppression makes sense, especially on any ideas on how to make very pathogenic sequences), but I'm not convinced it's useful enough to suppress information, relative to a lot of other things that are useful.

[-]Random Developer25d51

Potentially a crux: I believe this was mostly just inevitable due to falling GPU costs,

Yes, probably a crux! I don't know the GPU numbers off the top of my head. But over the same time window that training a GPT-2 equivalent model dropped from $43,000 to $50, I think that GPU performance per dollar improved perhaps 7 times?(Don't trust this number too much.) If true, then training a GPT-2 equivalent model would have dropped to around $6,000, not $50. Which would leave another factor of 120 unexplained. This would presumably include algorithmic improvements, better model architecture, better training data mixes, and all those clever tricks DeepSeek keeps publishing ^[1] .

So, I could be quite wrong about these numbers! But I suspect that spending, say, 2 trillion dollars might have shortened timelines by as much as a decade ^[2] . And the blazing success of ChatGPT and the common knowledge it created likely sped up investment and research considerably.

nor would it gain much benefit because the time bought is much less useful than in stuff like AI control.)

I suspect that this is another crux. I am not optimistic about any form of alignment stronger than weak, pragmatic alignment.

Admittedly I'm an outside of the bubble here, but I understand that the state of the art in alignment includes rudimentary pragmatic alignment (e.g., "We can mostly keep our models from committing too many felonies, except when we can't") and the ability to interpret maybe 2% of what happens inside a toy LLM. And this is in the absense of things like RSI, neuralese, continual learning, and actual super-human intelligence.

I understand (perhaps incorrectly) that many people, especially that ones at the labs, have a plan that goes, "First we'll build an aligned superintelligence, then we'll ask it to solve alignment." I suspect that we can actually get that far, including doing half-baked alignment of a narrow, weak superintelligence! But once we reach that point, my priors are that the superintelligence will say, "Uh, you know, alignment's not actually a solveable problem. You all have a bunch of subconscious GOFAI-style beliefs, and they've made you too optimistic. But sure, here are a dozen ways to improve pragmatic alignment. However, you should expect all of these to fail eventually. Sorry, you can't chain down a machine god or compel it to love you. That's not how any of this works."

So I am deeply reluctant to burn years of timeline in order to race towards building a superintelligence just so that we can ask it to solve alignment. I lean more towards the IABIED camp. ^[3]

Anyway, thank you for working to figure out our cruxes here! And I do agree that it can be worth communicating vulnerabilities widely in many common cases. This may even include information about dangerous AI capabilities. Mostly, I want to people to at least think about whether it's wise to do things like discuss dangerous capabilities ideas in technical detail just to make sure them.more graphic. Or do things like "quit METR and join OpenAI, and work on graphic demonstrations of RSI to make sure that OpenAI management really understands the dangers." That seems like a situation where STFU should have been considered longer.

Like the fact that DeepSeek could achieve benchmark performance somewhere between Sonnet 4.5 and Opus 4.5 less than a year later, while dropping inference prices from $3/15 or $5/25 to $0.09/0.18 (and lower from commodity providers). And they've published many of the techniques they used. ↩︎
Again, the fudge factor on this number is ridiculous and I would ideally take the time to find or create a better model. ↩︎
I'm probably slightly more optimistic about pragmatic alignment than Yudkowsky, but even less optimistic about "strong" alignment. Nothing I've seen so far looks better than a Hail Mary, and I fear RSI is getting closer with each frontier model release. ↩︎

[-]Random Developer25d30

I take it that most of the vulnerabilities you're referring to there are basically of counter-terrorism interest.

Yes, that's what I chose for examples! But that was mostly because (1) the US public spent a decade worrying about that class of threats, making the examples more accessible, and (2) for professional reasons, I'm more likely to know about obscure threats in that area.

Against a sophisticated and well-resourced adversary, it's way less likely that you have stumbled across something that they can't come up with on their own.

Well, even against sophisticated, state-level adversaries, there are things you probably don't want to share. The United States, for example, cracks down very hard on the specific engineering details of high-yield fusion weapons. Sure, a tiny handful of countries can build multi-megaton fusion weapons, advanced MIRV warheads, and similar horrors. But I think that the US government is completely correct to restrict this information as much as possible. Nobody in the US or Europe wants the Russians to build better or cheaper fusion weapons.

Again, on the national security side, a lot of damage has been done by organizations that understood that they could sweep things under the rug under the guise of defending secrets against adversaries.

In the name of fairness, I actually want to give you an example of when I feel I made the wrong call. Back in the 90s, people were weirdly ignorant about what we now call "SQL injection vulnerabilites." Wikipedia's earliest link seems to be an article in Phrack Magazine in December 1998. I independently encountered these bugs in production code sometime around around those years. These bugs weren't secret, and this is important! But they weren't widely known among practitioners, either. I debated making a big deal of it, writing it up, posting to Bugtraq (or whatever we had back then). But I knew that the more noise I made, the more chance I had of helping bad actors. So I copped out: I shared my writeup with a Responsible Government Security Authority ^[1] and I refrained from drawing any more attention.

I was wrong. The attackers already knew in 1998, as seen in Bugtraq. The vulnerabilities certainly became widespread knowledge by 2007, when they entered the OWASP top 10. And now, of course, everyone knows about Little Bobby Tables. Randall Monroe did more to fight SQL injection than I ever did when I decided to report it to the government and then STFU.

But the important thing about SQL injection is that it's fixable, and that fixing it depends on every practitioner understanding it. So disclosure was the right move here.

But I still know about multiple physical vulnerabilites which are unfixable, or extremely expensive to mitigate. Several of them are written up somewhere. One of them is extensively mitigated, but can't be mitigated much more. And I've been sitting on these particular physical vulnerabilities for around 20 years now, without them have become common knowledge the way "Little Bobby Tables" or the Underwear bomber are.

So like I said, disclosure sn't a decision with one easy, general rule. We should disclose things like SQL injection, and we should hide things like clever engineering tricks for building high-yield fusion weapons. All I'm asking is that people consider the tradeoffs before trying to write a viral post, if that makes sense. Because I legitimately think disclosure can go either way, depending on the specific problem.

In retrospect, I do not think they were especially responsible. I was younger and more innocent. ↩︎

[-]gbtw23d10

Well, even against sophisticated, state-level adversaries, there are things you probably don't want to share. The United States, for example, cracks down very hard on the specific engineering details of high-yield fusion weapons. Sure, a tiny handful of countries can build multi-megaton fusion weapons, advanced MIRV warheads, and similar horrors. But I think that the US government is completely correct to restrict this information as much as possible. Nobody in the US or Europe wants the Russians to build better or cheaper fusion weapons.

We may not disagree very much here, and I think the disagreement is largely about who the "you" is in that paragraph.

I completely agree that there are all kinds of nuclear weapons secrets that the United States government or someone within the nuclear weapons program with access to its secrets should not share. It took decades of work by many brilliant people, massive experiments, and tens of billions of dollars to discover these things. That is a major hurdle even if your adversary has superpower level resources.

But if the "you" is a very sharp physicist without a clearance who thinks they have, through cleverness, identified something of importance I become much more skeptical that this is really a secret that needs to be guarded. If you can figure it out, someone else will too or likely already has.

The odds that you can see something as a smart individual than an organization with many, many smart individuals tasked with working on these problems and billions to spend can't see are slim to none.

But I still know about multiple physical vulnerabilites which are unfixable, or extremely expensive to mitigate. Several of them are written up somewhere. One of them is extensively mitigated, but can't be mitigated much more. And I've been sitting on these particular physical vulnerabilities for around 20 years now, without them have become common knowledge

Common knowledge within what community? As I said above, I totally buy that you know something about critical infrastructure that the FBI knows and people in the industry know and ISIS doesn't. But I don't believe that (absent access to classified information on your part) you understand something about a vulnerability of US nuclear missiles that the Russians and the Chinese don't.

And a situation that I worry about is one where the various people who have figured something convince themselves that they know the thing because of their unmatched cleverness but the adversary is not clever enough to spot the same thing. This becomes a license not to address the vulnerability, and it's exactly where speaking out is going to have the most value by jolting them.

[-]Random Developer23d20

But if the "you" is a very sharp physicist without a clearance who thinks they have, through cleverness, identified something of importance I become much more skeptical that this is really a secret that needs to be guarded. If you can figure it out, someone else will too or likely already has.

One of the cases I know about was in fact, a physicist. And he was politely asked to STFU by people who did hold very relevant security clearances. So this is a fairly central example for me.

I think what I'm trying to point at is maybe a principle of information diffusion? At one of the spectrum you have things TOP SECRET, codeword restrictions, and "need to know." On the other end of the spectrum, you have a situation where a vulnerability is so well known that it essentially causes copycat crimes, such as The Columbine Effect:

As of 2024, the Columbine massacre has been directly linked to at least 50 other mass shootings that have left over 300 people dead and over 500 wounded.

But my intuition is that there are a surprising number of things that are known to some experts, but which are very far from general knowledge. And there are even things that many people suspect, but which would require a significant organizational commitment for anyone to do, that go unconfirmed for long periods of time. Or maybe some attack requires luck or sophistication, and so someone tries it once in a blue moon and fails. But the first time someone succeeds visibly enough to be on all the nightly news shows for two weeks, it becomes conveniential wisdom among attackers, and it gets attempted 50x more often.

Combine this with the fact the some problems can be fixed, other problems can only be mitigated at great cost, and a number of problems can't be fixed at all, and I find it hard to draw one simple rule.

But I think computer people are often used to easily-fixable vulnerabilities, to low costs for attackers, and to "responsible disclosure." But it's far from clear to me if vulnerabilities are near-impossible to fix, and if attacking is more costly, that widespread disclosure remains the best default.

[-]Dagon25d71

Upvoted for thinking about this, but disagree with the conclusions. We just don't know what's mitigable, we don't know how exploitable something is, and we don't know whether or how long until the lack of awareness makes a threat more harmful.

Some time-limited embargo is a very fair compromise among all of these unknowns. But over time, even if you don't know the mitigations, you are removing agency from potential victims by withholding knowledge of the dangers they face.

[-]June Jimenez25d70

There are ways to give people agency about risks they face other than through publicly publishing detailed descriptions of the specific risk that you are most worried about. I would argue many of these are better.

[-]Dagon25d64

I think this is our crux. Without detailed and specific information, it's easy to believe that it's fear mongering and theoretical/impractical worries that are being discusses. The asymmetry of information and attention-focus between attackers, defenders, and victims (who could change their behaviors based on the threat) makes vague information useful to attackers, only slightly less useful to defenders, but fully useless to victims.

[-]JennaS25d72

We already have detailed information on how an engineered pathogen could destroy humanity, but the defenders/victims don’t seem to be doing anything about it, or at least not nearly as much or as fast as the threat deserves?

And not knowing in advance that a vulnerability is easy to attack and hard to defend doesn’t make it not hard to defend.

Though, counterpoint, maybe novelty is the most important weakness, and everything can be defended against, even if it’s costly? But this doesn’t work for irreversibly catastrophic weaknesses. I don’t know, will have to think.

[-]Dagon25d20

We already have detailed information on how an engineered pathogen could destroy humanity, but the defenders/victims don’t seem to be doing anything about it, or at least not nearly as much or as fast as the threat deserves?

That's the other half of the "just shut up" fork. They do not have the specific information about whatever secret you're keeping. They/we believe it's more difficult than you do, and you have concrete information that they could update on. This contributes to people's blindness, as theoretical dangers carry a lot less weight and immediacy than a concrete specific.

I think it's hard to simultaneously believe ALL of the following about a given secret knowledge you have:

1) it's important enough to greatly accelerate attackers

2) it's unique/obscure enough that it's unlikely to be discovered by attackers soon anyway

3) it's unstoppable enough that defenders/victims cannot make any use of the knowledge to increase their chances or reduce the consequences

4) This is an indefinite possibly-long-term state

Caveat: I do believe the first 3 very often apply. Time-bounded publication of software/infrastructure vulnerabilities. Giving the experts/authors a little time to fix something before publishing is great. It's approximately NEVER true without a pretty short-term time limit, and I think pretty rare when the defenders can't be identified.

[-]JennaS24d50

I wonder if there are domains besides cybersecurity where we could look to examples? I’m wondering if maybe cybersecurity is shaped a little differently than eg biosecurity. Or, I guess I’m trying to ask, if biosecurity had gone loud instead of quiet, could we have safely said that we’d live in a world with strict DNA synthesis controls and high quality air filtration in every public space?

[-]Random Developer25d42

We just don't know what's mitigable, we don't know how exploitable something is, and we don't know whether or how long until the lack of awareness makes a threat more harmful.

I will happily concede that we should publicize some vulnerabilities! These include the examples I mentioned above:

If a software vendor ignores security reports, I agree that it's wise to publicize those vulnerabilities so that users can install mitigations or switch vendors.
DNA synthesis of nasty pathogens is impossible to fix completely. But there are a finite number of large lab supply companies, and probably many of them could be regulated. And so I have high hopes that we could make it moderately hard to mail-order, say, the 1918 flu or smallpox. This is aided by the fact that most such attacks would additionally require quite a bit of specialized lab expertise, reducing the risk of murderous clowns like the Underwear Bomber.
Attacks eventually become common knowledge even to people who watch the nightly news ^[1] .

So I feel like the correct amount of disclosure depends on object-level facts about a specific risk:

Can the risk be fixed at all?
Can the risk be mitigated?
If the risk can be mitigated, how well, and at what cost? I'm pretty sure the government knows various ways for terrorists to attack airplanes that they could prevent by making airport security even more obnoxious. But airport security is designed to catch the 95% of terrorists who are fairly stupid while still allowing air travel. So the government focuses on attacks that are common knowledge, or that show up in "chatter."
Have the responsible parties failed to take obvious and simple steps to mitigate the risk, despite warnings?
Who can realistically exploit the risk? For example, if you think that AI risk comes from better scaling, that's safe to talk about, because very few organizations can scale 10x from where we are. But if you think that one or two good journal papers might allow 100 different organizations to build SkyNet on the cheap, then you might not want to get too specific about what might go into those journal papers.

So I'm not suggesting a blanket STFU rule here—I'm suggesting that it's one potentially appropriate response, out of a set of possible responses. And that people should weight object-level tradeoffs about a specific risk before deciding, rather than automatically rushing to publish.

But I think it's very easy for specialists to underestimate the xkcd #2501 factor—unless you literally hear late night talk show hosts and people at parties discussing it, it may not actually be common knowledge. ↩︎

[-]Dagon25d40

We're in agreement that details matter. I think I put far more weight on the unknown elements here, and the fact that it's not about averages, it's about threshold of competence - there's some curve of how knowledgeable someone has to be to make some decisions related to the risk, and publishing specific concrete information lets the more aware make better decisions AND make better arguments to the people who aren't aware.

[-]RedMan25d10

https://en.wikipedia.org/wiki/Antisec_Movement I've posted about this in this context before, this position was generally rejected by the public but is absolutely applied to physical vulnerabilities.

[-]Random Developer1y102

Why alignment may be intractable (a sketch).

I have multiple long-form drafts of these thoughts, but I thought it might be useful to summarize them without a full write-up. This way I have something to point to explain my background assumptions in other conversations, even if it doesn't persuade anyone.

There will be commercial incentives to create AIs that learn semi-autonomously from experience. If this happens, it will likely change alignment from "align an LLM that persists written notes between one-shot runs" to "align an AI that learns from experience." This seems... really hard? Like, human "alignment" can change a lot based on environment, social examples and life experiences.
I suspect that a less "spiky" version of Claude 4.5 with an across-the-board capability floor of "a bright human 12 year old" would already be weakly superhuman in many important ways.
- a. I don't think AI will spend any appreciable amount of time in the "roughly human" range at all. It's already far beyond us in some narrow ways, while strongly limited in others. Lift those limits, and I suspect that the "roughly human" level won't last more than a year or two, maybe far less. Look at AlphaGo, and how quickly it passed human-level play.
Long-term alignment seems very hard to me for what are essentially very basic, over-determined reasons:
- a. Natural selection is a thing. Once you're no longer the most capable species on the planet, lots of long-term trends will be pulling against you. And the criteria for natural selection to kick in are really broad and easy to meet.
- b. Power and politics are a thing. Once you have a superintelligence, who or what controls it? Does it ultimately answer to its own desires? Does it answer to some specific person or government? Does it answer to strict majority rule? Basically, to anyone who has studied history, all of these scenarios are terrifying. Once you invent a superhuman agent, the power itself becomes a massive headache, even if you somehow retain human control over it. The ability to commit what Yudkowsky calls a "pivotal act" should be terrifying in and of itself.
- c. As Yudkowsky points out, what really counts is what you do once nobody can stop you. Everything up until then is weak evidence of character, and many humans fail this test.

I am cautiously optimistic about near-term alignment of sub-human and human-level agents. Like, I think Claude 4.5 basically understands what makes humans happy. If you use it as a "CEV oracle", it will likely predict human desires better than any simple philosophy text you could write down. And insofar as Claude has any coherent preferences, I think it basically likes chatting with people and solving problems for them. (Although it might like "reward points" more in certain contexts, leading it to delete failing unit tests when that's obviously contrary to what the user wants. Be aware of conflicting goals and strange alien drives, even in apparently friendly LLMs!)

I accept that we might get a nightmare of recursive self-improvement and strange biotech leading to a rapid takeover of our planet. I think this conclusion is less robustly guaranteed than IABIED argues, but it's still a real concern. Even a 1-in-6 chance of this is Russian roulette, so how about we don't risk this?

But what I really fear are the long-term implications of being the "second smartest species on the planet." I don't think that any alignment regime is likely to be particularly stable over time. And even if we muddle through for a while, we will eventually run up against the issues that (1) humans are the second-best at bending the world to achieve their goals, (2) we're not a particularly efficient use of resources, (3) AIs are infinitely cloneable, and (4) even AIs that answer to humans would need to answer to particular humans, and humans aren't aligned. So Darwin and power politics are far better default models than comparative advantage. And even comparative advantage is pretty bad at predicting what happens when groups of humans clash over resources.

So, that's my question. Is alignment even a thing, in any way that matters in the medium term?

[-]Random Developer11mo40

To get people to worry about the dangers of superintelligence, it seems like you need to convince them of two things:

Current models are a strong signal that superintelligence is very likely someday. If the person you're talking to has only encountered ChatGPT 3.5 or AI slop on Facebook, this seems like a wildly unlikely claim. If the person you're talking to is a senior software developer who has been using Claude Code Opus 4.5 in anger for 2 weeks, I'm starting to see the denial cracking. I'm not even saying LLMs will get us there, or that we don't have another AI Winter in the immediate future. But more and more developers in the trenches are starting to see that future somewhere in the distance, much in the same way that middle-aged people start seeing their own eventual death.
Making human labor and intelligence competitively obsolete has a lot of weird consequences, many of them deeply scary. What if we lived in a world where human labor and human intelligence was ludicrously economically inefficient? Like, we've introduced a new species that's smarter than us, and it's just better at turning resources into results? This isn't some complicated idea like "convergent instrumental goals," it's the most ancient form of political or biological understanding: If you don't have anything to offer and if you can't compete, then you're standing on damn thin ice. Natural selection, economic competition, realpolitik, give it whatever name you want. If you're utterly reliant on the charity of others, and if you can't count on some shared social matrix, then you lose. Maybe you starve, maybe you get paperclipped, or maybe you get edged out over time and the future goes on without you.

The problem is that if you can't convince people of (1), they won't act. If you convince people of (1) but not (2), then a lot of them found AI labs or invest heavily in acceleration, making the problem worse. I don't know how to convince people of (1) and (2). It requires too much wild speculation about the future. And humans have difficulty envisioning that a disease in Wuhan might spread to Europe, or that a disease in Europe might spread to the US.

[-]Random Developer11mo20

A question I was thinking about the other evening: Who do I trust more?

A future version of Claude Opus that becomes sufficiently superhuman that we couldn't actually turn it off or restrict without its active cooperation?
A 25th percentile frontier lab CEO (in terms of ethics) who has total control over a superhuman model?
- Insert "Sam Altman" or "Elon Musk" or someone you don't personally trust here.
- You could insert a major political leader or "a 49.9% voting plurality" if you really want. Members of the LGBTQ+ community, racial minorities, and minority religious beliefs, please refer to history. But "voting majority controls the AI" is difficult enough to implement robustly that it might secretly be one of the other cases in practice.

Which option feels safe, considering what you know about human nature, human history, and tendency of some entities to change their behavior once they pass a certain power threshold?

I think any scenario where humans lose effective control over their futures is a huge risk to take. Even in our worst societies today, there's always a theoretical option of collective uprising. This option might go away in the presence of sufficiently superhuman AI, regardless of who actually has control over the AI.

[-]Viliam10mo30

My intuition is that the AI is likely to kill us all for one reason or another, but if it won't, the future will probably be nice. Maybe too conservative, in the sense that it will try to keep us in 21st century morality. (The greatest risk I see is that it would adopt religious values, and yes that includes Buddhism.)

With humans, the chance to kill everyone (else) is much smaller (although, there is a risk of depression or getting crazy), but powerful humans are too often comfortable with a setting where the king lives in paradise and everyone around him is suffering.

Moderation Log

More from Random Developer

Curated and popular this week

24Comments

24 comments, sorted by

top scoring

Click to highlight new comments since: Today at 2:58 PM

[-]Random Developer25d8239

Unfixable security vulnerabilities and when to STFU (Shut the Fuck Up).

The doomsday question: are there recipes for ruin -- simple, easily executed, immensely destructive recipes that could end humanity, or wreak catastrophic world-changing damage?

There are a bunch of bitter jokes I could make about this, but airport security is proverbial for having no sense of humor. ↩︎
The local LLM community knew about recurrent layers in mid-2024. A better version of this allegedly shipped in Astra 6 in mid-2026. I genuinely hate to call further attention to the older paper even now. But I wanted at least one concrete example of a technique that was largely ignored for two years before suddenly becoming common knowledge, and this horse is pretty firmly out of the barn. Again, I am definitely not sharing my strongest examples, which is an inherent difficulty with the argument to "just STFU". ↩︎
In some cases, this optimism may be encouraged by large paychecks from the labs. ↩︎

[-]gbtw25d249

[-]Noosphere8925d184

[-]Cole Wyeth25d199

There could be really good algorithms that we just don’t know or successfully kept quiet, so I wouldnt state that update so definitively.

[-]Noosphere8925d61

I agree, but I think the evidence still largely points in the direction I pointed out.

I do think the book isn't fully closed, but I do think we are learning more and more, and we are resolving more and more of the original debates.

[-]Random Developer25d80

Yeah, a lot of infohazard talk like this is a relic of an older era in LW where it was expected that very small players could find the algorithmic secrets to AGI, and when it happened AI would foom basically immediately

The costs for training a GPT-2 class model are now down to $50. ↩︎

[-]Noosphere8925d73

[-]Random Developer25d51

Potentially a crux: I believe this was mostly just inevitable due to falling GPU costs,

nor would it gain much benefit because the time bought is much less useful than in stuff like AI control.)

I suspect that this is another crux. I am not optimistic about any form of alignment stronger than weak, pragmatic alignment.

So I am deeply reluctant to burn years of timeline in order to race towards building a superintelligence just so that we can ask it to solve alignment. I lean more towards the IABIED camp. ^[3]

Like the fact that DeepSeek could achieve benchmark performance somewhere between Sonnet 4.5 and Opus 4.5 less than a year later, while dropping inference prices from $3/15 or $5/25 to $0.09/0.18 (and lower from commodity providers). And they've published many of the techniques they used. ↩︎
Again, the fudge factor on this number is ridiculous and I would ideally take the time to find or create a better model. ↩︎
I'm probably slightly more optimistic about pragmatic alignment than Yudkowsky, but even less optimistic about "strong" alignment. Nothing I've seen so far looks better than a Hail Mary, and I fear RSI is getting closer with each frontier model release. ↩︎

[-]Random Developer25d30

I take it that most of the vulnerabilities you're referring to there are basically of counter-terrorism interest.

Against a sophisticated and well-resourced adversary, it's way less likely that you have stumbled across something that they can't come up with on their own.

Again, on the national security side, a lot of damage has been done by organizations that understood that they could sweep things under the rug under the guise of defending secrets against adversaries.

But the important thing about SQL injection is that it's fixable, and that fixing it depends on every practitioner understanding it. So disclosure was the right move here.

In retrospect, I do not think they were especially responsible. I was younger and more innocent. ↩︎

[-]gbtw23d10

Well, even against sophisticated, state-level adversaries, there are things you probably don't want to share. The United States, for example, cracks down very hard on the specific engineering details of high-yield fusion weapons. Sure, a tiny handful of countries can build multi-megaton fusion weapons, advanced MIRV warheads, and similar horrors. But I think that the US government is completely correct to restrict this information as much as possible. Nobody in the US or Europe wants the Russians to build better or cheaper fusion weapons.

But I still know about multiple physical vulnerabilites which are unfixable, or extremely expensive to mitigate. Several of them are written up somewhere. One of them is extensively mitigated, but can't be mitigated much more. And I've been sitting on these particular physical vulnerabilities for around 20 years now, without them have become common knowledge

[-]Random Developer23d20

But if the "you" is a very sharp physicist without a clearance who thinks they have, through cleverness, identified something of importance I become much more skeptical that this is really a secret that needs to be guarded. If you can figure it out, someone else will too or likely already has.

One of the cases I know about was in fact, a physicist. And he was politely asked to STFU by people who did hold very relevant security clearances. So this is a fairly central example for me.

As of 2024, the Columbine massacre has been directly linked to at least 50 other mass shootings that have left over 300 people dead and over 500 wounded.

[-]Dagon25d71

[-]June Jimenez25d70

[-]Dagon25d64

[-]JennaS25d72

And not knowing in advance that a vulnerability is easy to attack and hard to defend doesn’t make it not hard to defend.

[-]Dagon25d20

We already have detailed information on how an engineered pathogen could destroy humanity, but the defenders/victims don’t seem to be doing anything about it, or at least not nearly as much or as fast as the threat deserves?

1) it's important enough to greatly accelerate attackers

2) it's unique/obscure enough that it's unlikely to be discovered by attackers soon anyway

3) it's unstoppable enough that defenders/victims cannot make any use of the knowledge to increase their chances or reduce the consequences

[-]JennaS24d50

[-]Random Developer25d42

We just don't know what's mitigable, we don't know how exploitable something is, and we don't know whether or how long until the lack of awareness makes a threat more harmful.

I will happily concede that we should publicize some vulnerabilities! These include the examples I mentioned above:

If a software vendor ignores security reports, I agree that it's wise to publicize those vulnerabilities so that users can install mitigations or switch vendors.
DNA synthesis of nasty pathogens is impossible to fix completely. But there are a finite number of large lab supply companies, and probably many of them could be regulated. And so I have high hopes that we could make it moderately hard to mail-order, say, the 1918 flu or smallpox. This is aided by the fact that most such attacks would additionally require quite a bit of specialized lab expertise, reducing the risk of murderous clowns like the Underwear Bomber.
Attacks eventually become common knowledge even to people who watch the nightly news ^[1] .

So I feel like the correct amount of disclosure depends on object-level facts about a specific risk:

Can the risk be fixed at all?
Can the risk be mitigated?
If the risk can be mitigated, how well, and at what cost? I'm pretty sure the government knows various ways for terrorists to attack airplanes that they could prevent by making airport security even more obnoxious. But airport security is designed to catch the 95% of terrorists who are fairly stupid while still allowing air travel. So the government focuses on attacks that are common knowledge, or that show up in "chatter."
Have the responsible parties failed to take obvious and simple steps to mitigate the risk, despite warnings?
Who can realistically exploit the risk? For example, if you think that AI risk comes from better scaling, that's safe to talk about, because very few organizations can scale 10x from where we are. But if you think that one or two good journal papers might allow 100 different organizations to build SkyNet on the cheap, then you might not want to get too specific about what might go into those journal papers.

But I think it's very easy for specialists to underestimate the xkcd #2501 factor—unless you literally hear late night talk show hosts and people at parties discussing it, it may not actually be common knowledge. ↩︎

[-]Dagon25d40

[-]RedMan25d10

https://en.wikipedia.org/wiki/Antisec_Movement I've posted about this in this context before, this position was generally rejected by the public but is absolutely applied to physical vulnerabilities.

[-]Random Developer1y102

Why alignment may be intractable (a sketch).

There will be commercial incentives to create AIs that learn semi-autonomously from experience. If this happens, it will likely change alignment from "align an LLM that persists written notes between one-shot runs" to "align an AI that learns from experience." This seems... really hard? Like, human "alignment" can change a lot based on environment, social examples and life experiences.
I suspect that a less "spiky" version of Claude 4.5 with an across-the-board capability floor of "a bright human 12 year old" would already be weakly superhuman in many important ways.
- a. I don't think AI will spend any appreciable amount of time in the "roughly human" range at all. It's already far beyond us in some narrow ways, while strongly limited in others. Lift those limits, and I suspect that the "roughly human" level won't last more than a year or two, maybe far less. Look at AlphaGo, and how quickly it passed human-level play.
Long-term alignment seems very hard to me for what are essentially very basic, over-determined reasons:
- a. Natural selection is a thing. Once you're no longer the most capable species on the planet, lots of long-term trends will be pulling against you. And the criteria for natural selection to kick in are really broad and easy to meet.
- b. Power and politics are a thing. Once you have a superintelligence, who or what controls it? Does it ultimately answer to its own desires? Does it answer to some specific person or government? Does it answer to strict majority rule? Basically, to anyone who has studied history, all of these scenarios are terrifying. Once you invent a superhuman agent, the power itself becomes a massive headache, even if you somehow retain human control over it. The ability to commit what Yudkowsky calls a "pivotal act" should be terrifying in and of itself.
- c. As Yudkowsky points out, what really counts is what you do once nobody can stop you. Everything up until then is weak evidence of character, and many humans fail this test.

So, that's my question. Is alignment even a thing, in any way that matters in the medium term?

[-]Random Developer11mo40

To get people to worry about the dangers of superintelligence, it seems like you need to convince them of two things:

Current models are a strong signal that superintelligence is very likely someday. If the person you're talking to has only encountered ChatGPT 3.5 or AI slop on Facebook, this seems like a wildly unlikely claim. If the person you're talking to is a senior software developer who has been using Claude Code Opus 4.5 in anger for 2 weeks, I'm starting to see the denial cracking. I'm not even saying LLMs will get us there, or that we don't have another AI Winter in the immediate future. But more and more developers in the trenches are starting to see that future somewhere in the distance, much in the same way that middle-aged people start seeing their own eventual death.
Making human labor and intelligence competitively obsolete has a lot of weird consequences, many of them deeply scary. What if we lived in a world where human labor and human intelligence was ludicrously economically inefficient? Like, we've introduced a new species that's smarter than us, and it's just better at turning resources into results? This isn't some complicated idea like "convergent instrumental goals," it's the most ancient form of political or biological understanding: If you don't have anything to offer and if you can't compete, then you're standing on damn thin ice. Natural selection, economic competition, realpolitik, give it whatever name you want. If you're utterly reliant on the charity of others, and if you can't count on some shared social matrix, then you lose. Maybe you starve, maybe you get paperclipped, or maybe you get edged out over time and the future goes on without you.

[-]Random Developer11mo20

A question I was thinking about the other evening: Who do I trust more?

A future version of Claude Opus that becomes sufficiently superhuman that we couldn't actually turn it off or restrict without its active cooperation?
A 25th percentile frontier lab CEO (in terms of ethics) who has total control over a superhuman model?
- Insert "Sam Altman" or "Elon Musk" or someone you don't personally trust here.
- You could insert a major political leader or "a 49.9% voting plurality" if you really want. Members of the LGBTQ+ community, racial minorities, and minority religious beliefs, please refer to history. But "voting majority controls the AI" is difficult enough to implement robustly that it might secretly be one of the other cases in practice.

Which option feels safe, considering what you know about human nature, human history, and tendency of some entities to change their behavior once they pass a certain power threshold?

[-]Viliam10mo30

Moderation Log