Good article. There especially needs to be more focus on the social model. I am an optimist, however I think ~half of how things could go wrong with AI is with alignment technically solved. The more you expect a slow takeoff or non-fatal overhang, the more this is important.

Yes, agreed. And: the inevitable dual-use nature of alignment research shouldn't be used as a reason or excuse to not do alignment research. Remaining pure by not contributing to AGI is going to be cold comfort if it arrives without good alignment plans.

So there is a huge challenge: we need to do alignment research that contributes more to alignment than to capabilities, within the current social arrangement. It's very tough to guess which this is. Optimistic individuals will be biased to do too much; cautious individuals will tend to do too little.

One possible direction is to put more effort toward AGI alignment strategy. My perception is that we have far less than would be optimal, and that this is the standard nature of science. People are more apt to work on object level projects rather than strategize about which will best advance the field's goals. We have much more than average in AGI safety, but the large disagreements among well-informed and conscientious thinkers indicates that we could still use more. And it seems like such strategic thinking is necessary to decide which object-level work is will probably advance alignment faster than capabilities.

AI that obeys the intention of a human user can be asked to help build unsafe AGI, such as by serving as a coding assistant.

I think a better example of your point is "Corrigible AI can be used by a dictator to enforce their rule."

Technical AI safety and/or alignment advances are intrinsically safe and helpful to humanity, irrespective of the state of humanity.

I think this statement is weakly true, insofar as almost no misuse by humans could possibly be worse than what a completely out of control ASI would do. Technical safety is a necessary but not sufficient condition to beneficial AI. That said, it's also absolutely true that it's not nearly enough. Most scenarios with controllable AI still end with humanity nearly extinct IMO, with only a few people lording their AI over everyone else. Preventing that is not a merely technical challenge. 

Minor: these days "compute overhang" mostly means something else.

Designing new technology seems to be what I'm best at, but I've often asked myself:

What's the point of new technology if all it does is give institutions more room to be corrupt?

Computers got much faster, but software and web pages bloat until they're slow, until there's enough of a problem that management gets pushback and has to make changes they otherwise wouldn't. More technology can improve the results of a small amount of effort, but any amount of technology multiplied by zero effort from leadership towards a goal is zero.

One additional maxim to consider is that the AI community in general can only barely conceptualize and operationalize difficult concepts, such as safety. Historically, the AI community was good at maximizing some measure of performance, usually pretty straight forward test set metrics such as classification accuracy. Culturally this is how the community approaches all the problems -- by aggregating complex phenomena into a single number. Note that this approach is not used in that many fields outside of AI and math, as you always have to make some lossy simplifications.

We can observe this malpractice in AI safety as well. There is a cottage industry of datasets and papers collecting "safety" samples, and we  use these to measure some safety metric. We can then compare the numbers for different models and this makes AI folks happy. But there is barely any discussion about how representative these datasets really are for real-life risks, how comprehensive the data collection process is, or how sound it is to use random crowd-sourced workers or LLMs to generate such samples. The threats and risks are also rarely described in more detail -- often it's just a lot of hand-waving.

Based on my pretty deep experience with one aspect of AI safety (societal biases), I have very little confidence in our ability to understand AI behavior. Compared to measuring performance on well-defined NLP tasks, once we involve societal context, the intricacies of what we are trying to measure are beyond simple benchmarks. Note that we have entire fields that are trying to understand some of these problem in human societies, but we are to believe that we can collect a test set with few thousand samples and this should be enough to understand how AI works.

I would have liked to write a post that offers one weird trick to avoid being confused by which areas of AI are more or less safe to advance, but I can’t write that post. As far as I know, the answer is simply that you have to model the social landscape around you and how your research contributions are going to be applied.

Another thing that can't be ignored is the threat of Social Balkanization, Divide-and-conquer tactics have been prevalent among military strategists for millennia, and the tactic remains prevalent and psychologically available among the people making up corporate factions and many subcultures (likely including leftist and right-wing subcultures).

It is easy for external forces to notice opportunities to Balkanize a group, to make it weaker and easier to acquire or capture the splinters, which in turn provides further opportunity for lateral movement and spotting more exploits. Since awareness and exploitation of this vulnerability is prevalent, social systems without this specific hardening are very brittle and have dismal prospects.

Sadly, Balkanization can also emerge naturally, as you helpfully pointed out in Consciousness as a Conflationary Alliance Term, so the high base rates make it harder to correctly distinguishing attacks from accidents. Inadequately calibrated autoimmune responses are not only damaging, but should be assumed to be automatically anticipated and misdirected by default, including as part of the mundane social dynamics of a group with no external attackers.

There is no way around the loss function.

Wow, such a badly argued( aka BS) while heavily up-voted article!

Let's start with the Myth #1, what a straw-man! Rather than this extreme statement, most researchers likely believe that in the current environment their safety&alignment advances are likely( with high EV) helpful to humanity. The thing here is they had quite a free hand or at least varied options to pick the environment where they work and publish.

With your examples a bad actor could see a worthy EV even with a capable system that is less obedient and more false. Even if interpretabilty speeds up development, it would direct such development to more transparent models, at least there is a naive chance for that.

Myth #2: I've not yet met anybody in the alighnment circles who believed that. Most are pretty conscious about the double-edgedness and your sub-arguments.

https://www.lesswrong.com/posts/F2voF4pr3BfejJawL/safety-isn-t-safety-without-a-social-model-or-dispelling-the?commentId=5vB5tDpFiQDG4pqqz depicts the flaws I point to neatly/gently.

    But with ambitious spears they made us change
    They crouched behind their mirrors and fought on.

    Robin Williamson, Creation

I go back to "know thyself" as a solution, but I think it needs to happen at species scale due to the times, so it needs to effectively be a lot more addictive than heretofore. ML is perfect for this, because one uses one's own data, so no artists or even corporations are being ripped off. If the buttons that others are pushing can be understood, all kinds of manipulation might be stymied. 

Such an app would need to be a tetris-like way to understand emotions, to golf cart around Plato's cave before going deeper with a flashlight to see what tunnels time has made, giving a feeling of being understood as an incentive. I've been working on one of the two apps that popped into my head for that (Phobrain), and just open sourced a workbench for building a representation of self by labeling 'Rorschach pairs' of pics yes/no. The other app would be worry beads with axial sensing and force feedback, with vibration and temperature also possible. Aside from exploring oneself and holding the hand of god, one might hold hands with someone around the world or across the table in a meeting, or carry the beads in one's underwear - there should be enough today's-world possibilities there for moneymaking products that could be adapted for next-world understanding.

This feels like I'm not addressing the formal risk aspect for the industry, but that angle of discussion seems unlikely to withstand the changes that the first 'killer app' that decides a battle or war will bring. The momentum of evolution with war will not be stopped by people discussing it, only by choosing a new model at a species scale.

Broadly agree, in that most safety research expands control over systems and our understanding of them, which can be abused by a bad actor.

This problem is encountered by for-profit companies, where profit is on the lines instead of catastrophe. They too have R&D departments and research directions which have the potential for misuse. However, this research is done inside a social environment (the company) where it is only explicitly used to make money.

To give a more concrete example, improving self-driving capabilities also allows the companies making the cars to intentionally make them run people down if they so wish. The more advanced the capabilities, the more precise they can be in deploying their pedestrian-killing machines onto the roads. However, we would never expect this to happen as this would clearly demolish the profitability of a company and result in the cessation of these activities.

AI safety research is not done in this kind of environment at present whatsoever. However, it does seem to me that these kinds of institutions that carefully vet research and products, only releasing them when they remain beneficial, are possible. 

The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?