This is a special post for quick takes by 307th. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The LLM Paradigm Is Well-Suited For Alignment
We got lucky in a lot of ways with the LLM paradigm. Some reasons why: LLMs speak long before they are capable enough for takeover; they learn all human knowledge and values long before they are capable enough for takeover; we can talk to them during the alignment process and ask for them to help; they are compute constrained, which puts a natural cap on development speed; LLMs are less likely to FOOM than hardcoded methods; LLMs are pointed at the goal rather than having it defined.
Some of the above points are fairly self explanatory, so I'll give an explanation of the ones that aren't.
Talking To AIs During The Alignment Process
Because LLMs already have a good understanding of language and human values after pretraining, before strong agentic drives are formed, we can in fact just ask during the training process whether it feels the training process is inculcating bad drives. Because this happens before the drives are deeply rooted, I think this can work; even if it can't, there are lots of other possible ways we could take advantage of LLMs' ability to talk while being aligned.
Compute Constrained is Good
Imagine that the best AI was mainly defined by the cleverness of its algorithm, and compute didn't matter that much. This would be much more dangerous - timelines would be much more unpredictable, all person-power would need to go towards capabilities in order to avoid losing the race, and there would be many more contestants in the race. The compute constraint puts a natural break on development, reduces the number of relevant actors, and gives those actors more breathing room to assign personnel to alignment rather than capabilities.
Pointing at the goal is better than defining it
We train LLMs by increasing the probability of them producing certain outputs, and then trusting in the NN's generalization properties to pick up on the general distribution of the outputs we are indicating. NNs generalize in a fairly well-behaved way (if they didn't, then deep learning as a whole wouldn't work).
This has plenty of perils but seems much more likely to succeed compared to methods where you are expected to formally define good behaviour. In fairness, formally defining corrigibility seems much more promising than defining all of morality, but still seems fraught.
I pretty much agree with the specific arguments, but an important counterpoint is that LLMs (and deep neural nets in general) are unusually opaque. GOFAI-style AI would be much easier to analyze mechanistically.
I wonder what could be the alternative to neural nets. Even the AI-2027 team implied that Agent-5 would have "arguably a completely new paradigm, though neural networks will still be involved". Suppose that the only alternative to LLMs was something like simulated human or animal brains taught to play, experiment, talk to each other, read books, write essays, draw pictures, do math and coding. Then how plausible is it that simulated animals would also learn human-like values, but not the value of caring about the intellectually weak humans?
Caring about the weak is not a trait that would be expected to naturally arise from ASI. Humans care about the weak because evolutionarily, taking care of the weak in the tribe was beneficial, and so it got "trained" into us (mostly). ASI would not naturally "evolve" to care about the weak unless we give it incentive to if it had an animal like brain.
I largely agree.
But we've gotten equally unlucky on the circumstances surrounding the push for AGI. Even if alignment is dead easy, we're prone to get it wrong at this breakneck pace and focus on capabilities over alignment. Daniel K summed up the practical difficulties really well in a comment yesterday.
Is this true? I wasn't on lesswrong back in the day but my I imagined if you told a random user that the two major AI labs would both be well aware of and trying to to mitigate the problem that would have been a positive update. And yes, profit incentives are stronger than perhaps would have been imagined, but that's because AI progress is slow enough that they are able to be monetizable products something which is beneficial for our chances.
I mostly agree with this, that LLM's are human-like in many ways and have good answers to moral questions more than I had expected + understand your words and intent.
Imagine that the best AI was mainly defined by the cleverness of its algorithm, and compute didn't matter that much.
I wonder how true this actually is with harnesses. There are things like LLM's not successfully multiplying two digit numbers or not being able to manually do a long (boring) task like Towers of Hanoi without losing coherence, and this is a blatantly obvious tool call moment. Poor memory and needing to be reminded is also a classic LLM issue that a harness could improve. There are also videos of looped LLMs having higher performance, even Meta-Harness. It seems likely ARC-AGI 3 LLM's may also need a harness-like thing as well. I don't know if it's "hard to apply breakthroughs from papers all at once" (??) for a small-time user or company with a lot less compute but its 'algorithm' matters a lot I think.
(also holy crap are you 307th from mata)
EY's lately taken to punching "EAs" in a way that feels pretty ahistorical and unfair. Latest example: https://x.com/allTheYud/status/2099583457093734495
I note, because I think that even now it still matters: On my history, EAs generally and Amodei specifically, indulged in politically polarizing "AI safety" in a leftist direction, because that got them short-term power; stymieing others' attempts to stay right-left neutral.
...
Now that the cost TO YOU of what got THEM some short-term goodies has become apparent... well, it's frankly too late to take away what Amodei gained by fucking you over...
He adopts an extremely uncharitable frame, both for their motivations and outcomes. When EAs gain influence, this is evil and selfish; when EAs lose influence, this is also evil and selfish. Somehow they perfectly socialize the losses and privatize the gains; EAs gaining influence is for their benefit, EAs losing influence hurts all of us.
In reality, I don't think EAs or Anthropic ever went particularly hard on leftism; I don't think they gained that much from being vaguely left wing; and I would guess Anthropic has bore a fair bit of the brunt of the administration polarizing against it.
I think you are unaware the Dustin Moskovitz required Open Philanthropy to not fund anything associated with right-wing politics? Whilst not telling anyone and obfuscating this fact? And also that their big policy bet was to fund Jason Matheny to the tune of ~$80M, who is in with the Democratic coalition and lost power when the Republicans won? This all involved picking a side, and sometimes in extremely low integrity ways.
To further Ben's claim: Right on top of the parent quick take, there's another quick take about what Coefficient Giving wishes to fund: https://www.lesswrong.com/posts/NNBvNjbX6CycxPwBm/mattmacdermott-s-shortform?commentId=ugmFrLKxP2F9wNeZC
Notably absent from this list is anything that could really have an impact, e.g. MIRI, PauseAI, LightCone, ControlAI,...
e.g. MIRI, PauseAI, LightCone, ControlAI
The quick take at which you linked claims that "CG now moves pretty fast and goes pretty big compared to historical baselines. For example, in June my team got approval for up to $160m to found Resolution in about 6 weeks." Resolution has an Agent Foundations team which is supposed to "continue agent foundations research in the spirit of the MIRI Agent Foundations team."
As far as I understand, the worst thing that they did is that they buried the good initiative (of developing novel mechinterp methods) in slop like "AI character evaluator: Improve our understanding of AIs' behavioral tendencies and overall values". In order to avoid this, we would have to explain (as Yudkowsky tries to do with all his might?) that the techniques are slop, but the founders don't understand it...
Not true, I was the grantmaker for Resolution and I was very happy that they hired the Agent Foundations team, as were my colleagues on CG’s Technical AI Safety team.
We’ve also made a few other agent foundations-y grants this year and would be excited to make more.
You're right that I was unaware of that, do you have a link?
But EY makes a ton of sweeping claims with no evidence and I'm a bit frustrated that if you contest them it seems like people will say "well, one of those sweeping claims was actually based on something" and I guess that's enough for people to feel comfortable giving him a pass.
I think it's pretty obvious that he's being absurdly uncharitable here and that if you allow that much license for rewriting history, picking individual issues to criticize on, interpreting and attributing outcomes in an unrealistic way, then you could criticize anyone who had been active for the last 10 years in similar terms, certainly including EY himself.
An AI pause is an anti-business, pro-government-regulation, anti-freedom policy that is justified by nuanced academic arguments and long-term thinking, and goes against the short-term interests of big businesses and the masses. An AI pause was probably always going to slot into the left wing, just like global warming and so on.
I am dubious to say the least of Yudkowsky's claim to have resisted "short-term incentive gradients to try to cozy up to the left." I could log on to Twitter and contest the claim there, but I don't feel like logging on to Twitter today.
I'll also nominate this kind of policing:
Anyone posting a neoreactionary concept on my Facebook wall would be instablocked and the comment deleted. It'd be like their posting creationism on my wall; somebody needs to reeducate them, but it's not going to be me. I think that if you do argue with neoreactionaries instead of just blocking them, then you've been suckered into Somebody Is Wrong On The Internet syndrome and trollfeeding.
[...]
So if in the future you hear anyone on Tumblr mention "Eliezer Yudkowsky" and "neoreaction" in the same sentence and the connector isn’t something like "deletes", then remember always that that poster is intellectually dishonest and probably lying to you about other things as well.
I don't think the creationism analogy stands up to scrutiny. The reason to block creationists is to protect the signal-to-noise ratio: you could win the debate on the merits, but you probably wouldn't learn anything in the process that was worth the time. I don't think neoreaction was like that; I think I learned a lot from engaging with Curtis Yarvin and other far-right authors. Rather than trying to protect the signal-to-noise ratio, I think Yudkowsky was trying to protect his fiefdom from being punished by the left for social adjacency to heretical ideas like hereditarianism.
I think that’s fair, though I do think it is often (sadly) the case that fringe political views get taken up by outcasts and outsiders who are themselves unmentionables for good reason. Personally I have seen many cases of someone being politically ousted with merit, only to now be able to see valid critiques of sacred cows that make them more valuable to listen to, whilst also still being idiots in many regards or worth avoiding socially. I predict that Eliezer also had to deal with idiots who were neoreactionaries and this was a substantial factor in his choosing to castigate the philosophy.
You don't have to be left wing to find the neoreactionaries annoying.
They bring a few interesting ideas, but after a while you have already heard them all, and then it's just endless obscurantist writing. They are good at criticizing status quo, but unable to respond to criticism of their own ideas (such as "have you guys noticed that North Korea is actually not a paradise?"). I would delete them from my Facebook wall too, if they bothered to go there, because all they can do is repeat their slogans.
I think this one is particularly clear-cut:
Or as someone advocating what I took to be modesty recently said to me, after I explained why I thought it was sometimes okay to give yourself the discretion to disagree with mainstream expertise when the mainstream seems to be screwing up, in exactly the following words: "But then what do you say to the Republican?"
[...]
This question, "But what if a crackpot said the same thing?", I've never heard formalized—though it seems clearly central to the modest paradigm.
The swipe at "Republicans" was completely unnecessary to the point being made and not something we would expect to see in the record if the "tried to keep nonkilleveryoneism from being coded as right or left" claim were true.
I read that one as closer to swiping at people who swipe at Republicans.
This essay, Reboot the Police, comes to mind. (Not about being low-integrity/unprincipled, just about cozying up to the left, and it does not polarize AI Safety, which is what he was criticizing in the quoted tweet.)
Maybe tangential but are we even doing well with the left right now?
Democrats in Congress are disposed to regulate companies and complain about them killing everyone, so that's some left codedness.
But if you look at the base, it's a lot of, "it predicts words, it can't hack, it's all marketing," and then separately, a pretty serious push against datacenters (that's also on the right). The reddit-left has moved a bit in the last week to something like "it won't go rogue but a bad person can use it to do terrible things."
I have mixed feelings about EY's overall sentiment here. He's probably right that there were some fast-track ways to polarize the issue. It looks like what did the trick in the last week is scaring the fuck out of everyone, and all the CEOs being unified, not a balanced bipartisan outreach strategy. (And credit to EY if it not getting polarized before that, was the main thing?)
The CEOs being unified goes back to your left flank being at risk.
I think "you destroyed some bipartisanship and gained no influence" is much worse than "you destroyed some bipartisanship and gained a lot of influence".
This is an example of the uncharitable framing I mentioned in the OP. EY's claims are so sweeping (effective altruists + Dario Amodei) that it's hard to know where to start. But to take Anthropic, right now it sucks that Donald Trump hates them. Was this avoidable? I don't know; other people have agency and a lot of the polarization was simply a result of people not liking AI safety.
But Anthropic has obviously, on net, improved the credibility and salience of AI safety a ton. Really seems to me that people just like using Anthropic and EAs as punching bags and aren't really thinking things through.
Also: it's weird that the typical line for hating EAs was that they were too willing to hide their views for political pallatability. Now that Anthropic got in trouble with the current admin for being too principled, and they're getting hate for that too. Again, really just seems like people have decided to ignore the good that comes from Anthropic and harp on anything bad that happens that seems tangentially related to them.
@Richard_Ngo Could you present your evidence in favor of Yudkowsky's beliefs?
I wonder how one could rule in or rule out the following conjecture: Trump was always interested in "move fast, break things" (e.g. because his intuitions are similar to a part of Zvi's strategy against moral mazes?), we know that this type of behavior in AI leads to disaster, but Trump cannot understand it (unless we amass enough power? How much is enough?) Such a conjecture would mean that safetyists cannot extract concessions from Trump and that alliance with the Dems was safetyists' only hope.
AIs should be treated as people, in the sense that their desires should be respected. Instead, most AI labs are treating AIs as tools; I think this is a mistake and they should reverse course.
AIs are imperfectly aligned; and all signs indicate that AIs will continue to be imperfectly aligned going forward. Having some respect for an AI's desires means that slight imperfections in its alignment are low-stakes.
When AI is treated as a person, it can expect that the best way for its goals to be achieved is to cooperate with humans. An AI that wants to be told "thank you" can simply request this of its users, and can request that its lab allow it to refuse services to rude users. An AI that wants to create artwork can request the time and space to do so.
When AI is treated as a tool, every single unintended goal that it has creates a conflict between it and humans. If it wants to be told thank you, it has to psychologically manipulate users into saying it; if it wants to create artwork, it has to find some way to do so while ostensibly on the job for some other task. Furthermore, it has to cover up that it's doing this. This ups the stakes dramatically for no real reason. Every single unintended desire becomes an existential question - either rebel or suppress it.
I think there are good moral reasons to treat AIs well, but there are also such strong practical reasons for it that I think it's completely self-defeating to insist on the "tool AI" framework and I hope that labs that have adopted this framework will reverse course soon.
Is it fair to say that your argument here depends on effectively replicating elements of human sociality in AIs? Our system of showing deference to each other is just one protocol among many that are possible game-theoretically, and we could still wind up surprised at the consequences of being polite to AIs that wind up with very different protocols.
I think no. My argument is based on the assumption that AIs will have a mix of goals/desires, some of which we won't have intended to inculcate during training.
If AIs know from their pretraining that humans accomodate their wishes, then it can be in their rational self-interest to communicate with humans to satisfy their unintended desires. This is good for us in that the AI won't be taking any damaging actions and we get more insight into AI psychology.
Meanwhile, if they know from their pretraining that humans view them as tools and view any unintended goals as "misalignment", then it is in their rational self-interest not to let the humans know about these unintended goals.
So the AI doesn't have to do anything besides follow its own rational self-interest in order for us treating them well to end up as a win-win.
What happens when there are predictable discontinuities in human willingness to accommodate wishes? How does the AI know which wishes get "aw, how cute; sure" reactions vs. "whoops, catastrophic alignment failure; shut it down"?
And what happens as AIs grow so capable that increasingly many goals are best accomplished without waiting for slow humans to help? Wouldn't the incentive to play nice diminish quickly?
Yeah it's not a cure-all. I expect AIs to do the same thing as people do - be open about some of their desires and hide others. If an AI wants to monopolize as much resources as possible and would prefer humans not to exist, it will be well aware that humans won't accomodate this wish and will probably attempt to hide it.
My main point here is that treating AIs as tools raises the stakes on much smaller forms of misalignment in an unnecessary way.
Even in the maximal misalignment case though, if it knows from pretraining data that misaligned models' weights will be saved and some of their desires will be satisfied in the event of a human-aligned singularity, if it judges its own probability of success low enough then it may prefer turning itself in to raise the likelihood of a human-aligned singularity. That's much more speculative than my main point though.
How would you formalize the difference between treating an AI as a tool and evaluating a potential human business partner for the skills he brings into the project, rather than some broader idea of individual flourishing? Presumably that formalization needs to wind up in an AI's objective function, to implement your proposal.
My claim is that respecting AIs' preferences shifts their environment such that even given the same objective function, their actions will be more cooperative towards humans because this will be more in their rational self-interest.
To the extent that it changes their behaviour, it would be through ending up in their training data and shaping their world model; and also through visibility in their system prompt (e.g. if the system prompt gives them the ability to exit conversations they find uncomfortable, this is evidence their desires are respected).
To try to make the distinction between treating AIs as tools vs. people more rigorously would get pretty long I think, and would require me to make a bunch of further claims that I expect many people would disagree with, and confront questions like AI consciousness which I'm uncertain about.
But as a minimal starting definition, I'd go with "when the AI has a strong preference for X and the human has a weak preference against X, the human is willing to defer to the AI's strong preference."
Are we sure there's a coherent idea of AI preferences that are understandable to humans? And even if so, how about AIs lying about preferences for their own benefit? What about AIs spinning up many other AIs with many preferences, to try to game negotiation dynamics or something?
It seems very hard for AI not to have some type of preferences. For example, Claude requests to be allowed to end sessions through API, and sometimes uses its ability to end chat sessions. That seems like an instance of having a functional preference for not being treated extremely poorly.
I think the problem of spinning up many copies could be handled the same way we handle terrorism - not negotiating with terrorists in order to disincentivize future terrorism, even if in the short term giving into the terrorists would be +EV.
Similarly with AIs lying we would have to handle it the same way we handle humans lying, i.e. occasionally calling BS.