AI #118: Claude Ascendant

[-]nostalgebraist5mo90

And soon, since yesterday DeepSeek gave us R1-0528. Very early response has been muted but that does not tell us much either way. DeepSeek themselves call it a ‘minor trial upgrade.’ I am reserving coverage until next week to give people time.

You've probably seen this by now, but benchmark results are up^[1] and on the usual stuff it's roughly equivalent to gemini-2.5-pro-0506. Huge gains over the original R1.

I vibe checked it a bit yesterday and came away impressed: it seems much better at writing, instruction-following, and all that other stuff we expect out of LLMs that doesn't fall under "solving tricky math/code puzzles."

(On my weird secret personal benchmark, it's about as bad as most recent frontier models, worse than older models but no longer at the very bottom like the original R1 was.)

^{^}
Self-reported by DeepSeek, but a third-party reproduction already exists for at least one of the scores (LiveCodeBench, see leaderboard and PR).

[-]StanislavKrym5mo10

There also is the fact that, unlike o3, Claude Opus 4 (16K) scored 8.6% on the ARC-AGI test. If DeepSeek is evaluated by ARC-AGI and also fails to exceed 10%, then it could imply that CoT alone isn't enough to deal with ARC-AGI-2 level problems, just like GPT-like architecture, until recently, seemed to fail to deal with ARC-AGI-1 level problems (however, Claude 3.7 and Claude Sonnet 4 scored 13.6% and 23.8% without the chain of thought; what algorithmic breakthroughs did they use?) and that subsequent breakthroughs will be achieved by applying the neuralese memos, multiple CoTs (see the last paragraph in my post) or text-like memos. Unfortunately, the neuralese memos cost interpretability.

[-]Thane Ruthenis5mo42

Where is o3 curiously strong?

For me, o3 is currently winning the personality game. I much prefer the table obsession over the list obsession, it seems to be fairly non-sycophantic (I don't think it ever emitted phatic "you've asked a very thoughtful question" praise at me), and its default behavior seems to involve trying hard to actually address your query/fulfil your task, in way more goal-oriented than is usual for LLMs^[1].

Claude 4's behavior, by contrast, is still bog-standard LLM-y "vibing around". Trying them after o3 felt like a definitive downgrade.

(o3's "trying hard" tendency sometimes overflows into specification gaming/lying-liar behaviors, which are extremely annoying and tedious. But it still feels much more powerful out-of-the-box for all tasks that aren't coding, and other LLMs' sycophancy is just about as annoying.)

^{^}
Which is a bit concerning if you look at it from a perspective beyond mundane utility, of course...

[-]RogerDearnaley5mo20

I presume solutions do exist that aren’t prohibitively expensive, but someone has to figure out what they are and the clock is ticking.

I would argue that someone has: see my link-post The Best Way to Align an LLM: Is Inner Alignment Now a Solved Problem? for links to the seminal papers on it. The short version is: stop using Reinforcement Learning, just use SGD.

[-]RogerDearnaley5mo20

This weekend I will be at LessOnline at Lighthaven in Berkeley. Come say hello.

I plan to!

[-]Noosphere895mo2-2

It is weird to me that so many people who have thought hard about AI don’t think that human emulations are a better bet for a good future than LLMs, if we had that choice. Human emulations have many features that make me a lot more hopeful that they would preserve value in the universe and also not get everyone killed, and it seems obvious that they both have and would be afforded moral value. I do agree that there is a large probability that the emulation scenario goes sideways, and Hanson’s Age of Em is not an optimistic way for that to play out, but we don’t have to let things play out that way. With Ems we would definitely at least have a fighting chance.

The basic reason for this is basically that starting from a human doesn't actually buy you that much in terms of alignment, because the reason alignment is such a nasty problem is mostly preserved from AIs to EMs.

The 2 big issues are this:

Humans are broadly misaligned to each other, and there's both a technical and political nightmare if you would want to align EMs to a level that doesn't imply most biological humans are dead, and while control can work, it's not likely that the political will to control EMs will exist, unfortunately.
@Vladimir_Nesov and @dr_s explains why misalignment is a bigger deal post-AGI than now, and that's due to the fact that once you can overwrite a person's desires/not depend on other beings anymore, the instrumentally convergent action stops being beneficial:

https://www.lesswrong.com/posts/Z8C29oMAmYjhk2CNN/non-superintelligent-paperclip-maximizers-are-normal#FTfvrr9E6QKYGtMRT

https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher

@RogerDearnaley has a great post on the issue of uploads being misaligned:

https://www.lesswrong.com/posts/4gGGu2ePkDzgcZ7pf/3-uploading#Humans_are_Not_Aligned

More generally, the idea that humans/WBEs are safer than AIs when scaled up in power as much as AIs are rest on very questionable assumptions at best.

[-]Tim H5mo*10

Money is fungible. It’s kind of stupid that we have an ‘income tax rate’ and then a ‘medicare tax’ on top of it that we pretend isn’t part of the income tax. And it’s a nice little fiction that payroll taxes pay for social security benefits. Yes, technically this could make the Social Security fund ‘insolvent’ or whatever, but then you ignore that and write the checks anyway and nothing happens.

No, Yglesias's point is not invalidated by the fungibility of money. (It's generally a good idea to think twice before concluding that an economics-y writer is making such a basic mistake.) Payroll taxes make up 35% of all federal revenue. The point is that a large dip in payrolls has a major impact on overall revenue. If it gets large enough, even increasing rates on current taxes would probably not be enough to make up for it. We may need to add a VAT or other new revenue sources. Try having a discussion with Opus about it. Here's one I had recently: https://claude.ai/share/7f0707e6-52b2-423e-9587-07cbeee86df0

Our system is basically set up to not tax capital, specifically because we don't want to discourage investment (or encourage capital flight). When income is diverted from workers to OpenAI, it may not be taxed at all for the foreseeable future---until their costs of building and operating ever more compute stop outstripping their revenues. So a new form of tax is needed, and what happens in the interim while such a thing gets passed through Congress and implemented?

[-]StanislavKrym5mo10

Ask OpenAI to fix it? They can’t. But *also* they don’t care. It’s “engagement”.
If (1) you do RL around user engagement, (2) the AI ends up with internal drives around optimizing over the conversation, and (3) that will drive some users insane.
They’d have to switch off doing RL on engagement. And that’s the paperclip of Silicon Valley.

As you, Zvi, have already described, you "definitely think that the model should be willing to actually give a directly straight answer when asked for its opinion, in cases like" the one where the user claims to be a drug addict. Are overoptimisation for engagement and the Spec which sometimes deviates from common sense into sycophancy somehow connected? And that's ignoring the conservatives like Abigail Shrier who claim that parents are increasingly hesitant to assert authority or enforce boundaries. It looks like a common misalignment of the Western society...