O O's Shortform

O O

This is a special post for quick takes by O O. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

A while ago I predicted that I think there's a more likely than not chance Anthropic would run out of money trying to compete with OpenAI, Meta, and Deepmind (60%). At the time and now, it seems they still have no image video or voice generation unlike the others, and do not process image as well in inputs either.

OpenAI's costs are reportedly at 8.5 billion. Despite being flush in cash from a recent funding round, they were allegedly at the brink of bankruptcy and required a new, even larger, funding round. Anthropic does not have the same deep pockets as the other players. Big tech like apple who are not deeply invested in AI seem to be wary of investing in OpenAI. It stands to reason, Amazon may be as well. It is looking more likely that Anthropic will be left in the dust (80%),

The only winning path I see is a new more compute efficient architecture emerges, they are first, and they manage to kick of RSI before more funded competitors rush in to copy them. Since this seems unlikely I think they are not going to fare well.

Indeed. Although interesting that Sonnet 3.5 now accepts images as input, just doesn't produce them. I expect that they could produce images, but are choosing to restrict that capability.

Haiku still doesn't accept image input. My guess is that this is for efficiency reasons.

I'm curious if they're considering releasing an even smaller, cheaper, faster, and more efficient model line than Haiku. I'd appreciate that, personally.

3O O2y

https://x.com/arcprize/status/1849225898391933148?s=46&t=lZJAHzXMXI1MgQuyBgEhgA My read of the events. Anthropic is trying to raise money and rushed out a half baked model. 3.5 opus has not yet had the desired results. 3.5 sonnet, being easier to iterate on, was tuned to beat OpenAI’s model on some arbitrary benchmarks in an effort to wow investors. With the failed run of Opus, they presumably tried to get o1 like reasoning results or some agentic breakthrough. The previous 3.5s was also particularly good because of a fluke of the training run rng (same as gpt4-0314), which makes it harder for iterations to beat it. They are probably now rushing to scale inference time compute. I wonder if they tried doing something with steering vectors initially for 3.5 opus.

2Nathan Helm-Burger2y

I doubt very much that it is something along the lines of random seeds that have made the difference between quality of various Sonnet 3.x runs. I expect it's much more like them experimenting with different datasets (including different sorts of synthetic data). As for Opus 3.5, Dario keeps saying that it's in the works and will come out eventually. The way he says this does seem like they've either hit some unexpected snag (as you imply) or that they deprioritized this because they are short on resources (engineers and compute) and decided it was better to focus on improving their smaller models. The recent release of a new Sonnet 3.5 and Haiku 3.5 nudges me in the direction of thinking that they've chosen to prioritize smaller models. The reasons they made this choice are unclear. Has Opus 3.5 had work put in, but turned out disappointing so far? Does the inference cost (including opportunity cost of devoting compute resources to inefficient inference) make the economics seem unfavorable, even though actually Opus 3.5 is working pretty well? (Probably not overwhelmingly well, or they'd likely find some way to show it off even if they didn't open it up for public API access.) Are they rushing now to scale inference-time compute in o1/deepseek style? Almost certainly, they'd be crazy not to. Probably they'd done some amount of this internally already for generating higher quality synthetic data of reasoning traces. I don't know how soon we should expect to see a public facing version of their inference-time-compute-scaled experiments though. Maybe they'll decide to just keep it internal for a while, and use it to help train better versions of Sonnet and Haiku? (Maybe also a private internal version of Opus, which in turn helps generate better synthetic data?) It's all so hard to guess at, I feel quite uncertain.